|
October 8, 2008 -- Sometimes disasters come all at once and overwhelm you. Other times, they start small and just go on and on. Now, I'm not talking about disasters on the magnitude of Katrina or Ike. I'm talking about the kinds of small disasters that make us just pull our hair out in frustration as they continue day after day. In the past week plus, I have gone through probably the single most frustrating vendor experience I have had in twenty-three years of IT consulting. The thing is, like a train wreck in slow motion, it's hard to look away. So, here is my log. Names are withheld to protect the guilty, but hold your hats, this is a ride.
To get to one of my clients, I take a subway train the crosses the East River on a bridge between Manhattan to Brooklyn. Last Tuesday, as I was crossing the bridge, my phone beeped and I retrieved a voice mail. A technician from the client's phone/Internet service vendor (ISP) had called to say that he had been been working at one of my client's sites and now the Internet was down -- but it wasn't his fault! As the message finished, we went into the tunnel on the Brooklyn side of the river and again out of cell-phone reach.
When I arrived at my client's main office about twenty minutes later, I was again informed that the Internet was down at the other site. This site is important because it is the hub of the organization's wide area network (WAN) and the location of a server that house's one of the organization's most important applications which staff members access via the WAN.
I called our ISP and, after speaking with several customer service reps, was given a ticket number. I was told that a tech would work on it as soon as possible and would get back to me. I called a number of times (about hourly) after that and was told repeatedly that a technician was working on the problem. Finally, after some insistent talk on my part, I was transferred to an actual technician about 6:00 PM. He could not, at first, find our ticket number. He finally did and said it was not a proper ticket number -- the ticketing system had been down, he explained. He did some tests and told me he could see our router and that he didn't know what the problem was. He said he would dispatch a technician, who should be there the next day.
By 10:00 AM Wednesday, no tech had arrived. I again called our ISP and was informed that no technician had been dispatched. Again, they said someone would get back to me. I called hourly, each time repeating my story, explaining how important this issue was to their customer, and noting how long service had been down. Each time, I was told that someone would get back to me. While I tried to be polite and keep cool, each time I spoke with someone I was a little more forceful and insistent.
By 3:00, I was being pretty forceful, when the rep said, "You know, you can ask to have the problem escalated." I had to ask? You mean you weren't doing something because I didn't know the magic word? Well, I'm asking now
Half an hour later, I got a call back from a very apologetic manager. Yes, yes, it was a shame more hadn't been done already. He would see what he could do. He called me back half an hour later to tell me that a tech was being dispatched and would be at the site at 6:00 PM.
I was there at 5:30 PM; the tech arrived at 7:00. He explained that they (the ISP) had put in new circuit and would now finish the install of the new router that the previous tech had installed the day before. They tried, and failed, to properly configure the router. They tried again, this time successfully, but by this time it is clear that they flying by the seat of their pants. Finally the data repair done, they decided to "upgrade" the phone system as well. When they finished we did a cursory test the phone system. We made calls out and received calls at several extensions. At 9:30 PM, 34 hours after the technician's voice mail, we are done -- or so I thought.
At 10:00 AM Thursday, Shirelle called me to tell me that calls for 5 phone lines were being misdirected to other extensions. Worse, people could not get their voice mail. I called the vendor again and got another ticket number along with assurance that someone will call back. No one calls me back. I called again Friday morning and I was told that they may have to dispatch a tech. Subsequently, Adam, a technician, called back and told me (a) that I have the wrong ticket number (again?) and (b) it is being worked on. He also told me that this appeared to be another failed upgrade, like there are lots of them. He promised to call back, but never did. By Friday night, the problem still existed.
After much back and forth Monday and Tuesday, a tech finally showed up on Wednesday. It happens that I had caught the runny-nose-totally-rundown thing and was going nowhere. Unfortunately, the tech called to tell me that the Internet connection was down (again), but it wasn't his fault (again). He informed me that the firewall appeared to be dead. I asked him to check the power connection. He said it looked okay.
I exchanged my slippers for sneakers put on a presentable shirt, and headed out the door with a spare firewall. The tech was gone when I arrived. I also had a voice mail from Deanna -- a calm and actually helpful voice at the vendor's end of the phone line -- that the telephone problem had been resolved. We tested it and everything seemed to be working.
The dead firewall? I found the power cable plug on the floor, probably kicked out by the technician when he was working behind the server rack.
It only took the vendor nine days to fix two problems that they had created. So, now you know where the emerging gray in my hair comes from. And, Deanna, I owe you some flowers. Better, tell your boss he or she owes you some flowers.
Trackback(0)
 |
Gary's Blog 
TrackBack URI for this entry
Subscribe to this comment's feed