Here is why the SLAGA email service went away for awhile..
glenn
===
16 Hours without FreeLists, or, why to not give SBC any of your money.
In short, we just got through a 16-hour outage, probably the worst in our history aside from that "lightning loves us" outage a couple years back. It was another long night for me (although I finally did get some sleep (unlike some previous outages :)) and I've posted my log of the saga below. Times are approximate: I wasn't actually writing all this down as it happened; I also wasn't expecting the story to be so convoluted.
It just goes to show you that no matter how much money you spend on internet services, no matter how good your ISP is, you're going to have outages that are well beyond your control and the control of those you contract with. Oh yeah, and that Ameritech really sucks. (My apologies to anyone here who works for them, the comment isn't directed at anyone in particular.)
10/31
8:30pm: IQuest internet services calls me. As soon as you hear "John? This is nnn, from the IQuest ops center," you know something's wrong. "Oh crap, don't even tell me the T1's down." Uhm yeah, quite down. Everything looks fine on our end, so I suggest he "open a ticket."
9:05pm: IQuest calls again. "Ok, ticket's in and at Time Warner." You see, in this day and age, you can't say "give me a line from here to there" unless you happen to own those lines. Otherwise, you've got to go through three or four separate parties to get your line, and it won't really go where you thought it would anyway. Whenever something goes wrong, chances are that you'll be spending a long time passing trouble tickets around. Blah.
9:45pm: Time Warner calls: "Yeah, we've got your ticket and it looks like you're down, so we're passing this ticket onto Ameritech." It's at this point that I realize we're all doomed. If your ticket goes to Ameritech, your outage will last at least 6 hours. It doesn't help that it's the night shift, so personnel is thin.
11:15pm: Time Warner calls again: "Ok, it looks like Ameritech is going to have to go out to check the T1 directly." Great, now I have to wait for someone to drive from western Kansas or something.
11/1
1:30am: Ameritech technician calls. "Yep, so there's a problem with yer T1, eh?" "Yeah, that's right." "Ok, welp, I'll be there in about... oh... 90 minutes at least." Awesome, only about two more hours, and everything will be ok. ...Right.
2:00am: Ameritech technician arrives. (90 minutes, eh?) "Yeah, look at that, no power coming down to your smart jack. ['smart jack' is basically a somewhat-smart device that terminates the telco's connection at the point of demarcation. It's used to allow them to determine whether the problem's on their end or yours.] So the problem's not here, I'll have to go check elsewhere." "Is there any point in my hanging around?" "Nah, get some sleep, I'll have you back up within the hour." ...Right.
3:00am: I go to sleep, confident that my internet services are in good hands. (You've seen the full-page SBC ads, right?)
7:30am: I wake up. *sniff* Smell's like the network's still down. I can feel it in my bones. Yeah, sure enough, it is. Call IQuest.
7:45am: Talking to IQuest, I learn that the technician checked stuff out in a manhole somewhere between us and the appropriate Ameritech CO and concludes that the problem isn't there. The tech and the people at said CO point fingers at each other, ticket gets escalated, managers get involved.
Note that at this point it's been about 11 hours since the line went down. Fortunately, Time Warner and IQuest are pulling for me here, but what can either of them really do? There's now way of them saying "we'll run our circuits over someone else's wire because you guys suck," no, the telco's decide how fast to fix problems. Argh.
9:15am: IQuest calls. "They're going around replacing hardware at the CO. That'll hopefully fix it. We'll keep you informed."
12:10pm: IQuest calls. More information. I was on the phone too long to remember any specific quote, so here's my paraphrased version: The technician, while in the manhole (remember?) apparently had gotten into an argument with the people at the CO and they just tell him to go home. Wonderful. So they're going to send another tech out to pump the water out of the manhole, which will hopefully bring the repeater back up.
Wait.
Water?
In the manhole?
Submerged electronics?
Right. Water - in the manhole. Submerged T1 components. Bits of email just floating around. Nice. Why wasn't this potential solution discussed 10 hours ago, when the first technician first went to the investigate?
12:20pm: T1 comes up briefly, goes down for a few minutes, then comes back up again. I hope they dried out the repeater really well. Maybe they wrapped it in paper towels to keep out future rainfall. Yay.
So there's my story, or at least the details of it that I can remember. Next time, I'll definitely take closer notes. Next time we're down, check the weather reports for the Indianapolis metro region - if there's rain, that's probably the cause.
John
- FreeLists Staff
~~~~~~~~ Glenn St. Louis, Mo
