[windows2000] FW: [Freelists News] 16 hours without FreeLists

  • From: "Jim Kenzig http://thethin.net" <jimkenz@xxxxxxxxxxxxxx>
  • To: <thin@xxxxxxxxxxxxx>, <windows2000@xxxxxxxxxxxxx>,<office2000@xxxxxxxxxxxxx>, <msexchange@xxxxxxxxxxxxx>
  • Date: Fri, 1 Nov 2002 15:44:46 -0500



-----Original Message-----
From: freelists-news-bounce@xxxxxxxxxxxxx
[mailto:freelists-news-bounce@xxxxxxxxxxxxx]On Behalf Of John Madden
Sent: Friday, November 01, 2002 1:03 PM
To: freelists-news@xxxxxxxxxxxxx
Subject: [Freelists News] 16 hours without FreeLists


===
- The FreeLists Newsletter -
 You're receiving this email because you are subscribed
 to the FreeLists Newsletter, a newsletter sent out to all 
 FreeLists list owners.  It contains important updates about 
 changes to FreeLists services.  For subscription information, 
 please see the bottom of this email.
 Administrative contact: weez@xxxxxxxxxxxxx
===


16 Hours without FreeLists, or, why to not give SBC any of your money.

In short, we just got through a 16-hour outage, probably the worst in our
history aside from that "lightning loves us" outage a couple years back.
It was another long night for me (although I finally did get some sleep
(unlike some previous outages :)) and I've posted my log of the saga
below.  Times are approximate: I wasn't actually writing all this down as
it happened; I also wasn't expecting the story to be so convoluted.

It just goes to show you that no matter how much money you spend
on internet services, no matter how good your ISP is, you're going to have
outages that are well beyond your control and the control of those you
contract with.  Oh yeah, and that Ameritech really sucks.  (My apologies
to anyone here who works for them, the comment isn't directed at anyone in
particular.)

10/31

8:30pm: IQuest internet services calls me.  As soon as you hear "John?
This is nnn, from the IQuest ops center," you know something's wrong.  "Oh
crap, don't even tell me the T1's down."  Uhm yeah, quite down.
Everything looks fine on our end, so I suggest he "open a ticket."

9:05pm: IQuest calls again.  "Ok, ticket's in and at Time Warner."  You
see, in this day and age, you can't say "give me a line from here to
there" unless you happen to own those lines.  Otherwise, you've got to go
through three or four separate parties to get your line, and it won't
really go where you thought it would anyway.  Whenever something goes
wrong, chances are that you'll be spending a long time passing trouble
tickets around.  Blah.

9:45pm: Time Warner calls:  "Yeah, we've got your ticket and it looks like
you're down, so we're passing this ticket onto Ameritech."  It's at this
point that I realize we're all doomed.  If your ticket goes to Ameritech,
your outage will last at least 6 hours.  It doesn't help that it's the
night shift, so personnel is thin.

11:15pm: Time Warner calls again: "Ok, it looks like Ameritech is going to
have to go out to check the T1 directly."  Great, now I have to wait for
someone to drive from western Kansas or something.

11/1

1:30am: Ameritech technician calls.  "Yep, so there's a problem with yer
T1, eh?"  "Yeah, that's right."  "Ok, welp, I'll be there in about...
oh... 90 minutes at least."  Awesome, only about two more hours, and
everything will be ok.  ...Right.

2:00am: Ameritech technician arrives.  (90 minutes, eh?)  "Yeah, look at
that, no power coming down to your smart jack. ['smart jack' is basically
a somewhat-smart device that terminates the telco's connection at the
point of demarcation.  It's used to allow them to determine whether the
problem's on their end or yours.]  So the problem's not here, I'll have to
go check elsewhere."  "Is there any point in my hanging around?"  "Nah,
get some sleep, I'll have you back up within the hour."  ...Right.

3:00am: I go to sleep, confident that my internet services are in good
hands.  (You've seen the full-page SBC ads, right?)

7:30am: I wake up.  *sniff*  Smell's like the network's still down.  I can
feel it in my bones.  Yeah, sure enough, it is.  Call IQuest.

7:45am: Talking to IQuest, I learn that the technician checked stuff out
in a manhole somewhere between us and the appropriate Ameritech CO and
concludes that the problem isn't there.  The tech and the people at said
CO point fingers at each other, ticket gets escalated, managers get
involved.

Note that at this point it's been about 11 hours since the line went down.
Fortunately, Time Warner and IQuest are pulling for me here, but what can
either of them really do?  There's now way of them saying "we'll run our
circuits over someone else's wire because you guys suck," no, the telco's
decide how fast to fix problems.  Argh.

9:15am: IQuest calls.  "They're going around replacing hardware at the CO.
That'll hopefully fix it.  We'll keep you informed."

12:10pm: IQuest calls.  More information.  I was on the phone too long to
remember any specific quote, so here's my paraphrased version:  The
technician, while in the manhole (remember?) apparently had gotten into an
argument with the people at the CO and they just tell him to go home.
Wonderful.  So they're going to send another tech out to pump the water
out of the manhole, which will hopefully bring the repeater back up.

Wait.

Water?

In the manhole?

Submerged electronics?

Right.  Water - in the manhole.  Submerged T1 components.  Bits of email
just floating around.  Nice.  Why wasn't this potential solution discussed
10 hours ago, when the first technician first went to the investigate?

12:20pm: T1 comes up briefly, goes down for a few minutes, then comes back
up again.  I hope they dried out the repeater really well.  Maybe they
wrapped it in paper towels to keep out future rainfall.  Yay.




So there's my story, or at least the details of it that I can remember.
Next time, I'll definitely take closer notes.  Next time we're down, check
the weather reports for the Indianapolis metro region - if there's rain,
that's probably the cause.

John




- FreeLists Staff



=======================
The FreeLists Newsletter

Keeping you in touch with the lastest FreeLists news
- For subscription info and list modes, please see:
http://www.freelists.org/cgi-bin/lsg2.cgi
- List webpage:
http://www.freelists.org/webpage/freelists-news
- Administrative contact:
weez@xxxxxxxxxxxxx
=======================



==================================
To Unsubscribe, set digest or vacation
mode or view archives use the below link.

http://thethin.net/win2000list.cfm

Other related posts:

  • » [windows2000] FW: [Freelists News] 16 hours without FreeLists