RE: OT moment of doubt

  • From: Muqthar Ahmed <Muqthar.Ahmed@xxxxxxxxxxxxxxxxx>
  • To: "'mcdonald.connor@xxxxxxxxx'" <mcdonald.connor@xxxxxxxxx>, <oracle-l@xxxxxxxxxxxxx>
  • Date: Thu, 16 Jun 2005 23:14:23 -0400



-----Original Message-----
From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx] On 
Behalf Of Connor McDonald
Sent: Wednesday, June 15, 2005 9:14 PM
To: oracle-l@xxxxxxxxxxxxx
Subject: Re: OT moment of doubt

My own personal hell story was moving database from one server to
another, but we were recycling some of the hardware (disks and
memory), so it was a unload-to-tape, reload-from-tape job.

The servers were in Port Hedland (a rat-infested dusty
110-degrees-plus 98% humidty hell hole... Hi to anyone in Port Hedland
... ).  I was not physically present - don't you love it when IT
companies think "remote login" is always the best way...Anyway,
because of the dust issues, I unloaded three copies of the database to
three tapes, and then did a verification read of all them.  An on-site
non-techie took care of unloading/reloading tapes for me.

He's chucks the disk drives and memory from the old server plus the
three tapes into a truck and drive 20 km's across town (where the new
server is). Then I spend an hour on the phone trying to explain blind
to him how to plug all this stuff in to the new server.

He turns it on... smoke starts billowing out of the box.  Emergency
shutdown (ie, rip out power cord).  He's plugged some of the memory in
wrong so its munched one of the boards.  So (over the phone again) we
talk through removing all the memory from just that board but leaving
the other memory in.

Finally, the machine does in fact boot nicely.  I log in as root and
mount the new disks and start restoring from tape.

30mins in... first tape dies with a read error.  About half the
database restored.  "No problem" I think... we load the second tape.

Another 30 mins, second tapes dies with a read error.  Everything
restored except SYSTEM tablespace.  "Luckily we've got that third
tape..." I think.

Third tape goes in... 'ufsrestore' command just hangs... Phone rings.=20
On-site techie says "I can see tape spewing out of the Exabyte

So all tapes used, and no SYSTEM tablespace... As they say in the classics,=

"Thats when you discover that adrenalin is brown and sticky".

I'm thinking about career moves etc, when the phone rings again.  Its my wi=

"How's thing going at work?" she asks.

"Don't ask!..." I say and give her the run-down on what's
happened..."Anyway, how's things at home?"

There's a long pause... followed by her bursting into tears....

"My pet budgerigar got out of the cage and has flown away....(sob)
(sob)... he's in the tree in the back yard...(sob) (sob) can you
please come home and try catch him..."

Not really what I needed at that point in time...

So I tell the techie I'm off to get a coffee... I jump in the car and
scoot home to console the wife..."Where is the little bird?"

"That tree there..." says my wife as she points to our 100 year old
citradora, around 150ft high.  With the binoculars I can just see the
little yellow budgie about 149.5ft up...

wife - "Can you climb up and get him?" =20

me (mentally) - "Are you out of your frickin mind ?!?!?!"
me (verbally) - "Of course dear, fetch me the ladder..."

So I start climbing this stupid tree, knowing full well that I'll
never get this stupid bird, and that even if I did, I would wring its
stupid little neck....30 mins later, I'm teetering at about 80ft and
the branches won't support my weight anymore.  So I climb back down,
covered in sweat and scratches, but the effort seems to have appeased
my wife...

So I spend another 10 mins crapping on about how "its better now that
the the little bird is free" etc etc, knowing full well the
neighbour's cat will have him for dinner within a matter of hours.

I scoot back to work...its amazing how teetering on the brink at 80ft
gives you some clarity.  I have a sneaking suspicion that the SYSTEM
tablespace was on one of the disks that was NOT recycled from the old
server.  I get the techie to drive back to the old server, get all the
remaining disks.

Another hour to get them all plugged in correctly (remember the good
old days of SCSI terminators....) and voila!  I find the SYSTEM
tablespace on the old disks.

So at the end of all this drama, I naturally send my status email to manage=

"Server moved, no errors encountered"



On 6/16/05, Mladen Gogala <mgogala@xxxxxxxxxxxxxxxxxxxx> wrote:
> Carel-Jan Engel wrote:
> >Jareds method is what I use most of the times.
> >When I'm really unsure, I run a small loop like (Apologies Jared, this
> >still isn't perl):
> >for file in *.dbf
> >do
> >   echo mv ${file} ${file}.GONE
> >done
> >
> >
> >

Connor McDonald
email: connor_mcdonald@xxxxxxxxx

"Semper in excremento, sole profundum qui variat"

* * * * * * * * *

The information contained in this E-mail message is privileged, confidential, 
and may be protected from disclosure; please be aware that any other use, 
printing, copying, disclosure or dissemination 
of this communication may be subject to legal restriction or sanction. If you 
think that you have received this E-mail message in error, please reply to the 
sender and delete it from your computer. T
hank you.

Other related posts: