Re: Controlfiles just got overwritten
- From: Tim Gorman <tim@xxxxxxxxx>
- To: Maureen English <maureen.english@xxxxxxxxxx>, oracle-l@xxxxxxxxxxxxx
- Date: Fri, 18 Jul 2014 14:51:27 -0600
What worked is honesty.
I think what really scares people is the "mysterious incident", where
somehow, *something* bad happened and nobody knows how or why. It's the
IT corollary to a street murder with plenty of bystanders but no witnesses.
Another personal vignette...
A little over 10 years ago, I was working in downtown LA and arrived in
the office early (5 am) to start a batch job. I had a card-key which
got me into the building and into the office during the day, but at
night there were locking doors in the elevator lobby of which I was
previously unaware. I banged on the doors, tried calling people, to no
avail. Finally, after a half hour, out of frustration, I grabbed one of
the door handles and just YANKED HARD.
It popped open.
I looked at it in surprise, thought "sweet!", walked in to the cubicle
farm, sat down, and started my batch job. All was good.
Around 7 am, the LAPD showed up. There were about a dozen people in the
office now, so the two officers began questioning folks nearest the
door. From the opposite side of the room, I stood up and called out,
"Over here". I 'fessed up.
They told me that, if I hadn't called them over immediately, they would
have arrested me by the time they got to me. Have a nice day, sir.
As Chris Rock said (after the Rodney King incident): "If the police
have to chase you, they'll bring a whupping with them." That works for
the every other part of the world, too...
On 7/18/14, 12:26, Maureen English wrote:
Tim, rest of list,
THANK YOU!
I was ready to start cleaning out my desk after realizing what I did.
My manager, though, was concerned about what
to tell the users and letting our director know what had happened.
Our director, someone I've worked with for almost
20 years, always remains calm and offered some suggestions for quick
fixes since he was a DBA years ago. Our CITO,
someone I expected to be very upset, responded with something like
'Mistakes happen, you're only human.'
- Maureen
On 7/17/2014 3:28 PM, Tim Gorman wrote:
Maureen,
About 4 years ago, or 26 years into my IT career, I dropped an index
on a 60 Tb table with 24,000 hourly partitions;
the index was over 15 Tb in size. It was the main table in that
production application, of course.
Over a quarter century of industry experience as a developer,
production support, systems administrator, and database
administrator; if that's not enough time to have important lessons
pounded into one's head, then how much time is needed?
My supervisor at the time was amazing. After the shock of watching
it all happen and still not quite believing it had
happened, I called him at about 9pm local time, and told him what
occurred. I finished speaking, and waited for the axe
to drop, for the entirely-justified anger to crash down on my head.
He was silent for about 3 seconds, then just said
calmly, "Well, I guess we need to fix it." And that was it. No
anger, no recriminations, no humiliating
micro-management. We launched straight into planning what needed to
happen to fix it.
He got to work notifying the organization what had happened, and I
got started on the rebuild, which eventually took
almost 2 weeks to complete.
I hope you had the same experience. Because it truly happens to all
of us. And anyone who pretends otherwise simply
hasn't been doing important work.
Hope this helps...
-Tim
On 7/17/14, 17:01, Maureen English wrote:
I ended up recovering the database from a backup done the day before
and rolling forward.
I also modified my instructions for recreating a standby database.
Instead of 'cp -p' to copy
my standby control file to the appropriate directory on the standby
server, I will now use 'cp -pi'.
I'm still feeling badly for making this mistake, but I'm amazed at
how understanding my coworkers
have been.
- Maureen
--
//www.freelists.org/webpage/oracle-l
Other related posts:
