Re: Controlfiles just got overwritten
- From: Justin Mungal <justin@xxxxxxx>
- To: Tim Gorman <tim@xxxxxxxxx>
- Date: Fri, 18 Jul 2014 15:59:57 -0500
Amen to honesty... everyone has screwed up; just own it and work as a team
to fix the problem. Deception or obfuscation of mistakes is a choice
however, and something I could never look past.
On Fri, Jul 18, 2014 at 3:51 PM, Tim Gorman <tim@xxxxxxxxx> wrote:
> What worked is honesty.
>
> I think what really scares people is the "mysterious incident", where
> somehow, *something* bad happened and nobody knows how or why. It's the IT
> corollary to a street murder with plenty of bystanders but no witnesses.
>
> Another personal vignette...
>
> A little over 10 years ago, I was working in downtown LA and arrived in
> the office early (5 am) to start a batch job. I had a card-key which got
> me into the building and into the office during the day, but at night there
> were locking doors in the elevator lobby of which I was previously unaware.
> I banged on the doors, tried calling people, to no avail. Finally, after
> a half hour, out of frustration, I grabbed one of the door handles and just
> YANKED HARD.
>
> It popped open.
>
> I looked at it in surprise, thought "sweet!", walked in to the cubicle
> farm, sat down, and started my batch job. All was good.
>
> Around 7 am, the LAPD showed up. There were about a dozen people in the
> office now, so the two officers began questioning folks nearest the door.
> From the opposite side of the room, I stood up and called out, "Over
> here". I 'fessed up.
>
> They told me that, if I hadn't called them over immediately, they would
> have arrested me by the time they got to me. Have a nice day, sir.
>
> As Chris Rock said (after the Rodney King incident): "If the police have
> to chase you, they'll bring a whupping with them." That works for the
> every other part of the world, too...
>
>
>
>
>
>
> On 7/18/14, 12:26, Maureen English wrote:
>
>> Tim, rest of list,
>>
>> THANK YOU!
>>
>> I was ready to start cleaning out my desk after realizing what I did. My
>> manager, though, was concerned about what
>> to tell the users and letting our director know what had happened. Our
>> director, someone I've worked with for almost
>> 20 years, always remains calm and offered some suggestions for quick
>> fixes since he was a DBA years ago. Our CITO,
>> someone I expected to be very upset, responded with something like
>> 'Mistakes happen, you're only human.'
>>
>> - Maureen
>>
>>
>> On 7/17/2014 3:28 PM, Tim Gorman wrote:
>>
>>> Maureen,
>>>
>>> About 4 years ago, or 26 years into my IT career, I dropped an index on
>>> a 60 Tb table with 24,000 hourly partitions;
>>> the index was over 15 Tb in size. It was the main table in that
>>> production application, of course.
>>>
>>> Over a quarter century of industry experience as a developer, production
>>> support, systems administrator, and database
>>> administrator; if that's not enough time to have important lessons
>>> pounded into one's head, then how much time is needed?
>>>
>>> My supervisor at the time was amazing. After the shock of watching it
>>> all happen and still not quite believing it had
>>> happened, I called him at about 9pm local time, and told him what
>>> occurred. I finished speaking, and waited for the axe
>>> to drop, for the entirely-justified anger to crash down on my head. He
>>> was silent for about 3 seconds, then just said
>>> calmly, "Well, I guess we need to fix it." And that was it. No anger,
>>> no recriminations, no humiliating
>>> micro-management. We launched straight into planning what needed to
>>> happen to fix it.
>>>
>>> He got to work notifying the organization what had happened, and I got
>>> started on the rebuild, which eventually took
>>> almost 2 weeks to complete.
>>>
>>> I hope you had the same experience. Because it truly happens to all of
>>> us. And anyone who pretends otherwise simply
>>> hasn't been doing important work.
>>>
>>> Hope this helps...
>>>
>>> -Tim
>>>
>>>
>>>
>>>
>>> On 7/17/14, 17:01, Maureen English wrote:
>>>
>>>> I ended up recovering the database from a backup done the day before
>>>> and rolling forward.
>>>>
>>>> I also modified my instructions for recreating a standby database.
>>>> Instead of 'cp -p' to copy
>>>> my standby control file to the appropriate directory on the standby
>>>> server, I will now use 'cp -pi'.
>>>>
>>>> I'm still feeling badly for making this mistake, but I'm amazed at how
>>>> understanding my coworkers
>>>> have been.
>>>>
>>>> - Maureen
>>>>
>>>>
>>
>>
> --
> //www.freelists.org/webpage/oracle-l
>
>
>
Other related posts:
