Re: OT - Getting fired for database oops

  • From: Riyaj Shamsudeen <riyaj.shamsudeen@xxxxxxxxx>
  • To: Mark.Bobak@xxxxxxxxxxxx
  • Date: Sun, 17 May 2009 23:47:25 -0500

Of course Mark :-)

I remember that dude. Decent guy, but just plain arrogant and careless. I
think, he made few other minor mistakes (like mapping luns incorrectly and
arguing about it) before hitting this big one.

Worse thing was that, after realizing that he resilvered from BCVs to Live
database, he tried to correct and hide the problem: He reversed the
direction and resilvered from live database to BCV (like somehow that will
magically correct it!) and corrupting BCVs too.

If he had stopped at first mistake and called us, we would have been able to
recover database from BCVs with zero data loss.[ Both archivelog files and
online redo log files were kept out of BCVs ]. After that second resilver,
only tape recovery was possible, of course, you know how painful that was
:-)

I am sure, there are few lessions learnt here:-)

Cheers
Riyaj



On Sun, May 17, 2009 at 10:04 AM, Bobak, Mark <Mark.Bobak@xxxxxxxxxxxx>wrote:

> We had a storage admin get fired for crashing the production database.  It
> was a system using EMC array, with BCVs for backups.
> He decided it would a good idea to re-silver the live disks from the BCVs,
> in the middle of the day, with the production system running.
> So, we have hosed primaries AND hosed BCVs.  Only option was recover from
> tape.....many hours later....we were back online.
>
> The storage admin was a contractor, so, it wasn't as difficult as firing an
> employee.  We just told the contract house we didn't want to see him
> again...;-)
>
> -Mark
>
> PS  If Riyaj is out there, I'm sure he remembers this episode.... :-)
> ________________________________________
> From: oracle-l-bounce@xxxxxxxxxxxxx [oracle-l-bounce@xxxxxxxxxxxxx] On
> Behalf Of LS Cheng [exriscer@xxxxxxxxx]
> Sent: Sunday, May 17, 2009 5:58
> To: cjnewman@xxxxxxxxxxxxx
> Cc: oracle-l@xxxxxxxxxxxxx
> Subject: Re: OT - Getting fired for database oops
>
> I met a guy who was working as dba onsite for one of largest telco company
> in the world, he lasted 3 weeks .
>
> This is the story him and his old collegues tole me:
>
> "I accidently deleted 12 production databases because I was doing find .
> rm"
>
> The customer director called his boss and said he doesnt want to see him
> again in his life.
>
>
>
>
> --
> LSC
>
>
> On Fri, May 15, 2009 at 5:39 PM, Newman, Christopher <
> cjnewman@xxxxxxxxxxxxx<mailto:cjnewman@xxxxxxxxxxxxx>> wrote:
> A colleague and I were just discussing DBA's getting fired for screwing
> something up/failing to do something (Not for being generally poor
> employees like showing up late, etc).  Neither of us had heard any
> stories of either scenario.  I know we sometimes have slightly off topic
> discussions on the list, so I was wondering if anyone would share any
> stories.
>
> Chris Newman
> Database Specialist
> AITS, University of Illinois
> 217-333-5429
>
> --
> //www.freelists.org/webpage/oracle-l
>
>
>
>
> --
> //www.freelists.org/webpage/oracle-l
>
>
>

Other related posts:

  1. Home
  2. oracle-l
  3. Archive
  4. 05-2009