Re: Urgent!!

  • From: Bill Zakrzewski <bill@xxxxxxxxxxxx>
  • To: Maria Gurenich <gurenich@xxxxxxxxx>
  • Date: Tue, 27 Apr 2010 17:19:17 -0400

Maria -

The good news, we had a standby database at an alternate location and a good 
backup on disk.  We were unable to determine the root cause of the loss of 
files - the timing coincided with a severe thunderstorm and there were no 
indications of anyone logging into the server for several days leading up to 
the incident.  There were several power outages at the data center, however our 
server had an uptime of 17 hours.

We took advantage of the timing and replaced the database server - we were 
planning to replace the server during the next maintenance window.

The Security team is going to continue their investigation to make sure there 
was no attack on the server.

FYI - As we investigated further the end result was close to running a find 
command for all files owned by the oracle user and piping it into rm -f or rm 
-Rf.  There were some files and logs remaining in the agent home, but 
everything else was cleaned.

On Apr 27, 2010, at 9:24 AM, Maria Gurenich wrote:

> Hi,
> I just wanted to ask you, if you don't mind, to share what was it? I cannot 
> suggest more, than others have already suggested, but am very itchy to know 
> if you've figured the problem. It's totally fine if you don't want to tell, 
> just worth asking. The problem is very weird..
> In any case, I wish you good luck and do not worry, it will all be OK 
> eventually!! :))
> Cheers,
> Masha
> On Apr 26, 2010, at 10:09, Bill Zakrzewski <bill@xxxxxxxxxxxx> wrote:
>> These are local drives, non-RAC, non-ASM, non-OCFS.  No information was 
>> found by the system administrator in the typical system log files.
>> I have experienced system crashes in the past and had to recover databases 
>> after power failures, but losing everything (and I mean everything) that 
>> wasn't on the OS partition/filesystem is a new one to me.
>> Thanks again
>> Bill
>>> -----Original Message-----
>>> From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx] 
>>> On Behalf Of Bill Zakrzewski
>>> Sent: Monday, April 26, 2010 8:38 AM
>>> To: Oracle-L@xxxxxxxxxxxxx
>>> Subject: Urgent!!
>>> We had some very bad storms in the area last night and this morning our 
>>> database server appears to have been wiped almost clean (the server did not 
>>> fail or reboot - uptime was 17 days).  The oracle software is gone and the 
>>> database files are also no longer visible.  The server was setup with 
>>> logical volumes and they all appear to be empty.  Has anyone had something 
>>> similar happen?  Opening a ticket with Red Hat, but figured I would hit the 
>>> list to see if I get a quicker response.
>>> RH 5
>>> Oracle
>>> Thanks,
>>> -Bill--
>> --


Other related posts: