RE: Urgent!!
- From: "Freeman, Donald" <dofreeman@xxxxxxxxxxx>
- To: "'bill@xxxxxxxxxxxx'" <bill@xxxxxxxxxxxx>, Maria Gurenich <gurenich@xxxxxxxxx>
- Date: Wed, 28 Apr 2010 08:23:23 -0400
It's not likely that you monitor power quality, not many places do, but before
I got into Oracle I worked in electronics for 30 years. If power conditions are
monitored and recorded I'd definitely want to see it. During a thunderstorm
strange things can happen to your power and electronics don't respond to it
well. It seems very unusual for directories and files just to disappear
because of a weather event. The weather would have to affect the power, and
the power would have to affect the ...? I'm not clearly seeing the chain of
circumstances with this outcome.
Donald Freeman
Database Administrator II
Commonwealth of Pennsylvania
Department of Health
Bureau of Information Technology
2150 Herr Street
Harrisburg, PA 17103
dofreeman@xxxxxxxxxxx
-----Original Message-----
From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx] On
Behalf Of Bill Zakrzewski
Sent: Tuesday, April 27, 2010 5:19 PM
To: Maria Gurenich
Cc: Oracle-L@xxxxxxxxxxxxx
Subject: Re: Urgent!!
Maria -
The good news, we had a standby database at an alternate location and a good
backup on disk. We were unable to determine the root cause of the loss of
files - the timing coincided with a severe thunderstorm and there were no
indications of anyone logging into the server for several days leading up to
the incident. There were several power outages at the data center, however our
server had an uptime of 17 hours.
We took advantage of the timing and replaced the database server - we were
planning to replace the server during the next maintenance window.
The Security team is going to continue their investigation to make sure there
was no attack on the server.
FYI - As we investigated further the end result was close to running a find
command for all files owned by the oracle user and piping it into rm -f or rm
-Rf. There were some files and logs remaining in the agent home, but
everything else was cleaned.
-Bill
On Apr 27, 2010, at 9:24 AM, Maria Gurenich wrote:
> Hi,
>
> I just wanted to ask you, if you don't mind, to share what was it? I cannot
> suggest more, than others have already suggested, but am very itchy to know
> if you've figured the problem. It's totally fine if you don't want to tell,
> just worth asking. The problem is very weird..
>
> In any case, I wish you good luck and do not worry, it will all be OK
> eventually!! :))
>
> Cheers,
> Masha
>
> On Apr 26, 2010, at 10:09, Bill Zakrzewski <bill@xxxxxxxxxxxx> wrote:
>
>> These are local drives, non-RAC, non-ASM, non-OCFS. No information was
>> found by the system administrator in the typical system log files.
>>
>> I have experienced system crashes in the past and had to recover databases
>> after power failures, but losing everything (and I mean everything) that
>> wasn't on the OS partition/filesystem is a new one to me.
>>
>> Thanks again
>> Bill
>>>
>>>
>>> -----Original Message-----
>>> From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx]
>>> On Behalf Of Bill Zakrzewski
>>> Sent: Monday, April 26, 2010 8:38 AM
>>> To: Oracle-L@xxxxxxxxxxxxx
>>> Subject: Urgent!!
>>>
>>> We had some very bad storms in the area last night and this morning our
>>> database server appears to have been wiped almost clean (the server did not
>>> fail or reboot - uptime was 17 days). The oracle software is gone and the
>>> database files are also no longer visible. The server was setup with
>>> logical volumes and they all appear to be empty. Has anyone had something
>>> similar happen? Opening a ticket with Red Hat, but figured I would hit the
>>> list to see if I get a quicker response.
>>>
>>> RH 5
>>> Oracle 10.2.0.4.0
>>>
>>> Thanks,
>>> -Bill--
>>> //www.freelists.org/webpage/oracle-l
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> //www.freelists.org/webpage/oracle-l
>>
>>
>
--
//www.freelists.org/webpage/oracle-l
--
//www.freelists.org/webpage/oracle-l
Other related posts: