RE: Urgent!!

Ok

Are these ext3 filesystems?  If no, what filesystems are they? (ext2, reiser, 
xfs, jfs, Other?)

If it's a journaling filesystem, its possible that the journal got screwed up, 
but not sure how that would happen.  (Some filesystems are more susceptible to 
this than others)

Have you checked to verify any cleanup jobs might have run during the time the 
files went missing? 

Sounds terribly like something ran that might have been using wildcards to 
delete data - Are the directories missing as well did you say?

If the directories are missing, and nothing is in the log files, it might have 
been an rm -rf  on the affected filesystems.

Just sort of thinking out loud here to see if it might help.

Chris Taylor
Sr. Oracle DBA
Ingram Barge Company
Nashville, TN 37205
Office: 615-517-3355
Cell: 615-354-4799
Email: chris.taylor@xxxxxxxxxxxxxxx
 

CONFIDENTIALITY NOTICE: This e-mail and any attachments are confidential and 
may also be privileged. If you are not the named recipient, please notify the 
sender immediately and delete the contents of this message without disclosing 
the contents to anyone, using them for any purpose, or storing or copying the 
information on any medium.


-----Original Message-----
From: Bill Zakrzewski [mailto:bill@xxxxxxxxxxxx] 
Sent: Monday, April 26, 2010 9:03 AM
To: Taylor, Chris David
Cc: 'Oracle-L@xxxxxxxxxxxxx'
Subject: Re: Urgent!!

These are local drives, non-RAC, non-ASM, non-OCFS.  No information was found 
by the system administrator in the typical system log files.

I have experienced system crashes in the past and had to recover databases 
after power failures, but losing everything (and I mean everything) that wasn't 
on the OS partition/filesystem is a new one to me.

Thanks again
Bill
On Apr 26, 2010, at 9:49 AM, Taylor, Chris David wrote:

> You may want to provide more detail, unless you just want to know if anyone 
> has ever had anything like this happen before, which in my case would be no.
> 
> I assume these are SAN volumes presented to the server in question?  If so, 
> who is the SAN provider?  You'll probably want to open up a case with them as 
> well.
> 
> Are you running RAC and OCFS2 (or OCFS) filesystems?  Or non-RAC, non-ASM, 
> non-OCFS ?
> 
> What is the disk space reported on the affected mount points?  Does it match 
> up with what is supposed to be there?
> 
> Have you checked the /var/log/messages file?  (I think that's the right path 
> - been a little while since I was on RHEL)
> 
> Any filesystem cleanup utilities running that are scheduled for end of month 
> or anything?
> 
> Any cron jobs scheduled to cleanup Oracle log files or anything, especially 
> using wildcards?
> 
> It is highly unlikely that a storm would cause data to get deleted.  Chances 
> are one of the following:
> 
> 1.) SAN and Server connectivity was lost and files are still on the SAN but 
> the server doesn't see them (due to the storm)
> Or
> 2.) A filesystem utility was run that inadvertently deleted data it shouldn't 
> have
> Or
> 3.) Someone maliciously deleted files
> Or
> 4.) any number of other system specific events occurred unrelated to the storm
> 
> 
> 
> 
> Chris Taylor
> Sr. Oracle DBA
> Ingram Barge Company
> Nashville, TN 37205
> Office: 615-517-3355
> Cell: 615-354-4799
> Email: chris.taylor@xxxxxxxxxxxxxxx
> 
> 
> CONFIDENTIALITY NOTICE: This e-mail and any attachments are confidential and 
> may also be privileged. If you are not the named recipient, please notify the 
> sender immediately and delete the contents of this message without disclosing 
> the contents to anyone, using them for any purpose, or storing or copying the 
> information on any medium.
> 
> 
> -----Original Message-----
> From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx] On 
> Behalf Of Bill Zakrzewski
> Sent: Monday, April 26, 2010 8:38 AM
> To: Oracle-L@xxxxxxxxxxxxx
> Subject: Urgent!!
> 
> We had some very bad storms in the area last night and this morning our 
> database server appears to have been wiped almost clean (the server did not 
> fail or reboot - uptime was 17 days).  The oracle software is gone and the 
> database files are also no longer visible.  The server was setup with logical 
> volumes and they all appear to be empty.  Has anyone had something similar 
> happen?  Opening a ticket with Red Hat, but figured I would hit the list to 
> see if I get a quicker response.
> 
> RH 5
> Oracle 10.2.0.4.0
> 
> Thanks,
> -Bill--
> http://www.freelists.org/webpage/oracle-l
> 
> 
> 
> 
> 



--
http://www.freelists.org/webpage/oracle-l

Other related posts: