Re: Urgent!!
- From: Andrew Kerber <andrew.kerber@xxxxxxxxx>
- To: ChrisDavid.Taylor@xxxxxxxxxxxxxxx
- Date: Mon, 26 Apr 2010 09:19:23 -0500
Local drives, and no system crash sounds like user error to me. Have you
logged in as a privileges user and verified that some clown didn't just
change the privileges on the files? I assume the archived logs are gone
also?
On Mon, Apr 26, 2010 at 9:09 AM, Taylor, Chris David <
ChrisDavid.Taylor@xxxxxxxxxxxxxxx> wrote:
> Ok
>
> Are these ext3 filesystems? If no, what filesystems are they? (ext2,
> reiser, xfs, jfs, Other?)
>
> If it's a journaling filesystem, its possible that the journal got screwed
> up, but not sure how that would happen. (Some filesystems are more
> susceptible to this than others)
>
> Have you checked to verify any cleanup jobs might have run during the time
> the files went missing?
>
> Sounds terribly like something ran that might have been using wildcards to
> delete data - Are the directories missing as well did you say?
>
> If the directories are missing, and nothing is in the log files, it might
> have been an rm -rf on the affected filesystems.
>
> Just sort of thinking out loud here to see if it might help.
>
> Chris Taylor
> Sr. Oracle DBA
> Ingram Barge Company
> Nashville, TN 37205
> Office: 615-517-3355
> Cell: 615-354-4799
> Email: chris.taylor@xxxxxxxxxxxxxxx
>
>
> CONFIDENTIALITY NOTICE: This e-mail and any attachments are confidential
> and may also be privileged. If you are not the named recipient, please
> notify the sender immediately and delete the contents of this message
> without disclosing the contents to anyone, using them for any purpose, or
> storing or copying the information on any medium.
>
>
> -----Original Message-----
> From: Bill Zakrzewski [mailto:bill@xxxxxxxxxxxx]
> Sent: Monday, April 26, 2010 9:03 AM
> To: Taylor, Chris David
> Cc: 'Oracle-L@xxxxxxxxxxxxx'
> Subject: Re: Urgent!!
>
> These are local drives, non-RAC, non-ASM, non-OCFS. No information was
> found by the system administrator in the typical system log files.
>
> I have experienced system crashes in the past and had to recover databases
> after power failures, but losing everything (and I mean everything) that
> wasn't on the OS partition/filesystem is a new one to me.
>
> Thanks again
> Bill
> On Apr 26, 2010, at 9:49 AM, Taylor, Chris David wrote:
>
> > You may want to provide more detail, unless you just want to know if
> anyone has ever had anything like this happen before, which in my case would
> be no.
> >
> > I assume these are SAN volumes presented to the server in question? If
> so, who is the SAN provider? You'll probably want to open up a case with
> them as well.
> >
> > Are you running RAC and OCFS2 (or OCFS) filesystems? Or non-RAC,
> non-ASM, non-OCFS ?
> >
> > What is the disk space reported on the affected mount points? Does it
> match up with what is supposed to be there?
> >
> > Have you checked the /var/log/messages file? (I think that's the right
> path - been a little while since I was on RHEL)
> >
> > Any filesystem cleanup utilities running that are scheduled for end of
> month or anything?
> >
> > Any cron jobs scheduled to cleanup Oracle log files or anything,
> especially using wildcards?
> >
> > It is highly unlikely that a storm would cause data to get deleted.
> Chances are one of the following:
> >
> > 1.) SAN and Server connectivity was lost and files are still on the SAN
> but the server doesn't see them (due to the storm)
> > Or
> > 2.) A filesystem utility was run that inadvertently deleted data it
> shouldn't have
> > Or
> > 3.) Someone maliciously deleted files
> > Or
> > 4.) any number of other system specific events occurred unrelated to the
> storm
> >
> >
> >
> >
> > Chris Taylor
> > Sr. Oracle DBA
> > Ingram Barge Company
> > Nashville, TN 37205
> > Office: 615-517-3355
> > Cell: 615-354-4799
> > Email: chris.taylor@xxxxxxxxxxxxxxx
> >
> >
> > CONFIDENTIALITY NOTICE: This e-mail and any attachments are confidential
> and may also be privileged. If you are not the named recipient, please
> notify the sender immediately and delete the contents of this message
> without disclosing the contents to anyone, using them for any purpose, or
> storing or copying the information on any medium.
> >
> >
> > -----Original Message-----
> > From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:
> oracle-l-bounce@xxxxxxxxxxxxx] On Behalf Of Bill Zakrzewski
> > Sent: Monday, April 26, 2010 8:38 AM
> > To: Oracle-L@xxxxxxxxxxxxx
> > Subject: Urgent!!
> >
> > We had some very bad storms in the area last night and this morning our
> database server appears to have been wiped almost clean (the server did not
> fail or reboot - uptime was 17 days). The oracle software is gone and the
> database files are also no longer visible. The server was setup with
> logical volumes and they all appear to be empty. Has anyone had something
> similar happen? Opening a ticket with Red Hat, but figured I would hit the
> list to see if I get a quicker response.
> >
> > RH 5
> > Oracle 10.2.0.4.0
> >
> > Thanks,
> > -Bill--
> > //www.freelists.org/webpage/oracle-l
> >
> >
> >
> >
> >
>
>
>
> --
> //www.freelists.org/webpage/oracle-l
>
>
>
--
Andrew W. Kerber
'If at first you dont succeed, dont take up skydiving.'
Other related posts:
