Help with database corruption issue
- From: Steve Montgomerie <stmontgo@xxxxxxxxx>
- To: oracle-l@xxxxxxxxxxxxx
- Date: Tue, 31 Jul 2012 23:58:56 -0400
Hello List!
We had database failure recently with /var/log/messages and the Alert
Log Entries listed below at the bottom.
Very odd. We rebooted the server with 20 databases. 19 came up and
when we tried to start the 20th
database we get the following and the mount point locks up. We can't
even ls on it.
Environment is RHEL 5.8 and Oracle 11.2.0.2. SA ran fsck on boot to
check for corrupted
blocks with nothing reported. We also ran RMAN VALIDATE LOGICAL and DB
VERIFY and nothing
was reported.
I don't recall what prompted me do this but we did the following
>Shutdown database
> cp /u02/app/oracle/blah/blahblah/undotbs01.dbf
> /u02/app/oracle/blah/blahblah/temp_undotbs01.dbf
> mv /u02/app/oracle/blah/blahblah/undotbs01.dbf
> /u02/app/oracle/blah/blahblah/original_undotbs01.dbf
> mv /u02/app/oracle/blah/blahblah/temp_undotbs01.dbf
> /u02/app/oracle/blah/blahblah/undotbs01.dbf
> startup database....
> database starts up fine
Trying to figure out root cause and why the "cp and mv" fixed the
issue. Any thoughts or suggestions?
Jul 27 01:55:46 blahblah kernel: This should not happen!! Data will be lost
Jul 27 01:55:46 blahblah kernel: EXT4-fs error (device dm-50):
ext4_ext_search_left: inode #443416591: (comm oracle) ix (48312
32) != EXT_FIRST_INDEX (0) (depth 0)!
Jul 27 01:55:46 blahblah kernel: EXT4-fs (dm-50): delayed block
allocation failed for inode 443416591 at logical offset 483234
4 with max blocks 8 with error -5
Jul 27 01:55:46 blahblah kernel:
Jul 27 01:55:46 blahblah kernel: This should not happen!! Data will be lost
Jul 27 01:55:46 blahblah kernel: EXT4-fs error (device dm-50):
ext4_ext_search_left: inode #443416591: (comm oracle) ix (48312
32) != EXT_FIRST_INDEX (0) (depth 0)!
and the type of corruption reported:
Fractured block found during crash/instance recovery
Data in bad block:
type: 0 format: 0 rdba: 0x00000000
last change scn: 0x0000.00000000 seq: 0x0 flg: 0x00
spare1: 0x0 spare2: 0x0 spare3: 0x0
consistency value in tail: 0x707a0297
check value in block header: 0x0
block checksum disabled
Reading datafile '/u02/app/oracle/blah/blahblah/undotbs01.dbf' for
corruption at rdba: 0x00c946ff (file 3, block 607999)
Reread (file 3, block 607999) found same corrupt data (no logical check)
Hex dump of (file 3, block 607989) in trace file
/u04/app/oracle/diag/rdbms/blahblah/blahblah/trace/blahblah_p022_5316.trc
Corrupt block relative dba: 0x00c946f5 (file 3, block 607989)
Completely zero block found during crash/instance recovery
Appreciate an feedback you can provide.
--
//www.freelists.org/webpage/oracle-l
Other related posts:
