Ask on mos if all the three standby are failing at the same time By the way is your primary facing the same corruption issue On 6 Jul 2011 20:35, "Zhu,Chao" <zhuchao@xxxxxxxxx> wrote: hi, guys We ran into weird corruption in standby nodes in our first production oracle/linux deployment, which is pretty frustrate; Would appreciate if anyone else has ran into similar issues and share with your solution; Working with oracle sucks, no progress in the past 2 months and now pointing fingers to HW(hp DL380G7); Which obviously is not the case as all 3 standby always consistently fails at the same block#/log#, and we tried move redo to different location/ASM/filesystem(datafile in primary still at ASM); Details: Application: Oracle RUEI on Linux(redhat 5.5) RDBMS: 11.1.0.7.4 ASM: 11.2.0.2 primary is fine; dataguard node: Tue Jul 05 18:10:00 2011 Slave exiting with ORA-752 exception Errors in file /oracle/RUEICEM/home/diag/rdbms/rueicem/RUEICEM/trace/RUEICEM_pr06_18594.trc: *ORA-00752: recovery detected a lost write of a data block* *ORA-10567: Redo is inconsistent with data block (file# 84, block# 266478)* ORA-10564: tablespace USERS ORA-01110: data file 84: '+DATA/rueicem/datafile/users.404.752266499' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 5082441 or: Errors in file /oracle/RUEICEM/home/products/diag/rdbms/rueicem/RUEICEM/trace/RUEICEM_pr0e_14272.trc (incident=101514): *ORA-00600: internal error code, arguments: [3020], [84], [266478], [352588014], [], [], [], [], [], [], [], []* *ORA-10567: Redo is inconsistent with data block (file# 84, block# 266478)* ORA-10564: tablespace USERS ORA-01110: data file 84: '+DATA/rueicem/datafile/rueicem_userruei01_35.dbf' ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 5082441 Incident details in: /oracle/RUEICEM/home/products/diag/rdbms/rueicem/RUEICEM/incident/incdir_101514/RUEICEM_pr0e_14272_i101514.trc some of our investigation done: 1. All 3 standby database failed at the same log#/block#; Including the standby with datafile/redo all on filesystem; 2. The standby database cant be recovered even set db_lost_write_protect = none; 3. The standby can still be open after it failed at that block#/log#; DBV is still clean for that datafile by that time; but no way to recover through; 4. The standby can be recovered through through “recover standby database allow 1 corruption”; But dbv did report corrupted block, and fts did report corrupted block and fts fails; Thanks for your sharing, if we dont have any progress, we will have to upgrade RDBMS to 11.2 as well (as 3rd party application, dont really want to do that.) -- Regards Zhu Chao