Test activation of standby database fails in Oracle 9iR2
- From: "Finn Jorgensen" <finn.oracledba@xxxxxxxxx>
- To: oracle-l <oracle-l@xxxxxxxxxxxxx>
- Date: Wed, 26 Dec 2007 11:15:58 -0500
Hello all.
The client I work for has asked me to put together a script to activate
their standby databases in case of a datacenter disaster/powerfailure. This
is due to the high number of standby's that need activation all at the same
time and the short window in which they need to be activated. Part of the
task is that the script should be able to run in a "TEST" mode in which the
standby's are activated, but the primaries are not shutdown/inaccessible
(client just wants to know IT can manage to bring all standby's up in the
allotted window, but without disrupting actual production).
All standby's are set up indentically using the LGWR process and standby
logfiles and RTA for 10g.
I pretty much have everything together and the actual failover (where the
primaries are not available) is working fine and the "TEST" mode works fine
too for Oracle 10g, because the "alter database recover standby database
finish FORCE" command gracefully kills the RFS processes. However, in 9i the
FORCE keyword doesn't exist yet and according to my research my only option
is to query the v$managed_standby view and find the "RFS" processes and kill
them at the OS level. After that bring the standby online.
As such, for 9i databases the order of events for a test are : Set
fal_server='' and fal_client='' on standby, set
log_archive_dest_state_2=DEFER on primary, kill -9 RFS processes on standby,
alter database recover standby database finish, alter database commit to
switchover to primary
However, when I do that I get the following errors in the alert log :
Fri Dec 21 13:51:54 2007
alter database recover managed standby database finish
Fri Dec 21 13:51:54 2007
Terminal Recovery: request posted
Fri Dec 21 13:52:15 2007
Warning: log 4 of thread 1 is being archived or modified
MRP0: Background Media Recovery terminated with error 261
Fri Dec 21 13:52:15 2007
Errors in file /oracle/admin/DGTEST9/bdump/dgtest9_mrp0_20366.trc:
ORA-00261: log 4 of thread 1 is being archived or modified
ORA-00312: online log 4 thread 1: '/DGTEST01/DGTEST9/standby01.log'
Recovery interrupted.
MRP0: Background Media Recovery process shutdown
Fri Dec 21 13:52:16 2007
Terminal Recovery: completion detected
Completed: alter database recover managed standby database
Fri Dec 21 13:52:16 2007
alter database commit to switchover to primary
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY
Database not recovered through End-Of-REDO
Database not recovered through End-Of-REDO
Switchover: Media recovery required - standby not in limbo
ORA-16139 signalled during: alter database commit to switchover to
primary...
I then try activating the standby using "skip standby logfile" (which would
mean data loss), but still I get an error.
Fri Dec 21 13:52:16 2007
alter database activate standby database skip standby logfile
ALTER DATABASE ACTIVATE [PHYSICAL] STANDBY DATABASE
Fri Dec 21 13:52:31 2007
Warning: log 4 of thread 1 is being archived or modified
Activate standby database received error 261
If I then (by hand) turn the MRP back on and then run through the sequence
by hand it works fine :
Fri Dec 21 13:56:04 2007
alter database recover managed standby database disconnect from session
Attempt to start background Managed Standby Recovery process
MRP0 started with pid=13
MRP0: Background Managed Standby Recovery process started
Media Recovery Waiting for thread 1 seq# 228
Fri Dec 21 13:56:10 2007
Completed: alter database recover managed standby database di
Fri Dec 21 13:57:19 2007
alter database recover managed standby database finish
Terminal Recovery: request posted
Fri Dec 21 13:57:24 2007
TERMINAL RECOVERY changing datafile format version from 8.0.0.0.0 to
9.0.0.0.0
Switching logfile format version from 8.0.0.0.0 to 9.0.0.0.0
Terminal Recovery: applying standby redo logs.
Terminal Recovery: thread 1 seq# 228 redo required
Terminal Recovery: /DGTEST01/DGTEST9/standby01.log
Identified end-of-REDO for thread 1 sequence 228
Incomplete recovery applied all redo ever generated.
Recovery completed through change 8295382144444
MRP0: Media Recovery Complete
Switching logfile format version from 9.0.0.0.0 to 8.0.0.0.0
Terminal Recovery: successful completion
Begin: Wait for standby logfiles to be archived
Fri Dec 21 13:57:25 2007
ARC0: Evaluating archive log 4 thread 1 sequence 228
Fri Dec 21 13:57:25 2007
ARC1: Evaluating archive log 4 thread 1 sequence 228
ARC1: Unable to archive log 4 thread 1 sequence 228
Log actively being archived by another process
Fri Dec 21 13:57:25 2007
ARC0: Beginning to archive log 4 thread 1 sequence 228
Creating archive destination LOG_ARCHIVE_DEST_1:
'/oracle/admin/DGTEST9/arch/DGTEST9_1_228.arc'
ARC0: Completed archiving log 4 thread 1 sequence 228
Fri Dec 21 13:57:40 2007
End: All standby logfiles have been archived
Resetting standby activation ID 799518267 (0x2fa7ae3b)
MRP0: Background Media Recovery process shutdown
Fri Dec 21 13:57:41 2007
Terminal Recovery: completion detected
Completed: alter database recover managed standby database fi
Fri Dec 21 13:57:57 2007
alter database commit to switchover to primary
Fri Dec 21 13:57:57 2007
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY
NORESETLOGS after complete recovery through change 8295382144444
Resetting resetlogs activation ID 0 (0x0)
Online log 1 of thread 1 was previously cleared
Online log 2 of thread 1 was previously cleared
Changing control file format version from 8.0.0.0.0 to 9.0.0.0.0
RESETLOGS changing datafile format version from 9.0.0.0.0 to 8.0.0.0.0
Switchover: Complete - Database shutdown required
Completed: alter database commit to switchover to primary
As such, it seems to be a timing issue with the RFS processes somehow. I
built in a sleep for 10 seconds after killing the RFS processes and tried
with 30 seconds as well, but still the same error. As stated above, if the
primary is unavailable (shutdown abort) I do not have this error.
Does anybody have any experience with this type of problem? Any ideas as to
what my problem may be? My testing has been conducted in Oracle 9.2.0.6 on
Solaris 9, but I would think it's reproducable in any version of 9iR2.
Thanks,
Finn
Other related posts:
- » Test activation of standby database fails in Oracle 9iR2
