something kills my standby after successful switchover

  • From: GG <grzegorzof@xxxxxxxxxx>
  • To: oracle-l@xxxxxxxxxxxxx
  • Date: Fri, 8 Apr 2016 18:29:17 +0200

Hi,

I've got simple dataguard configuration primary on normal fs and standby Standalone GI with role separation (grid unix user for GI+ASM and oracle for oracle) .
All generally works flawlessly until second switchover where something kills my oracle instance in GI configuration .
As a workaround I can just simply srvctl start database -d orcl2  .
Below some details (11.2.0.4) .

DGMGRL> show configuration verbose

Configuration - orcl_drc

  Protection Mode: MaxPerformance
  Databases:
    orcl  - Primary database
    orcl2 - Physical standby database

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    ObserverReconnect               = '0'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
    ObserverOverride                = 'FALSE'
    ExternalDestination1            = ''
    ExternalDestination2            = ''
    PrimaryLostWriteAction          = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS



DGMGRL> show database verbose orcl ;

Database - orcl

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    orcl

  Properties:
    DGConnectIdentifier             = 'orcl'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'auto'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '+DATA/ORCL2, /u02/oradata/orcl'
LogFileNameConvert = '+DATA/ORCL2, /u02/oradata/orcl, +DATA/ORCL2, /u02/fra/orcl'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    ApplyLagThreshold               = '0'
    TransportLagThreshold           = '0'
    TransportDisconnectedThreshold  = '30'
    SidName                         = 'orcl'
StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ol67a)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=orcl_DGMGRL.localdomain)(INSTANCE_NAME=orcl)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS


DGMGRL> show database verbose orcl2 ;

Database - orcl2

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds (computed 0 seconds ago)
  Apply Lag:       0 seconds (computed 0 seconds ago)
  Apply Rate:      40.00 KByte/s
  Real Time Query: ON
  Instance(s):
    orcl
    orcl2 (apply instance)

  Properties:
    DGConnectIdentifier             = 'orcl2'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u02/oradata/orcl, +DATA/ORCL2'
LogFileNameConvert = '/u02/oradata/orcl, +DATA/ORCL2, /u02/fra/orcl, +DATA/ORCL2'
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    ApplyLagThreshold               = '0'
    TransportLagThreshold           = '0'
    TransportDisconnectedThreshold  = '30'
    SidName(*)
    StaticConnectIdentifier(*)
    StandbyArchiveLocation(*)
    AlternateLocation(*)
    LogArchiveTrace(*)
    LogArchiveFormat(*)
    TopWaitEvents(*)
    (*) - Please check specific instance for the property value

Database Status:
SUCCESS


DGMGRL> show instance verbose orcl2

Instance 'orcl2' of database 'orcl2'

  Host Name: ol67b
  PFILE:
  Properties:
    SidName                         = 'orcl2'
StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.102)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=orcl2_DGMGRL.localdomain)(INSTANCE_NAME=orcl2)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = '+FRA'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Instance Status:
SUCCESS



########## starting with switchover
DGMGRL> switchover to orcl2 ;
Performing switchover NOW, please wait...
Operation requires a connection to instance "orcl2" on database "orcl2"
Connecting to instance "orcl2"...
Connected.
New primary database "orcl2" is opening...
Operation requires startup of instance "orcl" on database "orcl"
Starting instance "orcl"...
ORACLE instance started.
Database mounted.
Database opened.
Switchover succeeded, new primary is "orcl2"


############# all good now, comming back
DGMGRL> show configuration verbose

Configuration - orcl_drc

  Protection Mode: MaxPerformance
  Databases:
    orcl2 - Primary database
    orcl  - Physical standby database

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    ObserverReconnect               = '0'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
    ObserverOverride                = 'FALSE'
    ExternalDestination1            = ''
    ExternalDestination2            = ''
    PrimaryLostWriteAction          = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS


############# switching back
DGMGRL> switchover to orcl ;
Performing switchover NOW, please wait...
Operation requires a connection to instance "orcl" on database "orcl"
Connecting to instance "orcl"...
Connected.
New primary database "orcl" is opening...
Operation requires startup of instance "orcl2" on database "orcl2"
Starting instance "orcl2"...
ORACLE instance started.
Database mounted.
Database opened.
Switchover succeeded, new primary is "orcl"




########## all good except from orcl2 (new standby) alert log shows
Fri Apr 08 18:16:49 2016
RSM0 started with pid=37, OS id=4144
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='orcl2';
ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE SID='orcl2';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET db_file_name_convert='/u02/oradata/orcl','+DATA/ORCL2' SCOPE=SPFILE;
ALTER SYSTEM SET log_file_name_convert='/u02/oradata/orcl','+DATA/ORCL2','/u02/fra/orcl','+DATA/ORCL2' SCOPE=SPFILE;
ALTER SYSTEM SET fal_server='orcl' SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (orcl2)
Fri Apr 08 18:16:53 2016
MRP0 started with pid=38, OS id=4146
MRP0: Background Managed Standby Recovery process started (orcl2)
 started logmerger process
Fri Apr 08 18:16:58 2016
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Clearing online redo logfile 1 +DATA/orcl2/redo01.log
Clearing online log 1 of thread 1 sequence number 206
Fri Apr 08 18:16:59 2016
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 +DATA/orcl2/redo02.log
Fri Apr 08 18:17:00 2016
RFS[3]: Selected log 4 for thread 1 sequence 214 dbid 1435977192 branch 908196456
Clearing online log 2 of thread 1 sequence number 212
Fri Apr 08 18:17:00 2016
Archived Log entry 167 added for thread 1 sequence 213 ID 0x559d54e8 dest 1:
Fri Apr 08 18:17:01 2016
Starting ORACLE instance (normal)
Fri Apr 08 18:17:01 2016
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.102)(PORT=1521))' SCOPE=MEMORY SID='orcl2';
ALTER DATABASE MOUNT /* db agent *//* {0:0:1371} */
ORA-1154 signalled during: ALTER DATABASE MOUNT /* db agent *//* {0:0:1371} */...
Fri Apr 08 18:17:02 2016
Shutting down instance (abort)
License high water mark = 9
USER (ospid: 4191): terminating the instance
Instance terminated by USER, pid = 4191
Fri Apr 08 18:17:02 2016
Instance shutdown complete



############ and the instance after Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
### goes down ?!
### as a remedy
srvctl start database -d orcl2

### and all good again
DGMGRL> show configuration verbose

Configuration - orcl_drc

  Protection Mode: MaxPerformance
  Databases:
    orcl  - Primary database
    orcl2 - Physical standby database

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    ObserverReconnect               = '0'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
    ObserverOverride                = 'FALSE'
    ExternalDestination1            = ''
    ExternalDestination2            = ''
    PrimaryLostWriteAction          = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS


Suspecting GI but can't prove it right now :) .
Any ideas ?
Regards
GG



--
//www.freelists.org/webpage/oracle-l


Other related posts: