RE: 11.2.0.4 RAC ora.ons and ora.oc4j resources down
- From: "Stephens, Chris" <Chris.Stephens@xxxxxxx>
- To: Jared Still <jkstill@xxxxxxxxx>
- Date: Fri, 1 Nov 2013 10:20:11 -0500
Been through there. The only thing I recognize is a circular reference to
itself.
CRS-5017: The resource action "ora.ons clean" encountered the following error:
(:CLSN00009:)Utils:execCmd aborted. For details refer to "(:CLSN00106:)" in
"/u01/app/11.2.0.4/grid/log/admoract1n2/agent/crsd/oraagent_grid/oraagent_grid.log".
2013-11-01 08:34:39.847: [ora.ons][1059063552]{1:28227:4343} [clean]
(:CLSN00106:) clsn_agent::clean }
2013-11-01 08:34:39.848: [ AGFW][1061164800]{1:28227:4343} Agent sending
reply for: RESOURCE_CLEAN[ora.ons admoract1n2 1] ID 4100:16936
2013-11-01 08:34:43.802: [ora.ons][1050851072]{1:28227:4343} [clean] got lock
2013-11-01 08:34:43.802: [ora.ons][1050851072]{1:28227:4343} [clean]
tryActionLock }
2013-11-01 08:34:43.802: [ora.ons][1050851072]{1:28227:4343} [clean] abort }
2013-11-01 08:34:43.802: [ora.ons][1050851072]{1:28227:4343} [clean]
(:CLSN00110:) clsn_agent::abort }
2013-11-01 08:34:43.802: [ AGFW][1050851072]{1:28227:4343} Command: clean
for resource: ora.ons admoract1n2 1 completed with status: TIMEDOUT
2013-11-01 08:34:43.803: [ AGFW][1061164800]{1:28227:4343} Agent sending
reply for: RESOURCE_CLEAN[ora.ons admoract1n2 1] ID 4100:16936
2013-11-01 08:34:43.803: [ora.ons][1063266048]{1:28227:4343} [check]
getOracleHomeAttrib: oracle_home = /u01/app/11.2.0.4/grid
2013-11-01 08:34:43.803: [ora.ons][1063266048]{1:28227:4343} [check]
Utils:execCmd action = 3 flags = 6 ohome = /u01/app/11.2.0.4/grid/opmn/ cmdname
= onsctli.
2013-11-01 08:34:49.024: [ora.asm][1055053568]{2:24614:2} [check]
CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0
2013-11-01 08:34:49.055: [ora.asm][1055053568]{2:24614:2} [check]
AsmProxyAgent::check clsagfw_res_status 0
2013-11-01 08:34:59.707: [ora.LISTENER.lsnr][1069569792]{2:24614:2} [check]
Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl.
2013-11-01 08:34:59.810: [ora.LISTENER.lsnr][1069569792]{2:24614:2} [check]
execCmd ret = 0
2013-11-01 08:35:03.723: [ora.LISTENER_SCAN1.lsnr][1055053568]{2:24614:2}
[check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl.
2013-11-01 08:35:03.826: [ora.LISTENER_SCAN1.lsnr][1055053568]{2:24614:2}
[check] execCmd ret = 0
2013-11-01 08:35:03.827: [ora.LISTENER_SCAN1.lsnr][1055053568]{2:24614:2}
[check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1
2013-11-01 08:35:03.860: [ora.LISTENER_SCAN1.lsnr][1055053568]{2:24614:2}
[check] ScanLsnrAgent::checkDependentVipResource: scanVipResource =
ora.scan1.vip, statRet = 0
2013-11-01 08:35:19.021: [ AGFW][1061164800]{2:24614:2} Agent received the
message: AGENT_HB[Engine] ID 12293:16987
2013-11-01 08:35:43.812: [ AGENT][1050851072]{1:28227:4343} {1:28227:4343}
Created alert : (:CRSAGF00113:) : Aborting the command: check for resource:
ora.ons admoract1n2 1
2013-11-01 08:35:43.812: [ora.ons][1050851072]{1:28227:4343} [check]
(:CLSN00110:) clsn_agent::abort {
2013-11-01 08:35:43.812: [ora.ons][1050851072]{1:28227:4343} [check] abort {
2013-11-01 08:35:43.812: [ora.ons][1050851072]{1:28227:4343} [check]
Agent::abort last call info: "Agent::Agent refreshAttr"
2013-11-01 08:35:43.812: [ora.ons][1050851072]{1:28227:4343} [check] abort
command: check
2013-11-01 08:35:43.812: [ora.ons][1050851072]{1:28227:4343} [check]
tryActionLock {
2013-11-01 08:35:43.858: [ora.ons][1063266048]{1:28227:4343} [check]
(:CLSN00009:)Utils:execCmd abort request: killing
/u01/app/11.2.0.4/grid/opmn/bin/onsctli
2013-11-01 08:35:43.859: [ora.ons][1063266048]{1:28227:4343} [check]
clsnUtils::error Exception type=2 string=
CRS-5014: Agent "/u01/app/11.2.0.4/grid/bin/oraagent.bin" timed out starting
process "/u01/app/11.2.0.4/grid/opmn/bin/onsctli" for action "check": details
at "(:CLSN00009:)" in
"/u01/app/11.2.0.4/grid/log/admoract1n2/agent/crsd/oraagent_grid/oraagent_grid.log"
From: Jared Still [mailto:jkstill@xxxxxxxxx]
Sent: Friday, November 01, 2013 10:14 AM
To: Stephens, Chris
Cc: oracle-l@xxxxxxxxxxxxx
Subject: Re: 11.2.0.4 RAC ora.ons and ora.oc4j resources down
On Fri, Nov 1, 2013 at 7:41 AM, Stephens, Chris
<Chris.Stephens@xxxxxxx<mailto:Chris.Stephens@xxxxxxx>> wrote:
(:CLSN00009:)" in
"/u01/app/11.2.0.4/grid/log/admoract1n2/agent/crsd/oraagent_grid/oraagent_grid.log<http://11.2.0.4/grid/log/admoract1n2/agent/crsd/oraagent_grid/oraagent_grid.log>"
Can anyone get me started down some productive troubleshooting?
Probably it would be a good idea to examine the log file.
Jared Still
Certifiable Oracle DBA and Part Time Perl Evangelist
Oracle Blog: http://jkstill.blogspot.com
Home Page: http://jaredstill.com
CONFIDENTIALITY NOTICE:
This message is intended for the use of the individual or entity to which it is
addressed and may contain information that is privileged, confidential and
exempt from disclosure under applicable law. If the reader of this message is
not the intended recipient or the employee or agent responsible for delivering
this message to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this communication in error, please notify us
immediately by email reply.
Other related posts:
