Re: Oracle 12c agent troubleshooting (EM_PING_NOTIF_RESPONSE: BACKOFF::180000)

Hi Martin!
Is there any info in the diagnostic directory?
$AGENT_HOME/agent_inst/diag/ofm/emagent/emagent/incident/...
Did you change from the default ping interval?

If not, can you match it to the same as your other server's?  (It should show 
in the gcagent.log)
Setting the Ping Interval

In Release 2.1, the Management Server is designed to ping all targets on a 
pre-defined interval to monitor the state of all managed targets.
To manage the interval between pings, you can use the following property in the 
omsconfig.properties file to set the ping interval:
oms.vdp.ping_interval=<integer; time in minutes>  
Note that the interval set determines the interval at which the Management 
Server tests for node up/down, regardless of what you set in the event which 
contains a node up/down test.
 
Kellyn Pot'Vin
Sr. Database Administrator and Developer
DBAKevlar.com


________________________________
From: Martin Bach <development@xxxxxxxxxxxxxxxxx>
To: "oracle-l@xxxxxxxxxxxxx" <oracle-l@xxxxxxxxxxxxx>
Sent: Thursday, October 27, 2011 8:59 AM
Subject: Oracle 12c agent troubleshooting (EM_PING_NOTIF_RESPONSE: 
BACKOFF::180000)

Good afternoon!

It's been a busy day on the mailing list, and maybe I can benefit from 
this a little :) Before I begin I have to admit that I'm not the best 
agent troubleshooter, and 12.1 hasn't made that easier.

I have 2 agents that are deployed on a 2 node cluster, both have worked 
in the past. After a reboot, both stopped to function. Now I have this:

[oracle@rac11203node1 log]$ emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.1.0
OMS Version : (unknown)
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/product/agent_inst
Agent Binaries : /u01/app/oracle/product/core/12.1.0.1.0
Agent Process ID : 13270
Parent Process ID : 13215
Agent URL : https://rac11203node1.localdomain:3872/emd/main/
Repository URL : https://oem12oms.localdomain:4901/empbs/upload
Started at : 2011-10-26 18:30:17
Started by user : oracle
Last Reload : (none)
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload : 1,858
Size of XML files pending upload(MB) : 8.05
Available disk space on upload filesystem : 49.16%
Collection Status : Collections enabled
Last attempted heartbeat to OMS : 2011-10-27 15:42:47
Last successful heartbeat to OMS : (none)

---------------------------------------------------------------
Agent is Running and Ready

The settings are correct, I have verified that with another, uploading 
and otherwise fine agent.

I have also secured the agent, and 
$AGENT_BASE/agent_inst/sysman/log/secure.log as well as the emctl secure 
agent commands reported normal, successful operation.

Still the stubborn thing doesn't want to talk to the OMS - in the agent 
overview page both agents are listed as "unavailable", but not blocked. 
When I force an upload, I get this:

[oracle@rac11203node1 log]$ emctl upload
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS 
version not checked yet. If this issue persists check trace files for 
ping to OMS related errors. (OMS_DOWN)

However it's not down, I can reach it from another agent (which happens 
to be on the same box as the OMS)

[oracle@oem12oms 12.1.0.1.0]$ $ORACLE_HOME/bin/emctl status agent
Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
Copyright (c) 1996, 2011 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.1.0
OMS Version : 12.1.0.1.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/gc12.1/agent/agent_inst
Agent Binaries : /u01/gc12.1/agent/core/12.1.0.1.0
Agent Process ID : 2964
Parent Process ID : 2910
Agent URL : https://oem12oms.localdomain:3872/emd/main/
Repository URL : https://oem12oms.localdomain:4901/empbs/upload
Started at : 2011-10-15 21:00:37
Started by user : oracle
Last Reload : (none)
Last successful upload : 2011-10-27 15:46:38
Last attempted upload : 2011-10-27 15:46:38
Total Megabytes of XML files uploaded so far : 137.79
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 50.78%
Collection Status : Collections enabled
Last attempted heartbeat to OMS : 2011-10-27 15:48:34
Last successful heartbeat to OMS : 2011-10-27 15:48:34

---------------------------------------------------------------
Agent is Running and Ready

And no, the firewall is turned off and I can connect to the upload from 
any machine in the network:

[oracle@rac11203node1 log]$ wget --no-check-certificate 
https://oem12oms.localdomain:4901/empbs/upload
--2011-10-27 15:55:46-- https://oem12oms.localdomain:4901/empbs/upload
Resolving oem12oms.localdomain... 192.168.99.28
Connecting to oem12oms.localdomain|192.168.99.28|:4901... connected.
WARNING: cannot verify oem12oms.localdomainâ??s certificate, issued by 
â??/O=EnterpriseManager on oem12oms.localdomain/OU=EnterpriseManager on 
oem12oms.localdomain/L=EnterpriseManager on 
oem12oms.localdomain/STÊ/C=US/CN=oem12oms.localdomainâ??:
Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 314 [text/html]
Saving to: â??upload.1â??

100%[======================================>] 314 --.-K/s in 0s

2011-10-27 15:55:46 (5.19 MB/s) - â??upload.1â?? saved [314/314]

The agent complains about this in gcagent.log:

2011-10-27 15:56:08,947 [37:3F09CD9C] WARN - improper ping interval 
(EM_PING_NOTIF_RESPONSE: BACKOFF::180000)
2011-10-27 15:56:18,471 [167:E3E93C4C] WARN - improper ping interval 
(EM_PING_NOTIF_RESPONSE: BACKOFF::180000)
2011-10-27 15:56:18,472 [167:E3E93C4C] WARN - Ping protocol error
o.s.gcagent.ping.PingProtocolException [OMS sent an invalid response: 
"BACKOFF::180000"]

At least someone in Oracle has some humour when it comes to this :) For 
those who read all of this: have you seen that before? Any pointers 
appreciated.

Martin
--
http://www.linkedin.com/in/martincarstenbach
http://martincarstenbach.wordpress.com
--
http://www.freelists.org/webpage/oracle-l

--
http://www.freelists.org/webpage/oracle-l


Other related posts: