Hello,
we see below error fro cluster alert.log. Cluster is running fine but, we
might see alerts. We did not see this issue before, we recently got failed
over from node 1 to node 2.
uname -a
SunOS rstnvahdmdb02 5.11 11.3 sun4v sparc sun4v
*Error seen*
2017-11-15 16:07:42.317 [SRVM(43764)]CRS-10051: CVU found following errors
with Clusterware setup : PRVF-5408 : NTP Time Server "192.5.41.41" is
common only to the following nodes "rstnvahdmdb02"
PRVF-5408 : NTP Time Server "192.5.41.209" is common only to the following
nodes "rstnvahdmdb01"
PRVF-5416 : Query of NTP daemon failed on all nodes
PRVG-1453 : Oracle CTSS resource is not in ONLINE state on nodes
"rstnvahdmdb02"
PRCW-1015 : Wallet hdmresdb does not exist.
CLSW-9: The cluster wallet to be operated on does not exist. :[1015]
PRCW-1015 : Wallet hdmresdb does not exist.
CLSW-9: The cluster wallet to be operated on does not exist. :[1015]
*related info *
[grid@rstnvahdmdb02]$ /u01/app/12.1.0/grid/bin/cluvfy comp clocksync
-noctss -n rstnvahdmdb01,rstnvahdmdb02 -verbose
Verifying Clock Synchronization across the cluster nodes
Starting Clock synchronization checks using Network Time Protocol(NTP)...
Checking existence of NTP configuration file "/etc/inet/ntp.conf" across
nodes
Node Name File exists?
------------------------------------ ------------------------
rstnvahdmdb02 yes
rstnvahdmdb01 yes
The NTP configuration file "/etc/inet/ntp.conf" is available on all nodes
NTP configuration file "/etc/inet/ntp.conf" existence check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
rstnvahdmdb02 yes
rstnvahdmdb01 yes
Result: Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
Checking whether NTP daemon or service is using UDP port 123 on all nodes
Check for NTP daemon or service using UDP port 123
Node Name Port Open?
------------------------------------ ------------------------
rstnvahdmdb02 yes
rstnvahdmdb01 yes
NTP common Time Server Check started...
NTP Time Server "192.5.41.209" is common to all nodes on which the NTP
daemon is running
Check of common NTP Time Server passed
Clock time offset check from NTP Time Server started...
Checking on nodes "[rstnvahdmdb02, rstnvahdmdb01]"...
Check: Clock time offset from NTP Time Server
Time Server: 192.5.41.209
Time Offset Limit: 1000.0 msecs
Node Name Time Offset Status
------------ ------------------------ ------------------------
rstnvahdmdb02 -9.794 passed
rstnvahdmdb01 14.599 passed
Time Server "192.5.41.209" has time offsets that are within permissible
limits for nodes "[rstnvahdmdb02, rstnvahdmdb01]".
Clock time offset check passed
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Oracle Cluster Time Synchronization Services check passed
Verification of Clock Synchronization across the cluster nodes was
successful.
From Node 1
root@rstnvahdmdb01:~# ps -ef|grep -i ctssd
root 2577 1 0 Nov 14 ? 10:37
/u01/app/12.1.0/grid/bin/octssd.bin reboot
root 52678 39847 0 17:27:01 pts/2 0:00 grep -i ctssd
root@rstnvahdmdb01:~# ps -ef|grep -i ntp
root 668 1 0 Nov 14 ? 0:09 /usr/lib/inet/ntpd
--pidfile /var/run/ntp.pid --panicgate
root 52850 39847 0 17:27:07 pts/2 0:00 grep -i ntp
From Node 2
root@rstnvahdmdb02:~# ps -ef|grep -i ntp
root 606 1 0 May 16 ? 18:41 /usr/lib/inet/ntpd
--pidfile /var/run/ntp.pid --panicgate --logfile /var/ntp/ntp.log
root 21578 57373 0 17:27:15 pts/3 0:00 grep -i ntp
root@rstnvahdmdb02:~# ps -ef|grep ctssd
root 33426 57373 0 17:56:52 pts/3 0:00 grep ctssd
from node 1
root@rstnvahdmdb01:~# /usr/sbin/ntpq -pn
remote refid st t when poll reach delay offset
jitter
==============================================================================
+206.46.245.7 192.5.41.209 2 u 446 1024 377 69.957 14.599
56.949
*206.46.245.6 *192.5.41.40* 2 u 247 1024 377 40.694 10.985
35.162
from node 2
root@rstnvahdmdb02:~# /usr/sbin/ntpq -pn
remote refid st t when poll reach delay offset
jitter
==============================================================================
*206.46.245.6 *192.5.41.41* 2 u 115 1024 377 54.859 15.140
36.879
+206.46.245.7 192.5.41.209 2 u 22 1024 377 73.357 -9.782
20.544
Node 1
ora.ctssd
1 ONLINE ONLINE rstnvahdmdb01 OBSERVER,STABLE
Node 2
ora.ctssd
1 ONLINE OFFLINE STABLE
Thanks,
Balwanth
