Re: All RAC Nodes reboot without log-entry

  • From: Dan Norris <dannorris@xxxxxxxxxxxxx>
  • To: bkaltofen@xxxxxx, oracle-l@xxxxxxxxxxxxx
  • Date: Thu, 17 Jan 2008 11:45:09 -0800 (PST)

In absence of any solid theories, I'd start by shutting down one of the nodes 
and seeing if you still experience crashes. Then I'd repeat the test with the 
other node to see if somehow one node has a problem. If both nodes crash while 
running solo, then I would say that we should put clusterware low on the 
suspect list. That is, unless it has intermittent problems accessing its OCR 
and/or voting disk(s). Then, maybe it'd have a problem. 

All things equal, if this is a new environment, I would consider reinstalling 
the whole thing to see if you can reproduce the problem. However, if the 
problem doesn't come back immediately, you'll always fear it might. Then again, 
maybe you just missed a step or a prereq or something that you may catch during 

I acknowledge that I'm violating the BAAG pledge, but I don't know that we can 
establish any test cases at this point to improve our chances of finding the 
right solution.


----- Original Message ----
From: "bkaltofen@xxxxxx" <bkaltofen@xxxxxx>
To: oracle-l@xxxxxxxxxxxxx
Sent: Thursday, January 17, 2008 5:53:37 AM
Subject: Re: All RAC Nodes reboot without log-entry


More Information and replies:

The RAC is a new setup. The Nodes with OS-only where running for 4
weeks without issue before. I installed the RAC last week.


There is no cron or something. Only me and 2 administrators have access
to the system.

dmesg will only show entries starting with the last boot. So there is
no chance do get information from the affected system.

I'm with you about booting both nodes is not a common Cluster behavior.


There is nothing seen from the OS that indicates a Interconnect issue.
I have no access to the network and power supply, as I'm remote. The
local admins say there is no issue, too.


There are ASM and DB instances, but nothing in the logs. The logs show
normal behavior till they report the startup of the instances after the

regards, Björn

Dan Norris schrieb:

my past days when I was much more mischevious than I am now (or maybe
not :), I once set a cron job to reboot a system once a day. It was on
a SGI system sitting on a desk in a classroom at Silicon Graphics'
training facility, so I didn't do it to your system, but it might be
worth changing all the passwords just for good measure. 


You might also check the "dmesg" output to see if there's something
being reported in there that didn't make it into the logfiles. I
haven't ever seen Clusterware reboot a node without logging a message
about it *somewhere* (one of the places you said you looked), so my
theory is that this isn't caused by Oracle software. Therefore, I'd
investigate something in hardware or OS. Odd that they both
reboot--that's not a common Clusterware method to resolve anything


I presume that Oracle Clusterware is the only cluster manager. if you
also have VCS or SunCluster, obviously you need to check on their logs
as well.




Original Message ----

From: "bkaltofen@xxxxxx" <bkaltofen@xxxxxx>

To: oracle-l@xxxxxxxxxxxxx

Sent: Wednesday, January 16, 2008 10:01:53 AM

Subject: All RAC Nodes reboot without log-entry




I'm facing the problem, that all my RAC nodes (2 Node Cluster 

on Solaris 10, Oracle CRS + ASM) reboot at the same time without any

entries in /var/etc/messages, /var/log/syslog and all log files under 



The cluster is a fresh installation. Any suggestions? Where can I look 



Kind regards,









Other related posts: