you can try this:
<CRS_HOME>/crs/install/roothas.pl -deconfig -force
From: Mark J. Bobak <mark@xxxxxxxxx>
To: racattack@xxxxxxxxxxxxx
Sent: Tuesday, November 17, 2015 9:16 PM
Subject: [racattack] Re: root.sh hangs....
Ok, trying to run 'rootcrs.pl -deconfig -force' and that's apparently hanging
too.....
On Tue, Nov 17, 2015 at 7:12 PM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
Alex,Thanks for the suggestion. I remember having that problem with VB 5.0.2
and 5.0.4, but it wasn't core dumping for me on 5.0.10, but I tried that
solution anyhow. It didn't fix the problem.Seth,I'm out now. I'll try that as
soon as I get home.Thanks for all the help/feedback. I'll update when I know
more.-Mark
On Tue, Nov 17, 2015, 18:50 Alex Zaballa <zaballa@xxxxxxxxx> wrote:
Hi,
I got this problem after update ORACLE Virtual Box to 5.10!
More details here:https://blogs.oracle.com/UPGRADE/entry/oracle_virtualbox_5_0_x
Try to run $GRID_HOME/perl/bin/perl -v
If you got Segmentation fault, follow this note:
http://dbasolved.com/2015/08/24 /issue-with-perl-in-oracle_home-during-installs/
On Tue, Nov 17, 2015 at 9:19 PM, Seth Miller <sethmiller.sm@xxxxxxxxx> wrote:
I would suggest bouncing both nodes, deconfigure crs and rerun the root.sh.
On Tue, Nov 17, 2015 at 3:38 PM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
I found a trace file here:/u01/app/oracle/diag/crs/rac1/crs/trace/evmd.trc
which has the following error repeated over and over:2015-11-17 16:29:02.272847
: CRSOCR:2899277568: OCR context init failure. Error: PROC-32: Cluster
Ready Services on the local node is not running Messaging error
[gipcretConnectionRefused] [29]
Also found:/u01/app/oracle/crsdata/rac1/output/evmdOUT.trc
but the entire contents of that file is only:Redirected Oracle Clusterware
daemon standard output file
/u01/app/oracle/crsdata/rac1/output/evmdOUT.trcInitializing Oracle Clusterware
EVMD daemon with OS process ID 548 on host rac1 at local time
2015/11/17-10:03:22.912 Oracle Home is: /u01/app/12.1.0/grid Oracle Base
is: /u01/app/oracle NLS_LANG is: AMERICAN_AMERICA.AL32UTF8
Finally, I saw the evmd pid in the ouput above, tried strace on it, but strace
just hangs, tells me the process is blocked (waiting on I/O, port, or signal
from another process?), so, I took 3 pstack in a row, a few seconds apart, also
not changing, so, again indicates process is stuck waiting on
something....Here's the pstacks:[oracle@rac1 output]$ strace -p 548Process 548
attachedrestart_syscall(<... resuming interrupted call ...>^CProcess 548
detached <detached ...>[oracle@rac1 output]$ pstack 548Thread 5 (Thread
0x7f37ad0f9700 (LWP 553)):#0 0x0000003161a0ba0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1
0x00007f37af38d582 in sltspctimewait () from
/u01/app/12.1.0/grid/lib/libclntshcore.so.12.1#2 0x00007f37b2929b69 in
clsdadr_bucketThread () from /u01/app/12.1.0/grid/lib/libhasgen12.so#3
0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0#4
0x00000031616e896d in clone () from /lib64/libc.so.6Thread 4 (Thread
0x7f37acef8700 (LWP 555)):#0 0x00000031616df143 in poll () from
/lib64/libc.so.6#1 0x00007f37b2cabe60 in sgipcwWaitHelper () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#2 0x00007f37b2ca9191 in sgipcwWait ()
from /u01/app/12.1.0/grid/lib/libhasgen12.so#3 0x00007f37b2a07adf in
gipcWaitOsd () from /u01/app/12.1.0/grid/lib/libhasgen12.so#4
0x00007f37b29ede21 in gipcInternalWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#5 0x00007f37b297f655 in gipcWaitF ()
from /u01/app/12.1.0/grid/lib/libhasgen12.so#6 0x00007f37b2780466 in
clsdms_thdmai () from /u01/app/12.1.0/grid/lib/libhasgen12.so#7
0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0#8
0x00000031616e896d in clone () from /lib64/libc.so.6Thread 3 (Thread
0x7f37accf7700 (LWP 581)):#0 0x0000003161a0efbd in nanosleep () from
/lib64/libpthread.so.0#1 0x000000000045dcbc in CEvm_c::waitForOCR() ()#2
0x0000000000453f59 in CEvm_c::init() ()#3 0x0000000000440462 in Startup(void*)
()#4 0x00000000004f4137 in CLS_Threading::CLSthreadMain::cppStart(void*) ()#5
0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0#6
0x00000031616e896d in clone () from /lib64/libc.so.6Thread 2 (Thread
0x7f37acaf6700 (LWP 582)):#0 0x0000003161a0f32d in waitpid () from
/lib64/libpthread.so.0#1 0x00007f37b28de2d9 in scls_process_join () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#2 0x0000000000461567 in
restartchld(restartctx*) ()#3 0x0000003161a07a51 in start_thread () from
/lib64/libpthread.so.0#4 0x00000031616e896d in clone () from
/lib64/libc.so.6Thread 1 (Thread 0x7f37ad30e660 (LWP 548)):#0
0x00000031616df143 in poll () from /lib64/libc.so.6#1 0x00007f37b2cabe60 in
sgipcwWaitHelper () from /u01/app/12.1.0/grid/lib/libhasgen12.so#2
0x00007f37b2ca9191 in sgipcwWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#3 0x00007f37b2a07adf in gipcWaitOsd ()
from /u01/app/12.1.0/grid/lib/libhasgen12.so#4 0x00007f37b29ede21 in
gipcInternalWait () from /u01/app/12.1.0/grid/lib/libhasgen12.so#5
0x00007f37b297f655 in gipcWaitF () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#6 0x000000000051f097 in
cls::SocketSelector::select(int) ()#7 0x000000000041442d in
cls::EvmdApplication::run() ()#8 0x00000000004109b8 in evmdmain(int, unsigned
char**) ()#9 0x00007f37b28ba126 in clsbCMain () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#10 0x00000000004102a8 in main
()[oracle@rac1 output]$ [oracle@rac1 output]$ pstack 548Thread 5 (Thread
0x7f37ad0f9700 (LWP 553)):#0 0x0000003161a0ba0e in
pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0#1
0x00007f37af38d582 in sltspctimewait () from
/u01/app/12.1.0/grid/lib/libclntshcore.so.12.1#2 0x00007f37b2929b69 in
clsdadr_bucketThread () from /u01/app/12.1.0/grid/lib/libhasgen12.so#3
0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0#4
0x00000031616e896d in clone () from /lib64/libc.so.6Thread 4 (Thread
0x7f37acef8700 (LWP 555)):#0 0x00000031616df143 in poll () from
/lib64/libc.so.6#1 0x00007f37b2cabe60 in sgipcwWaitHelper () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#2 0x00007f37b2ca9191 in sgipcwWait ()
from /u01/app/12.1.0/grid/lib/libhasgen12.so#3 0x00007f37b2a07adf in
gipcWaitOsd () from /u01/app/12.1.0/grid/lib/libhasgen12.so#4
0x00007f37b29ede21 in gipcInternalWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#5 0x00007f37b297f655 in gipcWaitF ()
from /u01/app/12.1.0/grid/lib/libhasgen12.so#6 0x00007f37b2780466 in
clsdms_thdmai () from /u01/app/12.1.0/grid/lib/libhasgen12.so#7
0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0#8
0x00000031616e896d in clone () from /lib64/libc.so.6Thread 3 (Thread
0x7f37accf7700 (LWP 581)):#0 0x0000003161a0efbd in nanosleep () from
/lib64/libpthread.so.0#1 0x000000000045dcbc in CEvm_c::waitForOCR() ()#2
0x0000000000453f59 in CEvm_c::init() ()#3 0x0000000000440462 in Startup(void*)
()#4 0x00000000004f4137 in CLS_Threading::CLSthreadMain::cppStart(void*) ()#5
0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0#6
0x00000031616e896d in clone () from /lib64/libc.so.6Thread 2 (Thread
0x7f37acaf6700 (LWP 582)):#0 0x0000003161a0f32d in waitpid () from
/lib64/libpthread.so.0#1 0x00007f37b28de2d9 in scls_process_join () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#2 0x0000000000461567 in
restartchld(restartctx*) ()#3 0x0000003161a07a51 in start_thread () from
/lib64/libpthread.so.0#4 0x00000031616e896d in clone () from
/lib64/libc.so.6Thread 1 (Thread 0x7f37ad30e660 (LWP 548)):#0
0x00000031616df143 in poll () from /lib64/libc.so.6#1 0x00007f37b2cabe60 in
sgipcwWaitHelper () from /u01/app/12.1.0/grid/lib/libhasgen12.so#2
0x00007f37b2ca9191 in sgipcwWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#3 0x00007f37b2a07adf in gipcWaitOsd ()
from /u01/app/12.1.0/grid/lib/libhasgen12.so#4 0x00007f37b29ede21 in
gipcInternalWait () from /u01/app/12.1.0/grid/lib/libhasgen12.so#5
0x00007f37b297f655 in gipcWaitF () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#6 0x000000000051f097 in
cls::SocketSelector::select(int) ()#7 0x000000000041442d in
cls::EvmdApplication::run() ()#8 0x00000000004109b8 in evmdmain(int, unsigned
char**) ()#9 0x00007f37b28ba126 in clsbCMain () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#10 0x00000000004102a8 in main
()[oracle@rac1 output]$ pstack 548Thread 5 (Thread 0x7f37ad0f9700 (LWP 553)):#0
0x0000003161a0ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0#1 0x00007f37af38d582 in sltspctimewait () from
/u01/app/12.1.0/grid/lib/libclntshcore.so.12.1#2 0x00007f37b2929b69 in
clsdadr_bucketThread () from /u01/app/12.1.0/grid/lib/libhasgen12.so#3
0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0#4
0x00000031616e896d in clone () from /lib64/libc.so.6Thread 4 (Thread
0x7f37acef8700 (LWP 555)):#0 0x00000031616df143 in poll () from
/lib64/libc.so.6#1 0x00007f37b2cabe60 in sgipcwWaitHelper () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#2 0x00007f37b2ca9191 in sgipcwWait ()
from /u01/app/12.1.0/grid/lib/libhasgen12.so#3 0x00007f37b2a07adf in
gipcWaitOsd () from /u01/app/12.1.0/grid/lib/libhasgen12.so#4
0x00007f37b29ede21 in gipcInternalWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#5 0x00007f37b297f655 in gipcWaitF ()
from /u01/app/12.1.0/grid/lib/libhasgen12.so#6 0x00007f37b2780466 in
clsdms_thdmai () from /u01/app/12.1.0/grid/lib/libhasgen12.so#7
0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0#8
0x00000031616e896d in clone () from /lib64/libc.so.6Thread 3 (Thread
0x7f37accf7700 (LWP 581)):#0 0x0000003161a0efbd in nanosleep () from
/lib64/libpthread.so.0#1 0x000000000045dcbc in CEvm_c::waitForOCR() ()#2
0x0000000000453f59 in CEvm_c::init() ()#3 0x0000000000440462 in Startup(void*)
()#4 0x00000000004f4137 in CLS_Threading::CLSthreadMain::cppStart(void*) ()#5
0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0#6
0x00000031616e896d in clone () from /lib64/libc.so.6Thread 2 (Thread
0x7f37acaf6700 (LWP 582)):#0 0x0000003161a0f32d in waitpid () from
/lib64/libpthread.so.0#1 0x00007f37b28de2d9 in scls_process_join () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#2 0x0000000000461567 in
restartchld(restartctx*) ()#3 0x0000003161a07a51 in start_thread () from
/lib64/libpthread.so.0#4 0x00000031616e896d in clone () from
/lib64/libc.so.6Thread 1 (Thread 0x7f37ad30e660 (LWP 548)):#0
0x00000031616df143 in poll () from /lib64/libc.so.6#1 0x00007f37b2cabe60 in
sgipcwWaitHelper () from /u01/app/12.1.0/grid/lib/libhasgen12.so#2
0x00007f37b2ca9191 in sgipcwWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#3 0x00007f37b2a07adf in gipcWaitOsd ()
from /u01/app/12.1.0/grid/lib/libhasgen12.so#4 0x00007f37b29ede21 in
gipcInternalWait () from /u01/app/12.1.0/grid/lib/libhasgen12.so#5
0x00007f37b297f655 in gipcWaitF () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#6 0x000000000051f097 in
cls::SocketSelector::select(int) ()#7 0x000000000041442d in
cls::EvmdApplication::run() ()#8 0x00000000004109b8 in evmdmain(int, unsigned
char**) ()#9 0x00007f37b28ba126 in clsbCMain () from
/u01/app/12.1.0/grid/lib/libhasgen12.so#10 0x00000000004102a8 in main ()
On Tue, Nov 17, 2015 at 2:49 PM, Seth Miller <sethmiller.sm@xxxxxxxxx> wrote:
Does the evmd log on node 1 show any errors?
On Tue, Nov 17, 2015 at 12:24 PM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
Also, rac1 and rac2 can ping themeselves and each other on the private
addresses. (172.16.100.51,172.16.100.52)
On Tue, Nov 17, 2015 at 1:14 PM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
ok, ran gpnptol get on rac1:[oracle@rac1 bin]$ ./gpnptool getWarning: some
command line parameters were defaulted. Resulting command line:
./gpnptool.bin get -o-
<?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile Version="1.0"
xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile";
xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile";
xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile
gpnp-profile.xsd" ProfileSequence="1"
ClusterUId="b2f13eabb1847fadff49942f72181a42" ClusterName="rac-cluster-sca"
PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen"
HostName="*"><gpnp:Network id="net1" IP="192.168.78.0" Adapter="eth0"
Use="public"/><gpnp:Network id="net2" IP="172.16.100.0" Adapter="eth1"
Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile
id="css" DiscoveryString="" LeaseDuration="400"/><orcl:ASM-Profile id="asm"
DiscoveryString="++no-value-at-profile-creation--never-updated-through-ASM++"
SPFile="" Mode="legacy"/><ds:Signature
xmlns:ds="http://www.w3.org/2000/09/xmldsig#";><ds:SignedInfo><ds:CanonicalizationMethod
Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod
Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference
URI=""><ds:Transforms><ds:Transform
Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform
Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#";><InclusiveNamespaces
xmlns="http://www.w3.org/2001/10/xml-exc-c14n#"; PrefixList="gpnp orcl
xsi"/></ds:Transform></ds:Transforms><ds:DigestMethod
Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>5tZEzBB3UbfWa4jnEwxj/OVzkuQ=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>bXnVu7ARn1pPGrelGmWf/H/qx0aTDkmxHEl/8bjT+/L+hRz1HriKfu1+2xlrbViJxOQ5r1IEcO1n+jn8Ijl8Xg57DBAgCTLRUb5VV3qYJj5I6BOLUpYpz5N5Md/EB3+W/5LdboswwZ7rvMexNs1twonmLYKuQuaEoCyDf8KoDJY=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>Success.
and on rac 2:there is no gpnptool.
Not sure exactly how to parse all that, bu I see 172.16.100.0, which is the
correct network for the interconnect.
On Tue, Nov 17, 2015 at 12:44 PM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
Yep, looking at:/u01/app/oracle/diag/crs/rac1/crs/trace/alert.log
the last line is:2015-11-17 10:14:40.669 [ORAAGENT(534)]CRS-5011: Check of
resource "ora.asm" failed: details at "(:CLSN00006:)" in
"/u01/app/oracle/diag/crs/rac1/crs/trace/ohasd_oraagent_oracle.trc"
Looking at the trace file mentioned above, I see this recurring
error:2015-11-17 10:00:22.022868 : CLSDMC:2881476352: Error: gipcWait for
gipcConnect - ret_gipcreqinfo=gipcretConnectionRefused,
type_gipcreqinfo=gipcreqtypeConnect2015-11-17 10:00:22.023101
:CLSDYNAM:2881476352: [ora.gipcd]{0:0:2} [check] ClsdmClient::sendMessage
clsdmc_send error rmsg:0 ecode:-7 errbuf:2015-11-17 10:00:22.031220
:CLSDYNAM:2881476352: [ora.gipcd]{0:0:2} [check] Calling PID check for
daemon2015-11-17 10:00:22.031565 :CLSDYNAM:2881476352: [ora.gipcd]{0:0:2}
[check] Process id translated to r
2015-11-17 10:00:22.031713 : CLSDMC:2881476352: Connecting to
ipc://rac1_DBG_GIPCD2015-11-17 10:00:22.037201 :CLSDYNAM:2870970112:
[ora.evmd]{0:0:2} [check] PID FILE doesn't exist.
Not sure what that's all about.
Double checked that firewall is down and disabled, and SELinux set to disabled
as well.
On Tue, Nov 17, 2015 at 11:32 AM, Seth Miller <sethmiller.sm@xxxxxxxxx> wrote:
It's probably using the ADR then.
Check ORACLE_BASE/diag/crs/...
On Tue, Nov 17, 2015 at 10:24 AM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
Hi Seth,
On rac1, I see $GRID_HOME/log/rac1/alertrac1.log, but it's empty, 0 lines.
On rac2, the corresponding file does not exist.
On Tue, Nov 17, 2015 at 11:00 AM, Seth Miller <sethmiller.sm@xxxxxxxxx> wrote:
Can you post the grid alert log from both nodes?
Seth Miller
On Tue, Nov 17, 2015 at 9:34 AM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
Hi all,
Recently doing RAC Attack install on Virtualbox 5.0.10, using Oracle Linux 6.7
and Oracle 12.1.0.2.
No problem installing OS, running runInstaller on ASM/Grid, but when it's time
to run root.sh, everything looks fine, the clusterware and ASM instance both
seem to start fine on node 1, but the script never completes. After start of
ora.cssd, it just hangs.
Here's the log of the output...any suggestions?
Thanks,
-Mark
[oracle@rac1 app]$ sudo /u01/app/oraInventory/orainstRoot.sh Changing
permissions of /u01/app/oraInventory.Adding read,write permissions for
group.Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.The execution of the
script is complete.[oracle@rac1 app]$ sudo /u01/app/12.1.0/grid/root.sh
Performing root user operation.
The following environment variables are set as: ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/12.1.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]: The
contents of "dbhome" have not changed. No need to overwrite.The contents of
"oraenv" have not changed. No need to overwrite.The contents of "coraenv" have
not changed. No need to overwrite.
Creating /etc/oratab file...Entries will be added to the /etc/oratab file as
needed byDatabase Configuration Assistant when a database is createdFinished
running generic part of root script.Now product-specific root actions will be
performed.Using configuration parameter file:
/u01/app/12.1.0/grid/crs/install/crsconfig_params2015/11/17 09:56:09
CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2015/11/17 09:57:25 CLSRSC-4002: Successfully installed Oracle Trace File
Analyzer (TFA) Collector.
2015/11/17 09:57:27 CLSRSC-363: User ignored prerequisites during installation
OLR initialization - successful root wallet root wallet cert root cert
export peer wallet profile reader wallet pa wallet peer wallet keys pa
wallet keys peer cert request pa cert request peer cert pa cert peer root
cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer
cert TP profile reader pa cert TP profile reader peer cert TP peer user cert
pa user cert2015/11/17 09:59:30 CLSRSC-330: Adding Clusterware entries to file
'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.CRS-4123: Oracle
High Availability Services has been started.CRS-4133: Oracle High Availability
Services has been stopped.CRS-4123: Oracle High Availability Services has been
started.CRS-2672: Attempting to start 'ora.evmd' on 'rac1'CRS-2672: Attempting
to start 'ora.mdnsd' on 'rac1'CRS-2676: Start of 'ora.mdnsd' on 'rac1'
succeededCRS-2676: Start of 'ora.evmd' on 'rac1' succeededCRS-2672: Attempting
to start 'ora.gpnpd' on 'rac1'CRS-2676: Start of 'ora.gpnpd' on 'rac1'
succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'CRS-2672:
Attempting to start 'ora.gipcd' on 'rac1'CRS-2676: Start of 'ora.cssdmonitor'
on 'rac1' succeededCRS-2676: Start of 'ora.gipcd' on 'rac1' succeededCRS-2672:
Attempting to start 'ora.cssd' on 'rac1'CRS-2672: Attempting to start
'ora.diskmon' on 'rac1'CRS-2676: Start of 'ora.diskmon' on 'rac1'
succeededCRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
