Alex,
Thanks for the suggestion. I remember having that problem with VB 5.0.2
and 5.0.4, but it wasn't core dumping for me on 5.0.10, but I tried that
solution anyhow. It didn't fix the problem.
Seth,
I'm out now. I'll try that as soon as I get home.
Thanks for all the help/feedback. I'll update when I know more.
-Mark
On Tue, Nov 17, 2015, 18:50 Alex Zaballa <zaballa@xxxxxxxxx> wrote:
Hi,
I got this problem after update ORACLE Virtual Box to 5.10!
More details here:
https://blogs.oracle.com/UPGRADE/entry/oracle_virtualbox_5_0_x
Try to run $GRID_HOME/perl/bin/perl -v
If you got Segmentation fault, follow this note:
http://dbasolved.com/2015/08/24/issue-with-perl-in-oracle_home-during-installs/
On Tue, Nov 17, 2015 at 9:19 PM, Seth Miller <sethmiller.sm@xxxxxxxxx>
wrote:
I would suggest bouncing both nodes, deconfigure crs and rerun the
root.sh.
On Tue, Nov 17, 2015 at 3:38 PM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
I found a trace file here:
/u01/app/oracle/diag/crs/rac1/crs/trace/evmd.trc
which has the following error repeated over and over:
2015-11-17 16:29:02.272847 : CRSOCR:2899277568: OCR context init
failure. Error: PROC-32: Cluster Ready Services on the local node is not
running Messaging error [gipcretConnectionRefused] [29]
Also found:
/u01/app/oracle/crsdata/rac1/output/evmdOUT.trc
but the entire contents of that file is only:
Redirected Oracle Clusterware daemon standard output file
/u01/app/oracle/crsdata/rac1/output/evmdOUT.trc
Initializing Oracle Clusterware EVMD daemon with OS process ID 548 on
host rac1 at local time 2015/11/17-10:03:22.912
Oracle Home is: /u01/app/12.1.0/grid
Oracle Base is: /u01/app/oracle
NLS_LANG is: AMERICAN_AMERICA.AL32UTF8
Finally, I saw the evmd pid in the ouput above, tried strace on it, but
strace just hangs, tells me the process is blocked (waiting on I/O, port,
or signal from another process?), so, I took 3 pstack in a row, a few
seconds apart, also not changing, so, again indicates process is stuck
waiting on something....
Here's the pstacks:
[oracle@rac1 output]$ strace -p 548
Process 548 attached
restart_syscall(<... resuming interrupted call ...>^CProcess 548 detached
<detached ...>
[oracle@rac1 output]$ pstack 548
Thread 5 (Thread 0x7f37ad0f9700 (LWP 553)):
#0 0x0000003161a0ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007f37af38d582 in sltspctimewait () from
/u01/app/12.1.0/grid/lib/libclntshcore.so.12.1
#2 0x00007f37b2929b69 in clsdadr_bucketThread () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#3 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#4 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f37acef8700 (LWP 555)):
#0 0x00000031616df143 in poll () from /lib64/libc.so.6
#1 0x00007f37b2cabe60 in sgipcwWaitHelper () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#2 0x00007f37b2ca9191 in sgipcwWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#3 0x00007f37b2a07adf in gipcWaitOsd () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#4 0x00007f37b29ede21 in gipcInternalWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#5 0x00007f37b297f655 in gipcWaitF () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#6 0x00007f37b2780466 in clsdms_thdmai () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#7 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#8 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f37accf7700 (LWP 581)):
#0 0x0000003161a0efbd in nanosleep () from /lib64/libpthread.so.0
#1 0x000000000045dcbc in CEvm_c::waitForOCR() ()
#2 0x0000000000453f59 in CEvm_c::init() ()
#3 0x0000000000440462 in Startup(void*) ()
#4 0x00000000004f4137 in CLS_Threading::CLSthreadMain::cppStart(void*)
()
#5 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#6 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f37acaf6700 (LWP 582)):
#0 0x0000003161a0f32d in waitpid () from /lib64/libpthread.so.0
#1 0x00007f37b28de2d9 in scls_process_join () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#2 0x0000000000461567 in restartchld(restartctx*) ()
#3 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#4 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f37ad30e660 (LWP 548)):
#0 0x00000031616df143 in poll () from /lib64/libc.so.6
#1 0x00007f37b2cabe60 in sgipcwWaitHelper () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#2 0x00007f37b2ca9191 in sgipcwWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#3 0x00007f37b2a07adf in gipcWaitOsd () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#4 0x00007f37b29ede21 in gipcInternalWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#5 0x00007f37b297f655 in gipcWaitF () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#6 0x000000000051f097 in cls::SocketSelector::select(int) ()
#7 0x000000000041442d in cls::EvmdApplication::run() ()
#8 0x00000000004109b8 in evmdmain(int, unsigned char**) ()
#9 0x00007f37b28ba126 in clsbCMain () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#10 0x00000000004102a8 in main ()
[oracle@rac1 output]$
[oracle@rac1 output]$ pstack 548
Thread 5 (Thread 0x7f37ad0f9700 (LWP 553)):
#0 0x0000003161a0ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007f37af38d582 in sltspctimewait () from
/u01/app/12.1.0/grid/lib/libclntshcore.so.12.1
#2 0x00007f37b2929b69 in clsdadr_bucketThread () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#3 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#4 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f37acef8700 (LWP 555)):
#0 0x00000031616df143 in poll () from /lib64/libc.so.6
#1 0x00007f37b2cabe60 in sgipcwWaitHelper () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#2 0x00007f37b2ca9191 in sgipcwWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#3 0x00007f37b2a07adf in gipcWaitOsd () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#4 0x00007f37b29ede21 in gipcInternalWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#5 0x00007f37b297f655 in gipcWaitF () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#6 0x00007f37b2780466 in clsdms_thdmai () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#7 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#8 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f37accf7700 (LWP 581)):
#0 0x0000003161a0efbd in nanosleep () from /lib64/libpthread.so.0
#1 0x000000000045dcbc in CEvm_c::waitForOCR() ()
#2 0x0000000000453f59 in CEvm_c::init() ()
#3 0x0000000000440462 in Startup(void*) ()
#4 0x00000000004f4137 in CLS_Threading::CLSthreadMain::cppStart(void*)
()
#5 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#6 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f37acaf6700 (LWP 582)):
#0 0x0000003161a0f32d in waitpid () from /lib64/libpthread.so.0
#1 0x00007f37b28de2d9 in scls_process_join () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#2 0x0000000000461567 in restartchld(restartctx*) ()
#3 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#4 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f37ad30e660 (LWP 548)):
#0 0x00000031616df143 in poll () from /lib64/libc.so.6
#1 0x00007f37b2cabe60 in sgipcwWaitHelper () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#2 0x00007f37b2ca9191 in sgipcwWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#3 0x00007f37b2a07adf in gipcWaitOsd () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#4 0x00007f37b29ede21 in gipcInternalWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#5 0x00007f37b297f655 in gipcWaitF () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#6 0x000000000051f097 in cls::SocketSelector::select(int) ()
#7 0x000000000041442d in cls::EvmdApplication::run() ()
#8 0x00000000004109b8 in evmdmain(int, unsigned char**) ()
#9 0x00007f37b28ba126 in clsbCMain () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#10 0x00000000004102a8 in main ()
[oracle@rac1 output]$ pstack 548
Thread 5 (Thread 0x7f37ad0f9700 (LWP 553)):
#0 0x0000003161a0ba0e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007f37af38d582 in sltspctimewait () from
/u01/app/12.1.0/grid/lib/libclntshcore.so.12.1
#2 0x00007f37b2929b69 in clsdadr_bucketThread () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#3 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#4 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f37acef8700 (LWP 555)):
#0 0x00000031616df143 in poll () from /lib64/libc.so.6
#1 0x00007f37b2cabe60 in sgipcwWaitHelper () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#2 0x00007f37b2ca9191 in sgipcwWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#3 0x00007f37b2a07adf in gipcWaitOsd () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#4 0x00007f37b29ede21 in gipcInternalWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#5 0x00007f37b297f655 in gipcWaitF () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#6 0x00007f37b2780466 in clsdms_thdmai () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#7 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#8 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f37accf7700 (LWP 581)):
#0 0x0000003161a0efbd in nanosleep () from /lib64/libpthread.so.0
#1 0x000000000045dcbc in CEvm_c::waitForOCR() ()
#2 0x0000000000453f59 in CEvm_c::init() ()
#3 0x0000000000440462 in Startup(void*) ()
#4 0x00000000004f4137 in CLS_Threading::CLSthreadMain::cppStart(void*)
()
#5 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#6 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f37acaf6700 (LWP 582)):
#0 0x0000003161a0f32d in waitpid () from /lib64/libpthread.so.0
#1 0x00007f37b28de2d9 in scls_process_join () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#2 0x0000000000461567 in restartchld(restartctx*) ()
#3 0x0000003161a07a51 in start_thread () from /lib64/libpthread.so.0
#4 0x00000031616e896d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f37ad30e660 (LWP 548)):
#0 0x00000031616df143 in poll () from /lib64/libc.so.6
#1 0x00007f37b2cabe60 in sgipcwWaitHelper () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#2 0x00007f37b2ca9191 in sgipcwWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#3 0x00007f37b2a07adf in gipcWaitOsd () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#4 0x00007f37b29ede21 in gipcInternalWait () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#5 0x00007f37b297f655 in gipcWaitF () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#6 0x000000000051f097 in cls::SocketSelector::select(int) ()
#7 0x000000000041442d in cls::EvmdApplication::run() ()
#8 0x00000000004109b8 in evmdmain(int, unsigned char**) ()
#9 0x00007f37b28ba126 in clsbCMain () from
/u01/app/12.1.0/grid/lib/libhasgen12.so
#10 0x00000000004102a8 in main ()
On Tue, Nov 17, 2015 at 2:49 PM, Seth Miller <sethmiller.sm@xxxxxxxxx>
wrote:
Does the evmd log on node 1 show any errors?
On Tue, Nov 17, 2015 at 12:24 PM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
Also, rac1 and rac2 can ping themeselves and each other on the private
addresses. (172.16.100.51,172.16.100.52)
On Tue, Nov 17, 2015 at 1:14 PM, Mark J. Bobak <mark@xxxxxxxxx> wrote:
ok, ran gpnptol get on rac1:
[oracle@rac1 bin]$ ./gpnptool get
Warning: some command line parameters were defaulted. Resulting
command line:
./gpnptool.bin get -o-
<?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile
Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile";
xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile";
xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile
gpnp-profile.xsd" ProfileSequence="1"
ClusterUId="b2f13eabb1847fadff49942f72181a42"
ClusterName="rac-cluster-sca"
PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen"
HostName="*"><gpnp:Network id="net1" IP="192.168.78.0" Adapter="eth0"
Use="public"/><gpnp:Network id="net2" IP="172.16.100.0" Adapter="eth1"
Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile
id="css" DiscoveryString="" LeaseDuration="400"/><orcl:ASM-Profile
id="asm"
DiscoveryString="++no-value-at-profile-creation--never-updated-through-ASM++"
SPFile="" Mode="legacy"/><ds:Signature xmlns:ds="
http://www.w3.org/2000/09/xmldsig#";><ds:SignedInfo><ds:CanonicalizationMethod
Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod
Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/><ds:Reference
URI=""><ds:Transforms><ds:Transform Algorithm="
http://www.w3.org/2000/09/xmldsig#enveloped-signature"/><ds:Transform
Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#";><InclusiveNamespaces
xmlns="http://www.w3.org/2001/10/xml-exc-c14n#"; PrefixList="gpnp
orcl xsi"/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm="
http://www.w3.org/2000/09/xmldsig#sha1
"/><ds:DigestValue>5tZEzBB3UbfWa4jnEwxj/OVzkuQ=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>bXnVu7ARn1pPGrelGmWf/H/qx0aTDkmxHEl/8bjT+/L+hRz1HriKfu1+2xlrbViJxOQ5r1IEcO1n+jn8Ijl8Xg57DBAgCTLRUb5VV3qYJj5I6BOLUpYpz5N5Md/EB3+W/5LdboswwZ7rvMexNs1twonmLYKuQuaEoCyDf8KoDJY=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>
Success.
and on rac 2:
there is no gpnptool.
Not sure exactly how to parse all that, bu I see 172.16.100.0, which
is the correct network for the interconnect.
On Tue, Nov 17, 2015 at 12:44 PM, Mark J. Bobak <mark@xxxxxxxxx>
wrote:
Yep, looking at:
/u01/app/oracle/diag/crs/rac1/crs/trace/alert.log
the last line is:
2015-11-17 10:14:40.669 [ORAAGENT(534)]CRS-5011: Check of resource
"ora.asm" failed: details at "(:CLSN00006:)" in
"/u01/app/oracle/diag/crs/rac1/crs/trace/ohasd_oraagent_oracle.trc"
Looking at the trace file mentioned above, I see this recurring
error:
2015-11-17 10:00:22.022868 : CLSDMC:2881476352: Error: gipcWait for
gipcConnect - ret_gipcreqinfo=gipcretConnectionRefused,
type_gipcreqinfo=gipcreqtypeConnect
2015-11-17 10:00:22.023101 :CLSDYNAM:2881476352: [ora.gipcd]{0:0:2}
[check] ClsdmClient::sendMessage clsdmc_send error rmsg:0 ecode:-7
errbuf:
2015-11-17 10:00:22.031220 :CLSDYNAM:2881476352: [ora.gipcd]{0:0:2}
[check] Calling PID check for daemon
2015-11-17 10:00:22.031565 :CLSDYNAM:2881476352: [ora.gipcd]{0:0:2}
[check] Process id translated to r
2015-11-17 10:00:22.031713 : CLSDMC:2881476352: Connecting to
ipc://rac1_DBG_GIPCD
2015-11-17 10:00:22.037201 :CLSDYNAM:2870970112: [ora.evmd]{0:0:2}
[check] PID FILE doesn't exist.
Not sure what that's all about.
Double checked that firewall is down and disabled, and SELinux set
to disabled as well.
On Tue, Nov 17, 2015 at 11:32 AM, Seth Miller <
sethmiller.sm@xxxxxxxxx> wrote:
It's probably using the ADR then.
Check ORACLE_BASE/diag/crs/...
On Tue, Nov 17, 2015 at 10:24 AM, Mark J. Bobak <mark@xxxxxxxxx>
wrote:
Hi Seth,
On rac1, I see $GRID_HOME/log/rac1/alertrac1.log, but it's empty,
0 lines.
On rac2, the corresponding file does not exist.
On Tue, Nov 17, 2015 at 11:00 AM, Seth Miller <
sethmiller.sm@xxxxxxxxx> wrote:
Can you post the grid alert log from both nodes?
Seth Miller
On Tue, Nov 17, 2015 at 9:34 AM, Mark J. Bobak <mark@xxxxxxxxx>
wrote:
Hi all,
Recently doing RAC Attack install on Virtualbox 5.0.10, using
Oracle Linux 6.7 and Oracle 12.1.0.2.
No problem installing OS, running runInstaller on ASM/Grid, but
when it's time to run root.sh, everything looks fine, the
clusterware and
ASM instance both seem to start fine on node 1, but the script never
completes. After start of ora.cssd, it just hangs.
Here's the log of the output...any suggestions?
Thanks,
-Mark
[oracle@rac1 app]$ sudo /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[oracle@rac1 app]$ sudo /u01/app/12.1.0/grid/root.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/12.1.0/grid
Enter the full pathname of the local bin directory:
[/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file:
/u01/app/12.1.0/grid/crs/install/crsconfig_params
2015/11/17 09:56:09 CLSRSC-4001: Installing Oracle Trace File
Analyzer (TFA) Collector.
2015/11/17 09:57:25 CLSRSC-4002: Successfully installed Oracle
Trace File Analyzer (TFA) Collector.
2015/11/17 09:57:27 CLSRSC-363: User ignored prerequisites
during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
2015/11/17 09:59:30 CLSRSC-330: Adding Clusterware entries to
file 'oracle-ohasd.conf'
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
