Re: OPatchAuto failed

  • From: Sameer Choudhari <sameer.choudhari@xxxxxxxxxxxxxxx>
  • To: dimensional.dba@xxxxxxxxxxx
  • Date: Fri, 4 Oct 2019 19:00:58 +1000

Was the original $ORACLE_HOME = /u01/g01/grid/12.1/OPatch    ------>
YES. OPatch
version is same on each nodes and each homes


[root@uat-db-002 oracle]#
[root@uat-db-002 oracle]# echo $ORACLE_HOME
/u01/g01/grid/12.1
[root@uat-db-002 oracle]# env |grep PATH
MANPATH=:/opt/puppetlabs/puppet/share/man
PATH=/u01/g01/grid/12.1/bin:/u01/g01/grid/12.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/u01/d01/db/12.1.0/bin:/u01/d01/db/12.1.0/OPatch:/u01/g01/grid/12.1/bin
[root@uat-db-002 oracle]#
[root@uat-db-002 oracle]#
[root@uat-db-002 oracle]# echo $ORACLE_HOME/OPatch
/u01/g01/grid/12.1/OPatch
[root@uat-db-002 oracle]# /u01/g01/grid/12.1/OPatch/opatch version
OPatch Version: 12.2.0.1.17

OPatch succeeded.
[root@uat-db-002 oracle]#
[root@uat-db-002 oracle]#
[root@uat-db-002 oracle]# cd  /home/oracle/
[root@uat-db-002 oracle]# . ./UATEBS2_uat-db-002.env
[root@uat-db-002 oracle]#
[root@uat-db-002 oracle]#
[root@uat-db-002 oracle]# echo $ORACLE_HOME/OPatch
/u01/d01/db/12.1.0/OPatch
[root@uat-db-002 oracle]# /u01/d01/db/12.1.0/OPatch/opatch version
OPatch Version: 12.2.0.1.17

OPatch succeeded.
[root@uat-db-002 oracle]#


Cheers,
Sameer


On Fri, 4 Oct 2019 at 18:51, <dimensional.dba@xxxxxxxxxxx> wrote:

You set the PATH twice

[root@uat-db-002 .ssh]# export PATH=$PATH:$ORACLE_HOME/OPatch

[root@uat-db-002 .ssh]# export PATH=$PATH:/u01/g01/grid/12.1/OPatch

Was the original $ORACLE_HOME = /u01/g01/grid/12.1/OPatch or something
else



Run

Env | grep PATH

Does the path have different paths to OPatch?

Whatever is the first one in the PATH variable will be used first.



Is the version of opatchauto/opatch the same on each node and the required
version?









*From:* oracle-l-bounce@xxxxxxxxxxxxx <oracle-l-bounce@xxxxxxxxxxxxx> *On
Behalf Of *Sameer Choudhari
*Sent:* Thursday, October 3, 2019 8:41 PM
*To:* Oracle-L@xxxxxxxxxxxxx
*Subject:* OPatchAuto failed



Team



I am applying JULY 2019 Proactive Bunble Patch on our UAT env.



Env :



DB  : 12.1.0.2.0  (2 node rac)

OS  : Oracle Linux 7



I am able to patch node-1 i.e. *uat-db-001 *without any issue but got
below error while patching node -2 i.e. uat-db-002 .





[root@uat-db-002 .ssh]# export PATH=$PATH:$ORACLE_HOME/OPatch

[root@uat-db-002 .ssh]# export PATH=$PATH:/u01/g01/grid/12.1/OPatch
[root@uat-db-002 .ssh]# cd  /u01/stage/COMBO_JVM_DBBP_JUL2019/29699255
[root@uat-db-002 29699255]# $ORACLE_HOME/OPatch/opatchauto apply
/u01/stage/COMBO_JVM_DBBP_JUL2019/29699255/29698629

OPatchauto session is initiated at Fri Oct  4 13:13:54 2019

System initialization log file is
/u01/g01/grid/12.1/cfgtoollogs/opatchautodb/systemconfig2019-10-04_01-14-00PM.log.

Session log file is
/u01/g01/grid/12.1/cfgtoollogs/opatchauto/opatchauto2019-10-04_01-14-44PM.log
The id for this session is LBTP
Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
 Command output:
oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException:
Execution failed for host uat-db-001 due to : java.lang.Exception: Remote
command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
 Command output:
OPatchAuto failed.

OPatchauto session completed at Fri Oct  4 13:15:08 2019
Time taken to complete the session 1 minute, 14 seconds

 opatchauto failed with error code 42
[root@uat-db-002 29699255]#



Checked MOS and Google but no help. I have shared partial session log
below i.e.
/u01/g01/grid/12.1/cfgtoollogs/opatchauto/opatchauto2019-10-04_01-14-44PM.log 
.
Could you please let me know what could be the issue.







Oct 04, 2019 1:15:06 PM
com.oracle.cie.remote.linux.LinuxCommandExecutionHelper getSSHCmd
INFO: SSH path::/bin/ssh
Oct 04, 2019 1:15:06 PM
com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl
executeCommand
INFO: Command execution command: /bin/ssh -o FallBackToRsh=no -o
PasswordAuthentication=no -o StrictHostKeyChecking=no -o
NumberOfPasswordPrompts=0 uat-db-001 ls /u01/g01/grid/12.1 > /dev/null
Oct 04, 2019 1:15:07 PM
com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl
executeCommand
INFO: Command execution return code: 0
Oct 04, 2019 1:15:07 PM
com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl
executeCommand
INFO: command execution message:
Oct 04, 2019 1:15:07 PM
com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl
executeCommand
INFO: command executon failure message:
*** Authorized uses only. All activity may be monitored and reported. ***


Oct 04, 2019 1:15:07 PM
com.oracle.helper.remote.ssh.SSHEquivalenceCommandExecutor executeAndNotify
INFO: Executing command on host uat-db-001 :
/u01/g01/grid/12.1//perl/bin/perl
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl
-GRID_HOME=/u01/g01/grid/12.1 -OBJECTLOC=/u01/g01/grid/
12.1//cfgtoollogs/opatchautodb/hostdata.obj -CRS_ACTION=check
-CLUSTERNODES=uat-db-001,uat-db-002
-JVM_HANDLER=oracle/dbsysmodel/driver/sdk/productdriver/remote/RemoteOperationHelper
Oct 04, 2019 1:15:07 PM
com.oracle.cie.remote.linux.LinuxCommandExecutionHelper getSSHCmd
INFO: SSH path::/bin/ssh
Oct 04, 2019 1:15:07 PM
com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl
executeCommand
INFO: Command execution command: /bin/ssh -o FallBackToRsh=no -o
PasswordAuthentication=no -o StrictHostKeyChecking=no -o
NumberOfPasswordPrompts=0 uat-db-001 /u01/g01/grid/12.1//perl/bin/perl
/u01/g0
1/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl
-GRID_HOME=/u01/g01/grid/12.1
-OBJECTLOC=/u01/g01/grid/12.1//cfgtoollogs/opatchautodb/hostdata.obj
-CRS_ACTION=check -CLUSTERNODES=uat-db-001
,uat-db-002
-JVM_HANDLER=oracle/dbsysmodel/driver/sdk/productdriver/remote/RemoteOperationHelper
Oct 04, 2019 1:15:07 PM
com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl
executeCommand
INFO: Command execution return code: 2
Oct 04, 2019 1:15:07 PM
com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl
executeCommand
INFO: command execution message:
Oct 04, 2019 1:15:07 PM
com.oracle.cie.remote.linux.LinuxSSHUserEquivalenceConnHandlerImpl
executeCommand
INFO: command executon failure message:
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_6
4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.

Oct 04, 2019 1:15:07 PM
com.oracle.helper.remote.ssh.SSHEquivalenceCommandExecutor executeAndNotify
WARNING: Error while executing command "/u01/g01/grid/12.1//perl/bin/perl
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl
-GRID_HOME=/u01/g01/grid/12.1 -OBJECTLOC=/u01/g01/grid/12.1/
/cfgtoollogs/opatchautodb/hostdata.obj -CRS_ACTION=check
-CLUSTERNODES=uat-db-001,uat-db-002
-JVM_HANDLER=oracle/dbsysmodel/driver/sdk/productdriver/remote/RemoteOperationHelper
" on host uat-db-001 :
 Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_6
4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
 Command output:
Oct 04, 2019 1:15:07 PM
com.oracle.helper.remote.ssh.SSHEquivalenceCommandExecutor executeAndNotify
WARNING: Exception while executing command "" on host uat-db-001 : Remote
command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_6
4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
 Command output:
Oct 04, 2019 1:15:07 PM
com.oracle.helper.remote.ssh.SSHEquivalenceThreadHandler$SSHEquiCommandListener
notifyResult
INFO: Received command execution result for host uat-db-001
Oct 04, 2019 1:15:07 PM
oracle.dbsysmodel.driver.sdk.productdriver.remote.HostDataCollector
notifyResult
INFO: Consolidating result from host : uat-db-001
Oct 04, 2019 1:15:07 PM
oracle.dbsysmodel.driver.sdk.productdriver.remote.HostDataCollector
notifyResult
INFO: Execution failed for host uat-db-001 due to : Remote command
execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_6
4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
 Command output:
Oct 04, 2019 1:15:07 PM
oracle.dbsysmodel.driver.sdk.productdriver.remote.HostDataCollector
performRemoteOperForActions
INFO: Remote node list for home /u01/g01/grid/12.1 : []

2019-10-04 13:15:07,859 INFO  [1]
oracle.dbsysmodel.driver.sdk.util.OsysUtility - Output message :::
2019-10-04 13:15:07,870 INFO  [1]
com.oracle.cie.common.util.reporting.CommonReporter - Reporting console
output : Message{id='null', message='Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_6
4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
 Command output: '}
2019-10-04 13:15:07,873 INFO  [1]
com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBBaseProductSupport
- Space available after session: 33580 MB
2019-10-04 13:15:07,996 SEVERE [1] com.oracle.glcm.patch.auto.OPatchAuto -
OPatchAuto failed.
com.oracle.glcm.patch.auto.OPatchAutoException:
oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException:
Execution failed for host uat-db-001 due to : java.lang.Exception: Remote
command exe
cution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_6
4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
 Command output:
        at
com.oracle.glcm.patch.auto.OPatchAuto.prepareOrchestration(OPatchAuto.java:775)
        at
com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:397)
        at
com.oracle.glcm.patch.auto.OPatchAuto.orchestrate(OPatchAuto.java:344)
        at com.oracle.glcm.patch.auto.OPatchAuto.main(OPatchAuto.java:212)
Caused by:
oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverException:
Execution failed for host uat-db-001 due to : java.lang.Exception: Remote
command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_6
4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
 Command output:
        at
oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.checkForException(OUIDriver.java:846)
        at
oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.handleExceptions(OUIDriver.java:832)
        at
oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.fetchRemoteHostData(OUIDriver.java:791)
        at
oracle.dbsysmodel.driver.sdk.productdriver.OUIDriver.fetchRemoteNodeData(OUIDriver.java:690)
        at
com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.DBPatchingHelper.fetchPreInitiatedNodes(DBPatchingHelper.java:899)
        at
com.oracle.glcm.patch.auto.db.product.validation.validators.RemoteGIStatusValidation.isFirstNodeOfPatching(RemoteGIStatusValidation.java:274)
        at
com.oracle.glcm.patch.auto.db.product.validation.validators.RemoteGIStatusValidation.validate(RemoteGIStatusValidation.java:185)
        at
com.oracle.glcm.patch.auto.db.product.validation.DBValidationController.invoke(DBValidationController.java:91)
        at
com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.CRSTopologyBuilder.validate(CRSTopologyBuilder.java:367)
        at
com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.AbstractTopologyBuilder.buildTopology(AbstractTopologyBuilder.java:117)
        at
com.oracle.glcm.patch.auto.db.integration.model.productsupport.topology.CRSTopologyBuilder.buildTopology(CRSTopologyBuilder.java:112)
        at
com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBBaseProductSupport.loadTopology(DBBaseProductSupport.java:192)
        at
com.oracle.glcm.patch.auto.db.integration.model.productsupport.DBProductSupport.loadTopology(DBProductSupport.java:69)
        at
com.oracle.glcm.patch.auto.OPatchAuto.loadTopology(OPatchAuto.java:1732)
        at
com.oracle.glcm.patch.auto.OPatchAuto.prepareOrchestration(OPatchAuto.java:730)
        ... 3 more
2019-10-04 13:15:07,999 INFO  [1]
com.oracle.cie.common.util.reporting.CommonReporter - Reporting console
output : Message{id='null',
message='oracle.dbsysmodel.driver.sdk.productdriver.ProductDriverE
xception: Execution failed for host uat-db-001 due to :
java.lang.Exception: Remote command execution failed due to
*** Authorized uses only. All activity may be monitored and reported. ***

Can't locate Cwd.pm in @INC (@INC contains:
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1/x86_64-linux-thread-multi
/u01/g01/grid/12.1/perl/lib/site_perl/5.14.1
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_6
4-linux-thread-multi /u01/g01/grid/12.1/perl/lib/5.14.1 .) at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
BEGIN failed--compilation aborted at
/u01/g01/grid/12.1/OPatch/auto/database/bin/RemoteHostExecutor.pl line 2.
 Command output: '}
2019-10-04 13:15:08,003 INFO  [1]
com.oracle.cie.common.util.reporting.CommonReporter - Reporting console
output : Message{id='null', message='OPatchAuto failed.'}
[root@uat-db-002 29699255]#







So far , I tried



For "Can't locate Cwd.pm in @INC"  , I checked it's there. Also, checked
the permission for /u01/g01/grid/12.1/cfgtoollogs/opatchautodb  and its
owned by grid home user.



[root@uat-db-002 29699255]# find / -name Cwd.pm
/usr/lib64/perl5/vendor_perl/Cwd.pm
/u01/g01/grid/12.1/perl/lib/5.14.1/x86_64-linux-thread-multi/Cwd.pm
/u01/g01/grid/12.1/perl/lib/5.14.1/Cwd.pm
/u01/d01/db/12.1.0/perl/lib/5.14.1/x86_64-linux-thread-multi/Cwd.pm

/u01/em_agent/agent_13.2.0.0.0/perl/lib/5.14.4/x86_64-linux-thread-multi/Cwd.pm
[root@uat-db-002 29699255]#



Cheers

Sameer




This is an email from Ezidebit Pty Ltd. The email and any attachments may
be confidential, legally privileged and/or subject to copyright. If you are
not the intended recipient, you must not disclose or use the information
contained in this e-mail. If you have received this email in error, please
notify us immediately and delete the email and all copies. We do not
guarantee that this email and any attachments are free from virus or other
errors. We will not be responsible for loss or damage resulting (either
directly or indirectly) from any such virus or error. We will only be
responsible for a change to a document if we made the change. The content
of and opinions expressed in non-business emails are not that of Ezidebit
Pty Ltd and are that of the sender.


-- 
This is an email from Ezidebit Pty Ltd. The email and any attachments may 
be confidential, legally privileged and/or subject to copyright. If you are 
not the intended recipient, you must not disclose or use the information 
contained in this e-mail. If you have received this email in error, please 
notify us immediately and delete the email and all copies. We do not 
guarantee that this email and any attachments are free from virus or other 
errors. We will not be responsible for loss or damage resulting (either 
directly or indirectly) from any such virus or error. We will only be 
responsible for a change to a document if we made the change. The content 
of and opinions expressed in non-business emails are not that of Ezidebit 
Pty Ltd and are that of the sender.

Other related posts: