Actual error CommVault engineer pulled from the logs is 'connection with the
OraSbtThread failed, possibly due to a stuck DLL'
I think this could be the smoking gun
On Jan 9, 2016, at 4:46 PM, Mladen Gogala <gogala.mladen@xxxxxxxxx> wrote:--
On 01/08/2016 11:21 PM, MJ Mody wrote:
Suspected resource contention of sorts. Follow-up with backup vendor,
CommVault, yielded a file lock with cv/oracle library (i.e. oci.dll) from an
earlier backup. Will need to review the Oracle shadow copy interface.
Still a mystery as to why nothing was written to alert log.
Was there anything in the ClOraAgent.log? Did you open a case with CommVault
support? Do you have any other backup system on the machine? Simpana client
library is called ORASBT.DLL and usually resides in C:\Program
Files\Commvault\Simpana\Base. If there was another backup software on the
machine, it can put ORASBT.DLL either in %ORACLE_HOME%\lib or in
C:\Windows\system32. Please, search the entire machine for the ORASBT.DLL
libraries, since the library clash can be quite serious problem. The name of
the library is mandated by Oracle Corp. and all backup suites (Commvault,
TSM, NetBackup, Avamar) must use the same name.
Regards
Cheers
MJ
On Jan 8, 2016, at 9:26 PM, De DBA <dedba@xxxxxxxxxx> wrote:--
Usually I'd suspect an out-of-memory situation. The scenario reminds me of
a case I had where the backup software (Tivoli, I think), would use Windows
shadow copy to execute an RMAN backup. The shadow copy process never exited
and after a (large) number of backups the Oracle process memory was
exhausted and the database hang. From memory that was also W2K8R2/11.2.0.3.
I believe there was a bug in the Oracle shadow copy interface.
You did check the Windows Event Logs (Application and System) I presume?
Hth,
Tony
On 09/01/16 11:03, MJ Mody wrote:
Nothing like ringing in a new year than with a hung database.
In this case, running 11.2.0.3 EE with ASO/TDE on W2K8-R2 with last CPU.
From what is known, external backups (tape writes) were taking place in
backup software, both rman and data pump (the .dmp files were created
early in the evening). Additionally, nightly EOD processing was also
taking place.
Performed a server reboot to bring the database online. Point of concern
is from 9pm onwards, until the database came back online (~1:30am), there
is no entries in the alert log.
Feel free to share if anyone has come across this or have some words of
wisdom for troubleshooting and remediation.
Thank you in advance.
Best
MJ--
//www.freelists.org/webpage/oracle-l
//www.freelists.org/webpage/oracle-l
--
Mladen Gogala
Oracle DBA
http://mgogala.freehostia.com
--
//www.freelists.org/webpage/oracle-l
