Thank you Joe, the information you sent was helpful to find the root-cause!
tmon terminated the instance without giving much details in the alert log:
TMON (ospid: 14954): terminating the instance due to error 472
But as the relationship between tmon and tt0* processes has become clear
(thanks to your updates!), I looked for tt0 in the system state dump.
PROCESS 36: TT00
...
O/S info: user: oracle, term: UNKNOWN, ospid: 14994 (DEAD)
Somebody shot down the tt0!
But who might it be?
I recalled a similar case once upon a time where I had used DTrace back then to
find out the culprit: http://nenadnoveljic.com/blog/avaloq-database-crash/
It was this dodgy application again which runs on the database server under the
oracle account (the vendor imposed setup!) and executes kill -9 commands to
kill the application processes (take no prisoners!). Just sometimes they don't
get the PIDs right . So, this time the TT0 was an innocent victim of this
violence act.
Many thanks once again,
Nenad
From: Sweetser, Joe [mailto:JSweetser@xxxxxxxx]
Sent: Montag, 18. September 2017 15:52
To: Noveljic Nenad; ORACLE-L (oracle-l@xxxxxxxxxxxxx)
Subject: RE: tmon
Interesting that it's not listed here:
https://docs.oracle.com/database/121/REFRN/GUID-86184690-5531-405F-AA05-BB935F57B76D.htm
But from here:
http://akashpramanik.blogspot.com/2015/04/background-processes-in-oracle-database.html
I see this: TMON starts the TMON slave (called the TT processes) and monitors a
number of redo transport processes for hangs and death.
hth,
-joe
From: oracle-l-bounce@xxxxxxxxxxxxx<mailto:oracle-l-bounce@xxxxxxxxxxxxx>
[mailto:oracle-l-bounce@xxxxxxxxxxxxx] On Behalf Of Noveljic Nenad
Sent: Monday, September 18, 2017 7:41 AM
To: ORACLE-L (oracle-l@xxxxxxxxxxxxx<mailto:oracle-l@xxxxxxxxxxxxx>)
<oracle-l@xxxxxxxxxxxxx<mailto:oracle-l@xxxxxxxxxxxxx>>
Subject: tmon
Hi,
What does the tmon background process do?
Best regards,
Nenad
____________________________________________________
Please consider the environment before printing this e-mail.
Bitte denken Sie an die Umwelt, bevor Sie dieses E-Mail drucken.
Important Notice
This message is intended only for the individual named. It may contain
confidential or privileged information. If you are not the named addressee you
should in particular not disseminate, distribute, modify or copy this e-mail.
Please notify the sender immediately by e-mail, if you have received this
message by mistake and delete it from your system.
E-mail transmission may not be secure or error-free as information could be
intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also
processing of incoming e-mails cannot be guaranteed. All liability of the
Vontobel Group and its affiliates for any damages resulting from e-mail use is
excluded. You are advised that urgent and time sensitive messages should not be
sent by e-mail and if verification is required please request a printed version.
Confidentiality Note: This message contains information that may be
confidential and/or privileged. If you are not the intended recipient, you
should not use, copy, disclose, distribute or take any action based on this
message. If you have received this message in error, please advise the sender
immediately by reply email and delete this message. Although ICAT, Underwriters
at Lloyd's, Syndicate 4242, scans e-mail and attachments for viruses, it does
not guarantee that either are virus-free and accepts no liability for any
damage sustained as a result of viruses. Thank you.
<html xmlns="http://www.w3.org/1999/xhtml";>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css">p { font-family: Arial;font-size:9pt }</style>
</head>
<body>
<p>
<br>Important Notice</br>
<br>This message is intended only for the individual named. It may contain
confidential or privileged information. If you are not the named addressee you
should in particular not disseminate, distribute, modify or copy this e-mail.
Please notify the sender immediately by e-mail, if you have received this
message by mistake and delete it from your system.</br>
<br>E-mail transmission may not be secure or error-free as information could be
intercepted, corrupted, lost, destroyed, arrive late or incomplete. Also
processing of incoming e-mails cannot be guaranteed. All liability of the
Vontobel Group and its affiliates for any damages resulting from e-mail use is
excluded. You are advised that urgent and time sensitive messages should not be
sent by e-mail and if verification is required please request a printed
version.<br/>
</p>
</body>
</html>
