Re: Alert Once Functionality.

  • From: rjamya <rjamya@xxxxxxxxx>
  • To: niall.litchfield@xxxxxxxxx
  • Date: Tue, 17 Apr 2007 12:25:35 -0400

We have a custom perl script that notifies appropriate people about errors.
Now there are some errors that we deem critical. Those are flashed on our
dashboard and email is sent out every minute until acknowledged.

Operations know that if they receive certain types of emails more than 3
times in a 5 minute period, they are to call DBAs asap and silence dashboard
alarm (it is a big whoop alarm with sound control disabled and is set to
full volume) for 5 minutes. If DBAs don't ack, the alarm goes off in 5
minutes again and the call gets escalated to the whole group.

It sounds pretty convoluted and complex, but the procedures are laid out
cleanly in the SOP manual and operations folks do get special training on
this topic. They also understand that failure in such cases is a big no-no.

Certain types of emails, we get only once, certain types, we keep getting
them until they are fixed. e.g. I wrote a utl_tcp program that streams data
on to a 1200 baud line for some customers. Imagine what might happen when
someone accidently released a big story about 1.5mb in size? That day, I was
on call and I received about 200 emails by the time I fixed it (one email
every 58 seconds).

It is a good thing that my company has unlimited data plan for treo for all
the emails traffic we get from monitoring. Plus i have Outlook rules set-up
to do different things as well. A lot of informational alerts go directly to
'Deleted' folder.

These procedures work for us, YMMVW
Rjamya

On 4/17/07, Niall Litchfield <niall.litchfield@xxxxxxxxx> wrote:

All

I'm going to be talking about the perils of email notifications for Oracle
databases next week - namely that you either don't monitor what you should,
or you get drowned in alerts, often repeated. I see two basic alternatives
to this approach - the dashboard approach (see the eye candy thread) or
being alerted on the detection of a problem, and then not until it is
resolved. If anyone uses the second approach I'd love to hear what tools you
use.

--
Niall Litchfield
Oracle DBA
http://www.orawin.info




--
-----
Best regards
RJamya
-------------------------------------------

Other related posts: