Re: [foxboro] CAD Failure

  • From: "SLADE Jason -NANTICOKE" <jason.slade@xxxxxxx>
  • To: <foxboro@xxxxxxxxxxxxx>
  • Date: Wed, 18 Jan 2012 15:55:51 -0500

Thanks for getting back to me Terry.  I recall reading about that CAD problem 
vs the new workstation a few years ago - nasty!

Last year we added a couple Windoz I/A V8.5 boxes with some mesh and a CP270 
and ATS into our predominantly Unix I/A V6.5 & 7.1 environment and everything 
seemed fine for several months.  Since getting blindsided by the CAD crash, 
we've learned that the highest version I/A box is supposed to be the 
DEV_MONITOR master at all times; in our case the Windoz AW.  All indications 
say is was, but apparently somebody took exception to that and they fought 
about it at length.  We have 9 AIM* packages running on 9 separate AWs and all 
CADs seized up solid about the same time.

I'm sure all users could breathe easy if they knew their I/A system could/would 
somehow alarm or draw attention to the fact that CAD is broke.

Here's hoping someone has and will share that silver bullet ...

Jason Slade 
  
Control Technologist 
Protection & Control Systems Support 
Nanticoke G.S. 


-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx] On 
Behalf Of Terry Doucet
Sent: January 18, 2012 2:11 PM
To: foxboro@xxxxxxxxxxxxx
Subject: Re: [foxboro] CAD Failure

Jason,The closest problem for me to what you describe occurred when someone 
added new WP to the system without telling the rest of the system about this 
WP. At some later time, this WP took over as the Device Monitor MASTER for the 
system. Since this Device Monitor Master could not detect AW's, WP's and 
printers, it instructed the CP's to stop sending alarms to those devices.  I 
cannot remember if the Historian stopped receiving alarms or not but no CAD's 
received any alarms. Of course, you have to realize that no alarms are coming 
before you know that the CAD is dead.  Device Monitor was working correctly it 
was the installer who goofed.
Is there one AW or WP (capable of being Device Monitor Master) that is off by 
itself on a section of your network? Since the Historian kept receiving alarms 
perhaps the problem is with the AW running historian.
Terry


> Subject: [foxboro] CAD Failure
> Date: Wed, 18 Jan 2012 12:46:45 -0500
> From: jason.slade@xxxxxxx
> To: foxboro@xxxxxxxxxxxxx
> 
> We recently had a plant-wide CAD failure with all 46 workstations 
> being affected.  Due to the plant's running mode at the time, it took 
> OPS a while to notice that there were no process alarms coming from 
> dozens of CPs to CAD.  The historian however was still receiving the 
> messages!?
> No SYS_MON alarm - nothing to bring this failure to the attention of 
> OPS.  Even with all the log files, TAC could not ascertain what was 
> actually happening or what the cause was.  One particular log file 
> indicated that there was a fight going on for control of DEV_MONITOR.
> 
> We got CAD back up and running, but I was wondering:
> - has anyone else been hit by this problem?
> - does anyone have a site implemented work-around to bring to OPS 
> attention that CAD/DEV_MONITOR has failed?
> 
> Thanks,
> 
> Jason Slade
> 
> Control Technologist
> Protection & Control Systems Support
> Nanticoke GS
> 
> -----------------------------------------
> THIS MESSAGE IS ONLY INTENDED FOR THE USE OF THE INTENDED
> RECIPIENT(S) AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, 
> PROPRIETARY AND/OR CONFIDENTIAL. If you are not the intended 
> recipient, you are hereby notified that any review, retransmission, 
> dissemination, distribution, copying, conversion to hard copy or other 
> use of this communication is strictly prohibited. If you are not the 
> intended recipient and have received this message in error, please 
> notify me by return e-mail and delete this message from your system. 
> Ontario Power Generation Inc.
>  
>  
> ______________________________________________________________________
> _ This mailing list is neither sponsored nor endorsed by Invensys 
> Process Systems (formerly The Foxboro Company). Use the info you 
> obtain here at your own risks. Read 
> http://www.thecassandraproject.org/disclaimer.html
>  
> foxboro mailing list:             //www.freelists.org/list/foxboro
> to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>  
                                          
 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process Systems 
(formerly The Foxboro Company). Use the info you obtain here at your own risks. 
Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             //www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
 
 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             //www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
 

Other related posts: