[foxboro] Weird Mesh FBM failures

Hi All,

We are getting some intermittant failures every few weeks on a version
8.2 mesh system with ZCP270s and FCM100Ets.  Some FBMs will show as
failed for about a second, and recover immediately, but the BAD logic
cascades through the system and causes things to swap to manual.  The
typical sysmon messages we are getting are below:

2008-12-30 17:30:15 1CPxxx Equip = 1XX102 SYSMON -00123 Comm Device
Failed
2008-12-30 17:30:15 1CPxxx Equip = 1XX103 SYSMON -00123 Comm Device
Failed
2008-12-30 17:30:17 1CPxxx Equip = 1XX102 SYSMON -00121 Device on scan
2008-12-30 17:30:17 1CPxxx Equip = 1XX103 SYSMON -00121 Device on scan

We have seen this about four times over the last couple of months, and
the number of FBMs that fail not consistent, ranging from three or four
on one baseplate, up to 28 FBMs on all five of the baseplates.  The
PIOEGB counter will typically increment by 28, even if we don't see
sysmon messages on all of them.  There are no other sysmon errors except
for the FBMs failing and returning to normal.  No errors on the
switches, PIO bus, FCMs, or ZCP.  There are various analog, contact i/o,
and intelligent modules on this ZCP that fail.  We also have five other
FT ZCPs on the same network that have not had any problems, just this
one ZCP.

We also saw most of the I/O fail like this when we forced a swapover of
the FT ZCP by rebooting the Primary processor from SMDH.  Otherwise the
swapover looked normal.

My question to the list is has anyone seen anything similar on a v8.x
mesh system?  If so, how did you fix it?

Thanks,
=================================================
Gaylon Hicks
Tennessee Valley Authority, Browns Ferry Nuclear Plant
Mail Stop SAB-1B
PO Box 2000
Decatur, AL 35609
Phone:(256) 614-8386 Fax:(256) 729-2047
email gfhicks@xxxxxxx
FedEx Address: Shaw Road & Nuclear Plant Road, Athens, AL 35611
==================================================

NOTICE: This electronic message transmission contains information which
may be TVA SENSITIVE, TVA RESTRICTED or TVA CONFIDENTIAL. Any misuse or
unauthorized disclosure can result in both civil and criminal penalties.
If you are not the intended recipient be aware that any disclosure,
copying, distribution or use of the content of this information is
prohibited. If you have received this communication in error, please
notify me immediately by email and delete the original message.

 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             http://www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave

Other related posts: