Re: [foxboro] Measuring Overall System Reliability: aka CP60 Fieldbus Communication Errors
- From: tom.vandewater@xxxxxxxxxxxxxx
- To: foxboro@xxxxxxxxxxxxx
- Date: Tue, 15 Jun 2004 15:35:22 -0400
Scott,
CAR-1005217 Answers to your questions are:
How many 100series fbms are behind each DCM10E ?
We are using the remote connection option that connects the DCM to legacy
FBI's via the same twin-ax cabling used to connect to the old CP10/30/40
remote fieldbus. The DCM10E supports up to 24 Fieldbus Isolators, (FBIs),
on the remote Fieldbus per CP60. Each FBI segment has no more than 24 FBM's
as per Fox guidelines. We have up to five FBI segments communicating with
the DCM10E, but never more than 60 legacy FBMs total on any CP-60.
What iom82 version is installed as seen in sys. Mgr. info for DCMs ?
IOM 82 EEProm Rev. is 1.14 per the latest Quick Fix I've been able to find
on the Foxboro website, (QF1005124), but I've heard from another user that
Rev. 1.15 is already in the works.
Are you getting frequent/infrequent unexplained FCM Failed messages ?
We aren't getting FCM or DCM failed messages anymore, (we used to get those
frequently at IOM 82 Rev 1.07, 1.08, and 1.09) Now we are getting
individual FBM PIO bus access errors on different FBM's on the same CP at a
rate of 2-10/hour per each CP-60. I just grabbed activity in the last 5
minutes and the SYSMON messages, (which we capture and historize in log
files), look like this:
06-15-04 14:50:42 0 SYSMON = SYS09A 09CP03 Equip = 09CP03
SYSMON -00057 Currently using PIO bus B
06-15-04 14:50:44 0 SYSMON = SYS09A 09CP03 Equip = 090803
SYSMON -00068 Single PIO Bus Access Error on A
06-15-04 14:50:56 0 SYSMON = SYS09A 09CP03 Equip = 090803
SYSMON -00070 Single PIO Bus Access Recovery on A
06-15-04 14:53:51 0 SYSMON = SYS08A 08CP02 Equip = 08CP02
SYSMON -00057 Currently using PIO bus B
06-15-04 14:53:51 0 SYSMON = SYS08A 08CP02 Equip = 080503
SYSMON -00068 Single PIO Bus Access Error on A
06-15-04 14:54:13 0 SYSMON = SYS08A 08CP02 Equip = 080503
SYSMON -00070 Single PIO Bus Access Recovery on A
I have an active CAR-1005217, (the second one I had to write because
they closed out my other one even though the problem wasn't resolved). I
haven't received an official response from Foxboro yet on CAR-1005217. I
think it is because they haven't yet figured out how to resolve all the
communication problems associated with this and I don't want to continue to
be the guinea pig used for their testing.
I know that I can make all of the PIO bus access error messages stop
by forcing the CP to stay on one fieldbus or the other by configuring FBM 0
to BUS A: Disable Switching. I have run for weeks that way to stop the
constant stream of error messages but after installing IOM82 Rev. 1.11 the
CP got confused about which bus to use and switched to the one that was
disabled causing us to lose all communication to the FBM's! That caused a
process downing event. Since that time I have been afraid to force the CP
to communicate on only one fieldbus and so it switches up to 10 times an
hour and gives me the PIO bus access errors shown above.
I have one CP-60 where we installed an FBM with a physically bad "B"
fieldbus communication problem and on that one I can leave FBM 0 configured
to "BUS A: Enable Switching" but it doesn't switch because of the bad "B" on
one FBM. Amazingly, I get no PIO bus access errors on that CP anymore but
if more FBM's physically failed on "A" than "B" I think it would switch to
"B" and I would only lose communication with the one bad one on "B".
I originally sent this note out to see if anyone had any good
measure of System reliability but have only received several comments about
other people who have experienced the CP-60 fieldbus communication problems
that I have. One note, sent to me off list, said that CP-60 Rev L had the
problem and by switching out CP's to a more current Rev than L seemed to fix
his problem. Comments anyone?
Tom VandeWater
Control Systems Developer/Analyst
Dow Corning Corp.
Carrollton, KY USA
-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx
[mailto:foxboro-bounce@xxxxxxxxxxxxx]On Behalf Of Landry, Scott
Sent: Tuesday, June 15, 2004 11:40 AM
To: 'foxboro@xxxxxxxxxxxxx'
Subject: Re: [foxboro] Measuring Overall System Reliability
Importance: High
Hi Tom,
How many 100series fbms are behind each DCM10E ?
What iom82 version is installed as seen in sys. Mgr. info for DCMs ?
Are you getting frequent/infrequent unexplained FCM Failed messages ?
The smon messages can be logged at V6.2+ on the smon host hard drive.
mkdir /opt/fox/sysmgm/sysmon
touch /opt/fox/sysmgm/sysmon/smon_log
Any new messages will go to smon_log after the touch; no need to
reboot or restart smon.
If you visit ips.csc.invensys.com you will find a few customer
Notifications on CP60 related issues below that I think may apply to
Your Hardware types.
The local Foxboro/IPS service rep. will be able to help you, also.
Regards,
Scott L. Landry
Regional Lifeline Specialist
Site Contracts Manager
Southeast USA
Foxboro/IPS
CP60 and FCM Comm. Error Messages
Background
This advisory is to inform you that some CP60s, (P/N P0961FR), may exhibit
FCM communication errors. These errors can manifest themselves as cyclic FCM
(A or B) failure /recovery messages, followed by cyclic FBM single access
error/recovery messages hosted by the FCM in question.
Error message examples:
2003-05-07 12:00:18 CP6001 Equip = IOBP00 SYSMON -00172 FCM A module Failed
2003-05-07 12:00:18 CP6001 Equip = IOBP00 SYSMON -00182 FCM B module Failed
A multi-discipline EMC (Electro-Magnetic Capability) study was initiated to
understand the cause of these FCM communication errors.
Resolution
As a result, Invensys is recommending the following changes:
Ensure correct iom82 file is installed -
see Customer Advisory 2003023 Rev 2.0 - "CP60 PIO Bus Failure Indication
QF1005124"
Addition of EMI suppression ferrite cores to Fieldbus cables
Qualify minimum RG58U Fieldbus cable specifications -
see Customer Advisory 2003028 ABI - "RG58 Coaxial Cable Types"
Enclosure improvements to earth (ground) connections
A diagnostic routine (CG500BD) is to be used to help identify if a CP60
should be replaced. Noting that the minimum revision level is G or higher.
Customers who are concerned with performing EEprom updates online have the
option to EEprom the modules in an offline system prior to replacing the
module on their production system.
For completed details of these changes, refer to Field Service Installation
kits AG100XH, XM & XN. You may also refer to the instructions (CG800AJ)
available on the CSC website. If you have further questions regarding this
advisory or Fieldbus communication problems still persist, please contact
the nearest Invensys Customer Satisfaction Center:
-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx] On
Behalf Of tom.vandewater@xxxxxxxxxxxxxx
Sent: Tuesday, June 15, 2004 8:01 AM
To: foxboro@xxxxxxxxxxxxx
Subject: Re: [foxboro] Measuring Overall System Reliability
Dirk,
Thanks for the information. We are using DCM10E's which connect to
the CP-60's via very short runs of the thin white coax supplied by Foxboro.
The DCM's are connected to the old FBI's by the same copper twinax
recommended by Foxboro. Our longest run is less than 100 ft. We never had
any problems with that twinax or fieldbus communications when the same FBM's
and FBI's were connected to CP-10's and 30's but started having problems as
soon as we connected them to the CP-60's via the DCM's.
It is interesting that all of the old style FBM's that give us the
very familiar SYSMON message:
SYSMON -00069 Single PIO Bus Access Error on B
are discrete FBM's such as the FBM 41 that you mention.
You said in your note "upgrading the CP's minimised the
interference". I am very curious to know what you meant by that. Did you
physically replace the CP-60 hardware with new CP-60's at a different
hardware Rev. or just upgrade the CP-60 software image?
I would also like to know what hardware Rev your CP-60's are
running. All the ones I'm having trouble with are Rev. L. Could you check
what Rev. yours are and let me know? It can be seen in System Management
by:
selecting the CP-60
picking the CONFIG button at the bottom of the screen
picking the EQUIP INFO button at the bottom of the screen
picking the DOWN ARROW button at the bottom of the screen
The Rev is listed under PRIM HARD REV and SHAD HARD REV
Thanks for sharing with the user community,
Tom VandeWater
Control Systems Developer/Analyst
Dow Corning Corp.
Carrollton, KY USA
-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx
[mailto:foxboro-bounce@xxxxxxxxxxxxx]On Behalf Of PAUWELS, DIRK -
Eastman
Sent: Tuesday, June 15, 2004 4:00 AM
To: foxboro@xxxxxxxxxxxxx
Subject: Re: [foxboro] Measuring Overall System Reliability
We have both new and old type FBM and only new FBI's connected to
CP60's. We have 2 nests (old type mounting structures) that need to be
changed, FBM41 in certain slots go offline without any reason, but other
than that no problems. We did experience interference between FBI10E and
CP, causing the FT CP60 to go single. Renewing the connectors &
isolating the coaxcables and connectors (din rail) & upgrading the CP's
minimised the interference, we have not had com failures ever since. Are
you using fiber between FBI and CP or coax? Fiber eliminates the
interference. Our FBI's are in an MCC room +/- 130Mtrs away from the
CP's.
Rgds,
Dirk Pauwels Tel: 32 (0)3 570 95 97
DCS coordinator Fax: 32 (0)3 570 16 09
Eastman Belgium Mobile:32 (0)497 428 300
E-mail: dpauwels@xxxxxxxxxxx
-----Original Message-----
From: tom.vandewater@xxxxxxxxxxxxxx
[mailto:tom.vandewater@xxxxxxxxxxxxxx]=20
Sent: Monday, June 14, 2004 7:47 PM
To: foxboro@xxxxxxxxxxxxx
Subject: [foxboro] Measuring Overall System Reliability
Hi List,
I am trying to develop a measure of overall control system
reliability and wonder if anyone out there already has developed a
metric
for this purpose. Our twenty node, carrierband connected system is as
redundant/fault tolerant as we can make it with the current Foxboro
offerings with the exception of redundant I/O. We use dual carrierband,
dual nodebus and dual fieldbus functionality as well as dual power
sources
with UPS backup and are expected to provide 100% on-line time for our
24x7x365 operation. We also climate control almost every nest room
where
the FT controllers and non-FT I/O are located. We have experienced
pretty
good reliability with the exception of individual FBM failures that
control
critical process variables or more recently, fieldbus communication
failures
related to Foxboro's new FCM, DCM, FBI10E's. Foxboro continues to
search
for the solution to that problem with new EEPROM updates being issued
every
three or four months and we have only experienced one loss of
communication
that resulted in a process downing event, but volumes of intermittent
fieldbus communication errors.
What I am looking for is a measurement that we can continue to
track
that will indicate when system reliability starts to deteriorate. Since
2/3rds of our installed base still uses legacy FBM's that will need to
be
replaced sometime in the future, I have started to focus on them. We
have
recorded failure data on them since our initial 12 node installation in
1989. We saw an initial infant mortality of those modules in 1989-91
and
then the rate dropped off and continues to track along the bottom of the
bathtub curve at an acceptable electronic module failure rate of less
than
.5%/year. We have seen an increasing number of those modules that would
run, but failed to successfully burn new EEPROM updates during software
upgrades. We know that the old FBM's will eventually start failing at a
higher rate and climb up the other side of the bathtub curve but we
don't
know when. If we had some good prediction of that increase in failure
rate,
it could be used to justify to management the significant budgeting
needs
that will be required to replace all of the legacy FBM's.
When Invensys acquired TRICONEX I had hope that Foxboro and
TRICONEX
would pool their resources and come up with a Triple-Modular-Redundant,
(TMR), I/O module that could communicate directly to Foxboro or Triconex
controllers but I have seen no visible activity in that area.
If anyone has any ideas about an effective measure of system
reliability please contact me off-list or share the info with the rest
of
the users on the list.
Thanks in advance,
Tom VandeWater
Control Systems Developer/Analyst
Dow Corning Corp.
Carrollton, KY USA
=20
=20
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
=20
foxboro mailing list: http://www.freelists.org/list/foxboro
to subscribe: =
mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Djoin
to unsubscribe: =
mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Dleave
=20
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
foxboro mailing list: http://www.freelists.org/list/foxboro
to subscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
foxboro mailing list: http://www.freelists.org/list/foxboro
to subscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
foxboro mailing list: http://www.freelists.org/list/foxboro
to subscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
foxboro mailing list: http://www.freelists.org/list/foxboro
to subscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
Other related posts:
- » Re: [foxboro] Measuring Overall System Reliability: aka CP60 Fieldbus Communication Errors