Re: [foxboro] 100 Series FBM Reliability Data

  • From: "Johnson, Theodore (Ted) S" <TSJohnson@xxxxxxxxxxx>
  • To: <foxboro@xxxxxxxxxxxxx>
  • Date: Fri, 19 Oct 2007 15:33:06 -0800

This topic strikes a nerve at our facility.  =20

We have experienced multiple FBM 100 series 'failures'; however only in
our hydrocracker area.  The breakdown is as follows: 9 FBM04s (out of
26, or 35%) and 3 FBM01s (out of 8, or 37.5%) since 2003.  They are all
the same vintage (installed in 1997) and the problem can't be isolated
to a single CP/cabinet/1x8/slot.  We have not seen incidents with other
100 series FBM types (FBM02, FBM041) in this area, nor other FBM04/01
failures in another operating area (different control room, UPS power
feed, and vintage of FBMs); or 0%.

In 2004/5 we attempted to formally address this. Together with Invensys
Foxboro field service we inspected the power, grounding, cabling, and
fieldbus terminations.  I recall we did find a couple of minor things at
that time, but nothing I would call major discoveries. Since then on our
own we tightened field wiring terminations and termination cable
assemblies (nosecones); monitored power supply voltages, etc. In 2005/6,
we upgraded workstations/software (51F @ V7.1.1), CPs (30/40s to 60's
w/fiber hubs) and added P91 (@ V8.1) to host the Mesh and replace CLAN.
In 2007 we added FCP270s and 200 series I/O.  But to no avail, we have
experienced additional FBM04/01 'failures' during this overall time
frame.  Engineering evaluations on most returned modules have been
conducted by Invensys, but they have not been able to confirm any
failure mode (i.e. "no problem found").  We seem to be missing
something, somewhere...=20

In our case when an FBM04/01 'fails', it goes offline (red/green or dead
red) and will not reset (pull-push nosecone/fbm/lettterbug, and/or
eeprom update/download by themselves will not cause it to come back
on-line). In the case of FBM04s (output channels connected to control
valves) this causes a major process upset: the board operator can't see
what is going on (points are smurfed on graphics) and they lose control.
They must quickly scramble to place other operators in the field who put
things in manual/bypass,  closely monitor local indicators, and make
adjustments to manual valves (at the direction of main board operator
via radio) - until the FBM issue is resolved, things can be put back in
automatic, and the process settles out.  Needless to say but during this
time (because of critical temperatures and pressures associated with the
hydrocracker) adrenaline levels and emotions run higher.=20

On initial earlier incidents, we tried replacing only the FBM but
occasionally a problem would reoccur (seems there was still a problem
with the nosecone?). Other times then we replaced the FBM and nosecone,
but seemingly still had a problem with a letterbug (what can go wrong
with those?).  Of the 12 'failures' we have experienced to date, only
one was a repeat failure (though we are not certain that all fbm,
nosecone, and letterbug were replaced the first time).  Our internal
'best practice' now on a 100 series FBM 'failure' is to replace all
three components (FBM, nosecone, and letterbug) and then eeprom
update/download to put it back in to service.  We've found that this is
the surest quickest way to get the FBM back online, while at the same
time minimizing the time/effort involved and in general reducing the
possibility/concerns about a repeat event.

Considering the criticality of the process, status of 100 series I/O
(reference lifecycle policy, Mature Phase), our remote location, site
history, and operations desires, we increased the number of FBM04/01s we
wish to keep as on-site spares, 'just in case' of we have multiple
'failures' within a short period of time and do not have enough spares
on hand to recover.  Note: This is in addition to FBM access we enjoy
through the Module Exchange Program included with the annual
maintenance/support agreement we pay for.=20

We have been unable to convince Invensys Foxboro to cover the remaining
items under Module Exchange Program, as their autopsies do not confirm a
failure mode and they have said (as recently as last week) that they
have not experienced similar failure rates elsewhere (or on the same
range of serial numbers).  Maybe it is something more nosecone or
control room/installation specific, though the next opportunity we'll
have to power down the cabinets and disassemble things for closer
inspection will be at Turnaround 2009 (we may decide to replace 1x8s
mounting structures, cabling, etc. if deemed necessary by that time).

Our latest 'failure' (FBM04) was two weeks ago.  Not surprisingly,
hydrocracker Operations have become increasingly nervous about the
remaining 'unfailed' modules and want them replaced regardless ASAP (on
our own scheduled terms, as opposed to the unscheduled 'failure' usually
on night shift or during the weekend).  It has become increasingly
difficult to reason otherwise.

Our management has determined that we can not afford to take a wait and
see approach anymore.  Yesterday we issued a Purchase Order to replace
outright the remaining FBM04s, 01's and nosecones in question.  One
could view this as cheap insurance (considering alternative scenarios);
but only time will tell.  With any luck, some day we'll have extras to
sell on eBay.

I am also very interested to hear more about:

    1) Other sites' experiences with 100 series FBM 'failures' (types,
       quantities, 'failure' modes) and=20

    2) Invensys Foxboro's plan for 200 series (in a 100 series form
factor)
       replacements...


Ted Johnson
Process Control Engineer
Tesoro Refinery
Kenai, AK


-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx]
On Behalf Of dirk.pauwels@xxxxxxxxxx
Sent: Friday, October 19, 2007 5:44 AM
To: foxboro@xxxxxxxxxxxxx
Subject: Re: [foxboro] 100 Series FBM Reliability Data

I did not hear of it, but it seems like the logical way to go....after
all,
our systems are running  24/7, replacing an FBM100 by an FBM200 required
a
new baseplate, thus most of the time also a new cabinet......

Although it would be a good thing for emergency replacement for those of
us
who do not have spare FBM100's , I would not consider replacing all of
my
FBM100's by FBM200 in this way, because if we do a complete upgrade to
new
FBM's I would prefer new baseplates & re-wire new cabinets, just to get
rid
of the old & dusty wiring mess in the old cabinets...... We have some
new
FBM200 cabinets and it's so much easier for everyone to work on

Dirk Pauwels - DCS/MOC coordinator
Engineering dept.
Hexion Specialty Chemicals
E mail: dirk.pauwels@xxxxxxxxxx
T.  +32.(0)3.570.95.97
F.  +32.(0)3.570.16.09
Mob. +32.(0)497.428.300


=20

             "Moore, Kenneth,

             Celanese/US"

             <ken.e.moore@cela
To=20
             nese.com>                 <foxboro@xxxxxxxxxxxxx>

             Sent by:
cc=20
             foxboro-bounce@fr

             eelists.org
Subject=20
                                       Re: [foxboro] 100 Series FBM

                                       Reliability Data

             19-10-2007 14:39

=20

=20

             Please respond to

             foxboro@freelists

                   .org

=20

=20





 My sales rep. mentioned that Foxboro was working on a project to use
200 series hardware in a 100 series form factor. That way existing
customers could replace defective 100 series modules with 200 series
electronics, and not have to re-wire. Has anyone else heard of this
project?=3D20




Ken Moore
Celanese
Enoree, SC


_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html

foxboro mailing list:             //www.freelists.org/list/foxboro
to subscribe:         =
mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Djoin
to unsubscribe:      =
mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Dleave



=20
=20
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
=20
foxboro mailing list:             //www.freelists.org/list/foxboro
to subscribe:         =
mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Djoin
to unsubscribe:      =
mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Dleave
=20
 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             //www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
 

Other related posts: