This topic strikes a nerve at our facility. =20 We have experienced multiple FBM 100 series 'failures'; however only in our hydrocracker area. The breakdown is as follows: 9 FBM04s (out of 26, or 35%) and 3 FBM01s (out of 8, or 37.5%) since 2003. They are all the same vintage (installed in 1997) and the problem can't be isolated to a single CP/cabinet/1x8/slot. We have not seen incidents with other 100 series FBM types (FBM02, FBM041) in this area, nor other FBM04/01 failures in another operating area (different control room, UPS power feed, and vintage of FBMs); or 0%. In 2004/5 we attempted to formally address this. Together with Invensys Foxboro field service we inspected the power, grounding, cabling, and fieldbus terminations. I recall we did find a couple of minor things at that time, but nothing I would call major discoveries. Since then on our own we tightened field wiring terminations and termination cable assemblies (nosecones); monitored power supply voltages, etc. In 2005/6, we upgraded workstations/software (51F @ V7.1.1), CPs (30/40s to 60's w/fiber hubs) and added P91 (@ V8.1) to host the Mesh and replace CLAN. In 2007 we added FCP270s and 200 series I/O. But to no avail, we have experienced additional FBM04/01 'failures' during this overall time frame. Engineering evaluations on most returned modules have been conducted by Invensys, but they have not been able to confirm any failure mode (i.e. "no problem found"). We seem to be missing something, somewhere...=20 In our case when an FBM04/01 'fails', it goes offline (red/green or dead red) and will not reset (pull-push nosecone/fbm/lettterbug, and/or eeprom update/download by themselves will not cause it to come back on-line). In the case of FBM04s (output channels connected to control valves) this causes a major process upset: the board operator can't see what is going on (points are smurfed on graphics) and they lose control. They must quickly scramble to place other operators in the field who put things in manual/bypass, closely monitor local indicators, and make adjustments to manual valves (at the direction of main board operator via radio) - until the FBM issue is resolved, things can be put back in automatic, and the process settles out. Needless to say but during this time (because of critical temperatures and pressures associated with the hydrocracker) adrenaline levels and emotions run higher.=20 On initial earlier incidents, we tried replacing only the FBM but occasionally a problem would reoccur (seems there was still a problem with the nosecone?). Other times then we replaced the FBM and nosecone, but seemingly still had a problem with a letterbug (what can go wrong with those?). Of the 12 'failures' we have experienced to date, only one was a repeat failure (though we are not certain that all fbm, nosecone, and letterbug were replaced the first time). Our internal 'best practice' now on a 100 series FBM 'failure' is to replace all three components (FBM, nosecone, and letterbug) and then eeprom update/download to put it back in to service. We've found that this is the surest quickest way to get the FBM back online, while at the same time minimizing the time/effort involved and in general reducing the possibility/concerns about a repeat event. Considering the criticality of the process, status of 100 series I/O (reference lifecycle policy, Mature Phase), our remote location, site history, and operations desires, we increased the number of FBM04/01s we wish to keep as on-site spares, 'just in case' of we have multiple 'failures' within a short period of time and do not have enough spares on hand to recover. Note: This is in addition to FBM access we enjoy through the Module Exchange Program included with the annual maintenance/support agreement we pay for.=20 We have been unable to convince Invensys Foxboro to cover the remaining items under Module Exchange Program, as their autopsies do not confirm a failure mode and they have said (as recently as last week) that they have not experienced similar failure rates elsewhere (or on the same range of serial numbers). Maybe it is something more nosecone or control room/installation specific, though the next opportunity we'll have to power down the cabinets and disassemble things for closer inspection will be at Turnaround 2009 (we may decide to replace 1x8s mounting structures, cabling, etc. if deemed necessary by that time). Our latest 'failure' (FBM04) was two weeks ago. Not surprisingly, hydrocracker Operations have become increasingly nervous about the remaining 'unfailed' modules and want them replaced regardless ASAP (on our own scheduled terms, as opposed to the unscheduled 'failure' usually on night shift or during the weekend). It has become increasingly difficult to reason otherwise. Our management has determined that we can not afford to take a wait and see approach anymore. Yesterday we issued a Purchase Order to replace outright the remaining FBM04s, 01's and nosecones in question. One could view this as cheap insurance (considering alternative scenarios); but only time will tell. With any luck, some day we'll have extras to sell on eBay. I am also very interested to hear more about: 1) Other sites' experiences with 100 series FBM 'failures' (types, quantities, 'failure' modes) and=20 2) Invensys Foxboro's plan for 200 series (in a 100 series form factor) replacements... Ted Johnson Process Control Engineer Tesoro Refinery Kenai, AK -----Original Message----- From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx] On Behalf Of dirk.pauwels@xxxxxxxxxx Sent: Friday, October 19, 2007 5:44 AM To: foxboro@xxxxxxxxxxxxx Subject: Re: [foxboro] 100 Series FBM Reliability Data I did not hear of it, but it seems like the logical way to go....after all, our systems are running 24/7, replacing an FBM100 by an FBM200 required a new baseplate, thus most of the time also a new cabinet...... Although it would be a good thing for emergency replacement for those of us who do not have spare FBM100's , I would not consider replacing all of my FBM100's by FBM200 in this way, because if we do a complete upgrade to new FBM's I would prefer new baseplates & re-wire new cabinets, just to get rid of the old & dusty wiring mess in the old cabinets...... We have some new FBM200 cabinets and it's so much easier for everyone to work on Dirk Pauwels - DCS/MOC coordinator Engineering dept. Hexion Specialty Chemicals E mail: dirk.pauwels@xxxxxxxxxx T. +32.(0)3.570.95.97 F. +32.(0)3.570.16.09 Mob. +32.(0)497.428.300 =20 "Moore, Kenneth, Celanese/US" <ken.e.moore@cela To=20 nese.com> <foxboro@xxxxxxxxxxxxx> Sent by: cc=20 foxboro-bounce@fr eelists.org Subject=20 Re: [foxboro] 100 Series FBM Reliability Data 19-10-2007 14:39 =20 =20 Please respond to foxboro@freelists .org =20 =20 My sales rep. mentioned that Foxboro was working on a project to use 200 series hardware in a 100 series form factor. That way existing customers could replace defective 100 series modules with 200 series electronics, and not have to re-wire. Has anyone else heard of this project?=3D20 Ken Moore Celanese Enoree, SC _______________________________________________________________________ This mailing list is neither sponsored nor endorsed by Invensys Process Systems (formerly The Foxboro Company). Use the info you obtain here at your own risks. Read http://www.thecassandraproject.org/disclaimer.html foxboro mailing list: //www.freelists.org/list/foxboro to subscribe: = mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Djoin to unsubscribe: = mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Dleave =20 =20 _______________________________________________________________________ This mailing list is neither sponsored nor endorsed by Invensys Process Systems (formerly The Foxboro Company). Use the info you obtain here at your own risks. Read http://www.thecassandraproject.org/disclaimer.html =20 foxboro mailing list: //www.freelists.org/list/foxboro to subscribe: = mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Djoin to unsubscribe: = mailto:foxboro-request@xxxxxxxxxxxxx?subject=3Dleave =20 _______________________________________________________________________ This mailing list is neither sponsored nor endorsed by Invensys Process Systems (formerly The Foxboro Company). Use the info you obtain here at your own risks. Read http://www.thecassandraproject.org/disclaimer.html foxboro mailing list: //www.freelists.org/list/foxboro to subscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=join to unsubscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave