Re: [foxboro] PI I/A Interface and FoxAPI "Bad Input" Fix + CP60 reboot
- From: tom.vandewater@xxxxxxxxxxxxxx
- To: foxboro@xxxxxxxxxxxxx
- Date: Tue, 12 Apr 2005 14:03:36 -0400
Thanks Alex!
The document that we were looking at said that FoxAPI Ver. 4.3 was
still in Beta testing. Is that still true?
In regard to using rsom on 21CP03 before turning points off and on
Scan from the PI interface, we have done this a hundred times with no
incidents before. I use som on the FOXAPI host and rsom on the CP's
extensively to understand object manager issues but alas, I did not use it
this time, and wish I had. It was one of those late Friday afternoon
decisions;)
BTW, do you know a way to execute som and rsom in a way that dumps
all opdb and opvr data to a text file for easy import to a spreadsheet? I
would find this very useful.
Also, how useful have the dump files been in identifying reasons for
CP errors like we experienced. Do Foxboro still analyze them to establish
root cause of failures?
Again, Thanks for your rapid feedback.
Tom V.
-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx] On
Behalf Of Johnson, Alex (Foxboro)
Sent: Tuesday, April 12, 2005 1:48 PM
To: foxboro@xxxxxxxxxxxxx
Subject: Re: [foxboro] PI I/A Interface and FoxAPI "Bad Input" Fix + CP60
reboot
Tom,
Two points:
1) The latest FoxAPI V4.3. Contact the CSC for a copy.
2) There is one step that should have been taken before opening
and closing the PI lists - Use rsom to review the communication
resource availability.
I suspect that something was consuming communication resources in the
CP and not releasing them. It might be worth reviewing this periodically.
I can't prove it, of course.
Regards,
Alex Johnson
Invensys Process Systems
Invensys Systems, Inc.
10707 Haddington
Houston, TX 77043
713.722.2859 (voice)
713.722.2700 (switchboard)
713.932.0222 (fax)
ajohnson@xxxxxxxxxxx
-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx] On
Behalf Of tom.vandewater@xxxxxxxxxxxxxx
Sent: Tuesday, April 12, 2005 12:39 PM
To: foxboro@xxxxxxxxxxxxx
Subject: Re: [foxboro] PI I/A Interface and FoxAPI "Bad Input" Fix + CP60
reboot
Hi List,
Thanks to Philip Pulas for his contribution about the Fox IA/OSI PI
interface issues. It was very timely, as we have seen some of the same
issues. As a result, we are going to:
1. Upgrade from 4.2.2 to the current version of the FoxAPI (4.2.8)
2. Upgrade from 2.2.4 to the current version of the Foxboro I/A Interface
(fxbais) 2.2.7.35. This version has some fixes for handling bad inputs.
3. Install the quick fix "QF1005118 - om_server initial scan update fix"
dated 11/05/03.
4. Modify our OSI PI lists to insure that no more than 250 values exist per
CP in any one PI list.
The following scenario occurred on Friday, Apr. 8th, and I am
currently writing up a CAR to dig deeper into the instance, but am
interested in seeing if any other users have experienced anything like this.
It is either an extreme coincidence or a potential bug/loophole in the
interface!
At 4:30pm on Friday afternoon, (as is always the case with serious
system problems), the PI administrator called me and said that PI was
receiving "Bad Input" from the Fox IA system on several critical points used
for mass balance calculations. The points went bad at 11:05am the same day.
Upon further investigation I discovered that all of the data points were
coming from one CP. I looked through our System Monitor logs to see if
there had been any issues with the CP in question, (a FT CP-60). There had
been no system alarms. I looked at the station loading compound and the CP
wasn't heavily loaded or experiencing any overruns. Finally I looked at the
data points PI was having trouble with and the values were still changing
and the IA historian was collecting data for them.
Together, we decided to turn the PI points "Off Scan" and then back
on again to correct the comm error, (this has worked in the past). When the
PI Administrator turned the points "Off Scan" I noticed the IA screen I was
looking at, turned CYAN and the Sys top menu bar key began to flash red. By
the time I opened SysMgmt the CP-60 was red and then turned yellow showing
that both FT CP's had failed and then one began rebooting. I watched as all
of the ECB's/FBM's were brought back online and finally the control returned
and I could again call up C:B:P values on the DM. The other CP60 did a
memory dump to its boot host, (dump is found at the following location):
/usr/fox/sysmgm/softmgr/dump/21CP03.1 through 21CP03.8
and the shadow finally rebooted, which took about 36 minutes to complete as
can be seen below in the list of System alarms received.
The CP's most recent event captured in our System Monitor logs that
was related to this CP was a checkpoint that occurred just over 24 hours
before the unplanned CP reboot:
04-07-05 14:01:23 0 SYSMON = SYS21A 21CP03 Software Manager
SYSMON -00021 Checkpoint Successful
As a result, the CP booted with all of the setpoints and block valve
positions that were current when the checkpoint occurred a day before the
reboot. As one can imagine, things had changed and the processes controlled
by this CP were bumped when the reboot occurred. Lucky for us the processes
controlled by this CP weren't some of the more hazardous ones in our plant.
We haven't had a simultaneous failure of both FT CP's in several years and
this is a very disturbing occurrence that we can't ignore. As I said, I am
writing a CAR and will send the dump files to Foxboro for analysis, but am
interested to see if any other users have experienced anything like this in
conjunction with turning PI points off scan?
Thanks,
Tom VandeWater
Control System Developer/Analyst
Dow Corning Corporation
Carrollton, KY USA
04-08-05 16:46:16 0 SYSMON = SYS21A 21CP03 Error Protocol
FTXSS 000080 Memory violation occured 00006C0FAF69
04-08-05 16:46:16 0 SYSMON = SYS21A 21CP03 Error Protocol
FTXSS 000080 Memory violation occured 00006C0FA3EF
04-08-05 16:47:53 0 SYSMON = SYS21A 21CP03 Software Manager
SYSMON -00003 Powerup reboot OK. ROM Addr = 00006C0FA3EF
04-08-05 16:48:01 0 SYSMON = SYS21A 21CP03 Station
SYSMON -00041 Equipment on-line
04-08-05 16:48:12 0 SYSMON = SYS21A 21CP03 Software Manager
SYSMON -00019 Database Load Successful
04-08-05 16:48:12 0 SYSMON = SYS21A 21CP03 Process = downld
CIO_DB 000001 DATABASE DOWNLOAD: RESOLVE LINKAGES SUCCESSFUL
04-08-05 16:48:43 0 SYSMON = SYS21A 21CP03 Process = downld
CIO_DB 000003 DATABASE DOWNLOAD COMPLETE
04-08-05 16:48:55 0 SYSMON = SYS21A 21CP03 Equip = 21CP03
SYSMON -00051 Equipment has been added on-line
04-08-05 16:48:55 0 SYSMON = SYS21A 21CP03 Equip = 21A100
SYSMON -00051 Equipment has been added on-line
04-08-05 16:48:55 0 SYSMON = SYS21A 21CP03 Equip = 21A101
SYSMON -00051 Equipment has been added on-line
..... and all of the rest of the FCM's and FBM's successfully came on line!
04-08-05 17:10:36 0 SYSMON = SYS21A 21CP03 Software Manager
SYSMON -00023 Memory Dump Successful. File name = 21CP03.*
04-08-05 17:12:05 0 SYSMON = SYS21A 21CP03 Software Manager
SYSMON -00003 Powerup reboot OK. ROM Addr = 00006C0FAF69
04-08-05 17:12:13 0 SYSMON = SYS21A 21CP03 Fault Tolerant Exec
SM_MSG -00046 Fault Tolerant Modules Now Married
04-08-05 17:12:15 0 SYSMON = SYS21A 21CP03 Equip = 21CP03
SYSMON -00056 Currently using PIO bus A
-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx] On
Behalf Of Pulas, Philip (GE Contractor)
Sent: Wednesday, April 06, 2005 11:13 AM
To: foxboro@xxxxxxxxxxxxx
Subject: [foxboro] PI I/A Interface and FoxAPI "Bad Input" Fix
All,
Since I have noticed a lot of PI users out there, I just thought I'd
pass along this fix (at the bottom of this message) in case anyone
hasn't seen it yet. This is from OSIsoft's tech support website. We
have been experiencing this same issue for some time with one I/A system
and the fixes so far haven't really solved it, but this sounds like it
will.
Thanks,
Philip Pulas
Tesoro Corporation
Golden Eagle Refinery
Martinez, CA
Foxboro I/A Interface (fxbais) and FoxAPI "Bad Input" issue
04-Apr-2005
There is a known problem with the PI Foxboro I/A interface (fxbais) and
the FoxAPI that only affects I/A series control systems running version
6.5.1.
The Issue:
The issue is that for some of the lists, the FoxAPI does not receive an
initial value for the objects and returns a status of zero, which is
sent to the PI server as a "Bad Input." When the object changes, the
update goes to the FoxAPI and the interface sends the correct value to
PI. But, for values that don't change often (i.e. setpoints), the PI tag
can show "Bad Input" for a long time.
The issue appears to have been resolved with a Quick fix from Foxboro
(QF1005118). The description of the bug from Foxboro is:
"FM#27455/CAR1005118
The Foxapi uses OM open list functions add/activate to open data sets.
If Foxapi does an activate which involves more than 23 points, the
initial scan update for points after the 23rd may be missed.
The fix is not to the FoxAPI itself, but to the om_server process
(Object Manager) that sits under the FoxAPI."
Determining if you need the Quick Fix:
If you are getting a lot of "Bad Input" values from the Foxboro
interface, make sure:
1. You are running the current version of the FoxAPI (4.2.8)
2. You are running the current version of the Foxboro I/A Interface
(fxbais) 2.2.7.35. This version has some fixes for handling bad inputs.
3. If both are current and you are still getting the Bad Inputs, then
you should contact Foxboro Customer Support and ask for the quick fix
"QF1005118 - om_server initial scan update fix" dated 11/05/03.
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
foxboro mailing list: //www.freelists.org/list/foxboro
to subscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
foxboro mailing list: //www.freelists.org/list/foxboro
to subscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
foxboro mailing list: //www.freelists.org/list/foxboro
to subscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe: mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
Other related posts:
