Re: [foxboro] Automation of upload/checkpoint/save_all at sites with IACC

This is a very good and useful discussion.  As a very small IA system
end user that has recently moved to V8-Mesh and IACC I am monitoring the
responses with great interest.  We are still trying to develop a
complete backup strategy and determine if we truly have ourselves
completely covered.  I will throw out a few thoughts and ideas to fuel
the discussion.  Again, my disclaimer is that we have a very small
system (2 ZCP FT pairs, 3 P92 WPs, 1 P91 Server), so you real DCS users
can take this with a grain of salt.  Of course, I hope that in learning
what works for you folks will allow us to develop procedures that will
also work for us on a much smaller and easier to implement scale.

Our current practice for control database backup is to simply do an IACC
database backup on a weekly basis and then keep these incremental
backups off platform on our corporate network server where they will
also be backed-up by our IT dept.  We did the same thing with disk
images of save-alls on our old V6-Nodebus ICC system.  We have always
realized that, without doing an upload, the parameters that have been
changed outside of the control configurator would not be saved in the
backup.  Our solution to this has always been procedural.  We require
any person making true control changes, either in or outside the control
configurator, to record those changes on an MOC document.  If the
changes were made outside of the configurator and are to be permanent
(alarm points, tuning parameters, constant values in Calc blacks, etc),
these changes must be entered into and downloaded from the configurator
when possible so they become a permanent part of the control database.
The MOC document is then updated accordingly.  We follow the same
procedure for our PLC based control systems.  At our level this has
remained a workable system, but I am already seeing the associated
overhead increasing as our system expands (and have discovered instances
where this procedure was not followed).

There are many block parameters that are inherently dynamic during
operation and do not necessarily represent a true change to the control
configuration and may not even be desirable to upload into the database.
Our IA system runs a batch process operation.  Parameters such as
setpoints, calculated block output values, and even alarm points and
inhibits (INHALM), will change during different portions of a batch
and/or for different products.  Many Sequence blocks (DEP, MON Cases,
EXC) are only active during particular phases of the batch, and sections
of continuous block logic may also be inactive or bypassed (such as CALC
block steps).  We actually ran into some trouble a number of years ago
when all parameters were uploaded to the then ICC database while a batch
was in progress.  When a Sequence block was modified and downloaded at a
later date some of the BOs that controlled discrete devices and were ON
at the time of the upload were set during the download turning on the
devices - surprise!  Because this was a DEP "Phase" block the initial
state is inactive and the logic that would normally have controlled
these outputs was not running.  We have since tried to address our logic
design and procedures, but it is still something to be aware of.

As Tim states below it is possible to do an upload and compare / sync to
CP function in IACC but this is a time consuming manual process.  It can
be done at the compound level in individual CPs which does not take
exceedingly long, and I have made use of this to compare and identify
specific changes.  Again, this would not be practical for any complete
system.  We are currently looking at expanding our system - adding
another ZCP pair and 2 more workstations (wow!).  We are taking a look
at a 3rd party configuration and documentation management tool, Foxray
from Limeware, Inc., that appears to have the capability to track and
document changes and identify discrepancies between the control database
and actual CP images.  This would not address the automated upload and
backup procedure, but hopefully could address our documentation
procedure and identify places where it was not followed or updated.  Any
thoughts or experience from the group?

Tim's idea laid out below sounds like it may be feasible, and I am
waiting to hear responses from other more experienced IA users than
myself. One thought I have is that IACC can import the Savealls created
by the automated ICC driver task script.  We had to do this when we
ported from ICC to IACC.  It was ugly at the time because we had not
defined LoopIds in ICC so all blocks imported into single, large CSDs
based just on the compounds.  It took a long time to organize these into
functionally logical CSDs (which is one of the big advantages I see to
IACC).  However, if the control database was originally created in IACC
the blocks should be able to be imported back into their original CSDs
using the LoopId if it was left as the IACC default CSD Name.  This
could be a way of bringing the IACC database up to the current CP image
in the case of a catastrophic failure recovery as described by Tim
below.  This may or may not be any quicker or less labor intensive than
stepping through the Upload / Compare / Sync function.  Especially if
someone actually took the time to neatly arrange the blocks and
connections in the CSD drawings (which we gave up on).  Just a thought.

Please keep the discussion going folks.  I'm counting on you.


Tom Badura
Plastics Engineering Company
920-458-2121 x3366
tbadura@xxxxxxxxxx



 


-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx]
On Behalf Of Lowell, Timothy
Sent: Wednesday, October 29, 2008 6:34 PM
To: foxboro@xxxxxxxxxxxxx
Subject: [foxboro] Automation of upload/checkpoint/save_all at sites
with IACC

After Tom Vandewater's question on automating uploads, checkpoints, and
save_alls using IACC was met with stony silence, I'm assuming it can't
be done.  This is not an acceptable situation in my opinion, since these
tasks need to be part of a sensible backup strategy.  Simply backing up
the IACC database without doing regular uploads is not a good strategy,
since many important parameters, such as tuning and alarm settings, can
and will be modified via the detail displays or other displays, and
these changes need to be recorded and saved in a regular backup file.
Apparently, the only way to do uploads with IACC is to click on a button
using the GUI, and then wait 3 to 4 hours while the upload occurs, with
IACC being unavailable the whole time.  This is not practical when you
have 20 or 30 or more CP's to upload.
 

Given this state of affairs, I'd like to propose a backup regime that,
while not being ideal, at least accomplishes the task of providing a
regular unattended save_all backup with uploads that can be used in case
of emergency.  I'd like all of you out there to poke holes in this and
tell me why this wouldn't work, or even better, how to make it work.

 

First of all, you would use the save_all.sh script at
TheCassandraProject.org  that has been tried and tested on systems with
ICC for years.  It would run on weekends or late at night, as it is
typically set up to do, using either crontab of Windows Task Scheduler.
I know that it says specifically NOT to use it on IACC systems in the
manual, but I don't see why it wouldn't work.  Even with IACC present,
you can still do uploads, checkpoints, and save_alls using iccdrvr.tsk.
I just tested it today, and it works fine.  The caveat is that you can't
have someone using IACC while you do that, which if you schedule the job
to run in the middle of the night or on a weekend, the chances of that
are slim.

 

Using save_all.sh will do the normal upload, checkpoint, and save_all
(and optional shrink) of the CP database, like we've all been doing
since the beginning of I/A time.

 

Of course, IACC will know nothing about any of this, and as far as I can
tell, does not need to.  The purpose of the upload/checkpoint/save_all
is to create an "emergency" copy of the CP database, updated with all
the latest information.  You would only use it if your CP was
irretrievably hosed, and you had to initialize and re-load it, or you
lost the host hard drive along with the checkpoint and work files.  You
would never, ever use it for anything else.

 

IACC has its own database or databases (which you would backup
separately).  This database has all the parameters available for you to
edit, but you generally do not use IACC to edit things like tuning
parameters, and if you do, you will overwrite the current value when you
download anyway.  More typically, you use IACC to modify block
connections or CALCA block steps or add and delete blocks.  None of
these things are affected by the upload process.  As soon as you
download IACC changes, they become part of the workfile and are
immediately checkpointed, but they don't affect other parameters that
were not changed.

 

As you go along, week by week, doing automated uploads and save_alls,
the IACC database will become increasingly different from the CP because
of the upload process, but I don't really see how this is a problem,
since the only things that will be different will be things you
typically don't change using IACC anyway.

 

If in the event of a catastrophic CP failure, you would use your
uploaded save_all file created by save_all.sh as the load_all to the
failed CP.  After you finished loading the CP, you would go into IACC
and do a "Compare to CP".  This process would identify any differences
and allow you update the IACC database with those differences.

 

If the hard drive failed, you would take your backup copy hard drive
image that you created using Symantec Backup Exec Save and Restore (or
something similar) and re-create the image.  This re-created image would
include, presumably, the latest save_alls you created using save_all.sh,
as well the IACC database(s), and really nothing will have changed,
since the CP will continue to be cranking along using its current image.
You'd lose any changes since the last backup, but that's what happens in
that scenario no matter what you do.  And of course, you still can't
recreate your workfile from the running CP image, after decades of
asking for that feature, and having IACC or ICC won't make any
difference.

 

If the CP double-reboots for some reason and needs to reload its
checkpoint file, at least this way you have an uploaded version of the
checkpoint file.  If you went the IACC GUI-only route and never bothered
to upload because it can't be automated and takes too long, you'd be
stuck with a non-uploaded checkpoint file that doesn't have any new
tuning parameters, setpoint changes, or alarm settings, etc.

 

I know there a million scenarios that need to be considered, and that's
why I'm putting this forth as a proposal to be shot down.  Here's one:
What if you add a block with IACC, download it to the CP, and then the
CP crashes and you have to rebuild it?  Can you take the old save_all
you did last night and do a load_all, and reconcile it with IACC using
Compare to CP, or will you get some kind of crazy error because IACC has
a block that the CP doesn't have?   How about if you DELETE a block
using IACC, and the CP crashes?  Can you take your day-old save_all, do
a load_all, and reconcile with IACC, or will IACC say that block you
deleted is still on the CP and you can't get rid of it?  I don't know
the answers, but I'd like to find out.

 

Tim Lowell

Control Systems Engineer

Tesoro Petroleum Company

210-283-2929 (w)

210-253-0225 (c)

tlowell@xxxxxxxxxxx

 


 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             http://www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
 
 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             http://www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave

Other related posts: