Re: [foxboro] network storm

  • From: Michael Jaudon <mjaudon2@xxxxxxxxx>
  • To: foxboro@xxxxxxxxxxxxx
  • Date: Fri, 6 Feb 2009 19:37:44 -0600

Just to update everyone about the "network storm" issue.
We had another *event* today while we were at the root switches.  Again
cycling the power corrected the problem (this time in less than 4 minutes).

No electrical cord was anywhere near that MCC.  (We can rule that one out.)

We have new switches ordered (for a lack of anything else to try).

We checked the configuration and looked for errors on each root switch
utilizing FoxMass' instructions.  Nothing is showing up (everything is
normal).  The "sys_mon.log" file does not show anything leading up to the *
event*.

I may have to re-think the star architecture because the entire DCS Network
is dependent on the ROOT SWITCHES.

My advice to everyone who has a MESH network is 'you should at least have a
spare root switch'.

I will try to keep everyone posted as to what was the cause and the
solution.

Gaylon, I did not have time to call today but I will be talking to you at
the SEUG meeting, provided we resolve this problem by then.

Later,

On Fri, Feb 6, 2009 at 7:00 AM, Doucet, Terrence <tdoucet@xxxxxxxxxxxxxxxxxx
> wrote:

> Mike,
>
> The extension cord if used with a drill or saw or similar tool, could have
> been the cause of electrical noise entering your network. The Foxboro
> signals themselves are fibre so thus immune, but the switches are high speed
> computing devices that are susceptible to electrical noise. High frequency
> noise on the neutral could pass right through the power supply section of
> the switch and saturate the electronics.
>
> If you want to read about just how weird electrical noise can be, read
> through NPS-EC-07-002 (USA Naval Post-Graduate school) in Section 3.1.5
> where they discuss how a defective utility pole (wood) caused a problem at a
> naval radio station, two miles down the road!
>
> http://www.nps.edu/Research/publications/07techrpt.html
>
> I suspect you can easily justify a ban on the practice of using tools in
> the vicinity of your switches at your facility.
>
>
> Terry Doucet, Eng.
>
>
> -----Message d'origine-----
> De : foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx] De
> la part de Michael Jaudon
> Envoyé : February 5, 2009 6:46 PM
> À : foxboro@xxxxxxxxxxxxx
> Objet : Re: [foxboro] network storm
>
> Terry,
> Nobody was doing any work on the DCS at all.
>
> The only thing we have found, at this point, is a contractor had an
> extension cord plugged into a wall outlet located in the room with the root
> switch.  He was using this outlet during the time period we had trouble.
> I'm not an electrical guy but I'm told that the "neutral" is common with
> the
> UPS circuit and with the "dirty" regular power (sorry about that Gaylon)
> whatever that means.
>
> I know that's a stretch but nothing else at this point is obvious.
>
> Gaylon,  I got your document that was posted on the Cassandra website and
> it
> has been thoroughly discussed.  I may give you a call tomorrow...if not
> I'll
> see you at the SEUG meeting.
>
> All is well at the moment but everyone is uneasy.
>
> Thanks for everyone's input.
>
> Mike Jaudon
> Tronox, LLC
>
> On Thu, Feb 5, 2009 at 3:58 PM, Doucet, Terrence <
> tdoucet@xxxxxxxxxxxxxxxxxx
> > wrote:
>
> > Mike,
> >
> > Cycling power gets you out of trouble relatively quickly but you still do
> > not know what caused the trouble. Was there any work going on with any of
> > the switches or devices connected to the MESH? Gaylon mentions the Loop
> > Detection Policy. If this is not enabled you can easily trigger a storm
> by
> > interposing a HUB between a station on your MESH and then "accidentally"
> > connecting a second port on your HUB back to the MESH. But this would
> kill
> > you right away, so unless someone was playing with cables, that is not
> the
> > cause.
> >
> > Was anyone testing a new program on an AW? If they were performing a
> whole
> > lot of omget's (or the AIMAPI equivalent) without pausing, that might get
> > you a storm.
> >
> > Terry Doucet, Eng.
> >
> >
> >
> > -----Message d'origine-----
> > De : foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx]
> De
> > la part de Hicks, Gaylon F
> > Envoyé : February 5, 2009 4:00 PM
> > À : foxboro@xxxxxxxxxxxxx
> > Objet : Re: [foxboro] network storm
> >
> > Hi Mike,
> >
> > I am Chicken Little no more!  This happens to be one of my favorite pet
> > peeves and soapbox issues with the Foxboro mesh network.  Boring
> > discussion below.
> >
> > There are things you can do in the switches to reduce your vulnerability
> > to these network storms.  One is to configure all your broadcast packet
> > limits on all 100MB ports to 500 packets/second, and if your switches
> > will support it, implement the Loop Detection Policy (LDP) algorithm.
> > There is also a multicast packet suppression policy you can implement,
> > but Foxboro does not officially support that.  And (my personal
> > favorite) you can implement VLANs to segment your ZCP field bus from
> > your ZCP Control Network.  Also not officially support, but useful.
> >
> > One thing we have seen with the ZCPs while doing network storm testing
> > is that the ZCP/FCM communications start to get sloooooow, and I/O
> > updates slow down.  Depending on your process, this can be a problem.
> > We'll probably discuss this issue at the SEUG meeting next week.  Are
> > you going to make it?  Either way, feel free to give me a call or email
> > if you want to talk about this in more detail.
> >
> > Good luck with everything.
> >
> > Thanks,
> > Gaylon Hicks
> > TVA - Browns Ferry
> >
> >
> > -----Original Message-----
> > From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx]
> > On Behalf Of Michael Jaudon
> > Sent: Thursday, February 05, 2009 1:27 PM
> > To: foxboro@xxxxxxxxxxxxx
> > Subject: [foxboro] network storm
> >
> > We experienced a first for our plant site today.  We had all
> > workstations,
> > on the mesh side of the network, to "smurf" out.  Upon visually checking
> > our
> > ZCP270s I noticed that of each of the ZCP270s had gone single with it's
> > associated FT module gone to red/green.
> > Looking at the switches...we have 3 sets of switches including the root
> > switch.  On each of the switches the lights were all flashing rapidly
> > and in
> > unison.  It wasn't until we cycled the power on the root switches (one
> > at a
> > time) that the network came back.
> >
> > Talking to our field service rep...he said the situation we had was a
> > "network storm".  The cause is unknown.
> >
> > My question is has anyone ever experienced a "network storm" on a mesh
> > network and if so what caused it?
> >
> > Our mesh network has been in operation since Oct 2008 and this is the
> > first
> > issue we have had with it.
> >
> > We are running a Mesh Network version 8.4.1.
> >
> > --
> > Mike Jaudon
> > Tronox, LLC
> > Hamilton, MS
> >
> >
> >
> >
> > _______________________________________________________________________
> > This mailing list is neither sponsored nor endorsed by Invensys Process
> > Systems (formerly The Foxboro Company). Use the info you obtain here at
> > your own risks. Read http://www.thecassandraproject.org/disclaimer.html
> >
> > foxboro mailing list:             //www.freelists.org/list/foxboro
> > to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> > to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
> >
> >
> >
> > _______________________________________________________________________
> > This mailing list is neither sponsored nor endorsed by Invensys Process
> > Systems (formerly The Foxboro Company). Use the info you obtain here at
> > your own risks. Read http://www.thecassandraproject.org/disclaimer.html
> >
> > foxboro mailing list:             //www.freelists.org/list/foxboro
> > to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> > to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
> >
> >
> >
> > _______________________________________________________________________
> > This mailing list is neither sponsored nor endorsed by Invensys Process
> > Systems (formerly The Foxboro Company). Use the info you obtain here at
> > your own risks. Read http://www.thecassandraproject.org/disclaimer.html
> >
> > foxboro mailing list:             //www.freelists.org/list/foxboro
> > to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> > to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
> >
> >
>
>
> --
> Mike Jaudon
>
>
>
> _______________________________________________________________________
> This mailing list is neither sponsored nor endorsed by Invensys Process
> Systems (formerly The Foxboro Company). Use the info you obtain here at
> your own risks. Read http://www.thecassandraproject.org/disclaimer.html
>
> foxboro mailing list:             //www.freelists.org/list/foxboro
> to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>
>
>
> _______________________________________________________________________
> This mailing list is neither sponsored nor endorsed by Invensys Process
> Systems (formerly The Foxboro Company). Use the info you obtain here at
> your own risks. Read http://www.thecassandraproject.org/disclaimer.html
>
> foxboro mailing list:             //www.freelists.org/list/foxboro
> to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>
>


-- 
Mike Jaudon

 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             //www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave

Other related posts:

  1. Home
  2. foxboro
  3. Archive
  4. 02-2009