Re: [foxboro] network storm

  • From: "Boulay, Russ" <russ.boulay@xxxxxxxxxxxxxxxx>
  • To: <foxboro@xxxxxxxxxxxxx>
  • Date: Fri, 6 Feb 2009 21:02:33 -0500

Michael, whoo in FoxMass has looked at your switch information?
 
Of the over 500 Mesh installations we have, 90% of those are STAR topologies.
 
I would be very interested in seeing your info and what has been collected
________________________________

From: foxboro-bounce@xxxxxxxxxxxxx on behalf of Michael Jaudon
Sent: Fri 2/6/2009 8:37 PM
To: foxboro@xxxxxxxxxxxxx
Subject: Re: [foxboro] network storm



Just to update everyone about the "network storm" issue.
We had another *event* today while we were at the root switches.  Again
cycling the power corrected the problem (this time in less than 4 minutes).

No electrical cord was anywhere near that MCC.  (We can rule that one out.)

We have new switches ordered (for a lack of anything else to try).

We checked the configuration and looked for errors on each root switch
utilizing FoxMass' instructions.  Nothing is showing up (everything is
normal).  The "sys_mon.log" file does not show anything leading up to the *
event*.

I may have to re-think the star architecture because the entire DCS Network
is dependent on the ROOT SWITCHES.

My advice to everyone who has a MESH network is 'you should at least have a
spare root switch'.

I will try to keep everyone posted as to what was the cause and the
solution.

Gaylon, I did not have time to call today but I will be talking to you at
the SEUG meeting, provided we resolve this problem by then.

Later,

On Fri, Feb 6, 2009 at 7:00 AM, Doucet, Terrence <tdoucet@xxxxxxxxxxxxxxxxxx
> wrote:

> Mike,
>
> The extension cord if used with a drill or saw or similar tool, could have
> been the cause of electrical noise entering your network. The Foxboro
> signals themselves are fibre so thus immune, but the switches are high speed
> computing devices that are susceptible to electrical noise. High frequency
> noise on the neutral could pass right through the power supply section of
> the switch and saturate the electronics.
>
> If you want to read about just how weird electrical noise can be, read
> through NPS-EC-07-002 (USA Naval Post-Graduate school) in Section 3.1.5
> where they discuss how a defective utility pole (wood) caused a problem at a
> naval radio station, two miles down the road!
>
> http://www.nps.edu/Research/publications/07techrpt.html
>
> I suspect you can easily justify a ban on the practice of using tools in
> the vicinity of your switches at your facility.
>
>
> Terry Doucet, Eng.
>
>
> -----Message d'origine-----
> De : foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx] De
> la part de Michael Jaudon
> Envoyé : February 5, 2009 6:46 PM
> À : foxboro@xxxxxxxxxxxxx
> Objet : Re: [foxboro] network storm
>
> Terry,
> Nobody was doing any work on the DCS at all.
>
> The only thing we have found, at this point, is a contractor had an
> extension cord plugged into a wall outlet located in the room with the root
> switch.  He was using this outlet during the time period we had trouble.
> I'm not an electrical guy but I'm told that the "neutral" is common with
> the
> UPS circuit and with the "dirty" regular power (sorry about that Gaylon)
> whatever that means.
>
> I know that's a stretch but nothing else at this point is obvious.
>
> Gaylon,  I got your document that was posted on the Cassandra website and
> it
> has been thoroughly discussed.  I may give you a call tomorrow...if not
> I'll
> see you at the SEUG meeting.
>
> All is well at the moment but everyone is uneasy.
>
> Thanks for everyone's input.
>
> Mike Jaudon
> Tronox, LLC
>
> On Thu, Feb 5, 2009 at 3:58 PM, Doucet, Terrence <
> tdoucet@xxxxxxxxxxxxxxxxxx
> > wrote:
>
> > Mike,
> >
> > Cycling power gets you out of trouble relatively quickly but you still do
> > not know what caused the trouble. Was there any work going on with any of
> > the switches or devices connected to the MESH? Gaylon mentions the Loop
> > Detection Policy. If this is not enabled you can easily trigger a storm
> by
> > interposing a HUB between a station on your MESH and then "accidentally"
> > connecting a second port on your HUB back to the MESH. But this would
> kill
> > you right away, so unless someone was playing with cables, that is not
> the
> > cause.
> >
> > Was anyone testing a new program on an AW? If they were performing a
> whole
> > lot of omget's (or the AIMAPI equivalent) without pausing, that might get
> > you a storm.
> >
> > Terry Doucet, Eng.
> >
> >
> >
> > -----Message d'origine-----
> > De : foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx]
> De
> > la part de Hicks, Gaylon F
> > Envoyé : February 5, 2009 4:00 PM
> > À : foxboro@xxxxxxxxxxxxx
> > Objet : Re: [foxboro] network storm
> >
> > Hi Mike,
> >
> > I am Chicken Little no more!  This happens to be one of my favorite pet
> > peeves and soapbox issues with the Foxboro mesh network.  Boring
> > discussion below.
> >
> > There are things you can do in the switches to reduce your vulnerability
> > to these network storms.  One is to configure all your broadcast packet
> > limits on all 100MB ports to 500 packets/second, and if your switches
> > will support it, implement the Loop Detection Policy (LDP) algorithm.
> > There is also a multicast packet suppression policy you can implement,
> > but Foxboro does not officially support that.  And (my personal
> > favorite) you can implement VLANs to segment your ZCP field bus from
> > your ZCP Control Network.  Also not officially support, but useful.
> >
> > One thing we have seen with the ZCPs while doing network storm testing
> > is that the ZCP/FCM communications start to get sloooooow, and I/O
> > updates slow down.  Depending on your process, this can be a problem.
> > We'll probably discuss this issue at the SEUG meeting next week.  Are
> > you going to make it?  Either way, feel free to give me a call or email
> > if you want to talk about this in more detail.
> >
> > Good luck with everything.
> >
> > Thanks,
> > Gaylon Hicks
> > TVA - Browns Ferry
> >
> >
> > -----Original Message-----
> > From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx]
> > On Behalf Of Michael Jaudon
> > Sent: Thursday, February 05, 2009 1:27 PM
> > To: foxboro@xxxxxxxxxxxxx
> > Subject: [foxboro] network storm
> >
> > We experienced a first for our plant site today.  We had all
> > workstations,
> > on the mesh side of the network, to "smurf" out.  Upon visually checking
> > our
> > ZCP270s I noticed that of each of the ZCP270s had gone single with it's
> > associated FT module gone to red/green.
> > Looking at the switches...we have 3 sets of switches including the root
> > switch.  On each of the switches the lights were all flashing rapidly
> > and in
> > unison.  It wasn't until we cycled the power on the root switches (one
> > at a
> > time) that the network came back.
> >
> > Talking to our field service rep...he said the situation we had was a
> > "network storm".  The cause is unknown.
> >
> > My question is has anyone ever experienced a "network storm" on a mesh
> > network and if so what caused it?
> >
> > Our mesh network has been in operation since Oct 2008 and this is the
> > first
> > issue we have had with it.
> >
> > We are running a Mesh Network version 8.4.1.
> >
> > --
> > Mike Jaudon
> > Tronox, LLC
> > Hamilton, MS
> >
> >
> >
> >
> > _______________________________________________________________________
> > This mailing list is neither sponsored nor endorsed by Invensys Process
> > Systems (formerly The Foxboro Company). Use the info you obtain here at
> > your own risks. Read http://www.thecassandraproject.org/disclaimer.html
> >
> > foxboro mailing list:             //www.freelists.org/list/foxboro
> > to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> > to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
> >
> >
> >
> > _______________________________________________________________________
> > This mailing list is neither sponsored nor endorsed by Invensys Process
> > Systems (formerly The Foxboro Company). Use the info you obtain here at
> > your own risks. Read http://www.thecassandraproject.org/disclaimer.html
> >
> > foxboro mailing list:             //www.freelists.org/list/foxboro
> > to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> > to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
> >
> >
> >
> > _______________________________________________________________________
> > This mailing list is neither sponsored nor endorsed by Invensys Process
> > Systems (formerly The Foxboro Company). Use the info you obtain here at
> > your own risks. Read http://www.thecassandraproject.org/disclaimer.html
> >
> > foxboro mailing list:             //www.freelists.org/list/foxboro
> > to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> > to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
> >
> >
>
>
> --
> Mike Jaudon
>
>
>
> _______________________________________________________________________
> This mailing list is neither sponsored nor endorsed by Invensys Process
> Systems (formerly The Foxboro Company). Use the info you obtain here at
> your own risks. Read http://www.thecassandraproject.org/disclaimer.html
>
> foxboro mailing list:             //www.freelists.org/list/foxboro
> to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>
>
>
> _______________________________________________________________________
> This mailing list is neither sponsored nor endorsed by Invensys Process
> Systems (formerly The Foxboro Company). Use the info you obtain here at
> your own risks. Read http://www.thecassandraproject.org/disclaimer.html
>
> foxboro mailing list:             //www.freelists.org/list/foxboro
> to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>
>


--
Mike Jaudon



_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html

foxboro mailing list:             //www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave






* Confidentiality Notice:
This e-mail and any associated files are intended solely for the individual or 
entity to whom they are addressed. Please do not copy it or use it for any 
purposes, or disclose its contents to any other person. Further, this e-mail 
and any associated files may be confidential and further may be legally 
privileged. This email is from the Invensys Process Systems business unit of 
Invensys plc which is a company registered in England and Wales with its 
registered office at Portland House, Bressenden Place, London, SW1E 5BF 
(Registered number 166023).  For a list of European legal entities within the 
Invensys Process Systems business group, please click here 
http://www.invensys.com/legal/default.asp?top_nav_idw&nav_id?&prev_idw.

If you have received this e-mail in error, you are on notice of its status. 
Please notify us immediately by reply e-mail and then delete this message from 
your system. Thank you for your co-operation. You may contact our Helpdesk on 
+44 (0)20 7821 3859 / 2105 or email inet.hqhelpdesk@xxxxxxxxxxxxx This e-mail 
and any attachments thereto may be subject to the terms of any agreements 
between Invensys (and/or its subsidiaries and affiliates) and the recipient 
(and/or its subsidiaries and affiliates).


 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             //www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave

Other related posts:

  1. Home
  2. foxboro
  3. Archive
  4. 02-2009