[isapros] Re: [ISAServer] Firewall database corruption due to power outtage
- From: "Thomas W Shinder" <tshinder@xxxxxxxxxxx>
- To: <isapros@xxxxxxxxxxxxx>
- Date: Mon, 10 Jul 2006 12:32:07 -0500
Why get steamed? Seems purdy easy to me.
Thomas W Shinder, M.D.
Site: www.isaserver.org
Blog: http://blogs.isaserver.org/shinder/
Book: http://tinyurl.com/3xqb7
MVP -- ISA Firewalls
> -----Original Message-----
> From: isapros-bounce@xxxxxxxxxxxxx
> [mailto:isapros-bounce@xxxxxxxxxxxxx] On Behalf Of Jim Harrison
> Sent: Monday, July 10, 2006 12:28 PM
> To: isapros@xxxxxxxxxxxxx
> Subject: [isapros] Re: [ISAServer] Firewall database
> corruption due to power outtage
>
> Nope - just a workaround that makes Tim all steamy.
>
> -------------------------------------------------------
> Jim Harrison
> MCP(NT4, W2K), A+, Network+, PCG
> http://isaserver.org/Jim_Harrison/
> http://isatools.org
> Read the help / books / articles!
> -------------------------------------------------------
>
>
> -----Original Message-----
> From: isapros-bounce@xxxxxxxxxxxxx
> [mailto:isapros-bounce@xxxxxxxxxxxxx] On Behalf Of Thomas W Shinder
> Sent: Monday, July 10, 2006 09:20
> To: isapros@xxxxxxxxxxxxx
> Subject: [isapros] Re: [ISAServer] Firewall database
> corruption due to power outtage
>
> Then that would fix it? Right?
>
> Thomas W Shinder, M.D.
> Site: www.isaserver.org
> Blog: http://blogs.isaserver.org/shinder/
> Book: http://tinyurl.com/3xqb7
> MVP -- ISA Firewalls
>
>
>
> > -----Original Message-----
> > From: isapros-bounce@xxxxxxxxxxxxx
> > [mailto:isapros-bounce@xxxxxxxxxxxxx] On Behalf Of Thor
> > (Hammer of God)
> > Sent: Monday, July 10, 2006 11:15 AM
> > To: isapros@xxxxxxxxxxxxx
> > Subject: [isapros] Re: [ISAServer] Firewall database
> > corruption due to power outtage
> >
> > Right... That's what I said... Disable lockdown.
> >
> > t
> >
> >
> > On 7/10/06 8:29 AM, "Jim Harrison" <Jim@xxxxxxxxxxxx>
> spoketh to all:
> >
> > > Like unto thusly:
> > >
> > http://www.microsoft.com/technet/prodtechnol/isa/2004/plan/dis
> > ablelockdownonlo
> > > gfailure.mspx
> > >
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > Jim Harrison
> > > MCP(NT4, W2K), A+, Network+, PCG
> > > http://isaserver.org/Jim_Harrison/
> > > http://isatools.org
> > > Read the help / books / articles!
> > > -------------------------------------------------------
> > >
> > >
> > > -----Original Message-----
> > > From: isapros-bounce@xxxxxxxxxxxxx
> > [mailto:isapros-bounce@xxxxxxxxxxxxx] On
> > > Behalf Of Thomas W Shinder
> > > Sent: Monday, July 10, 2006 08:10
> > > To: isapros@xxxxxxxxxxxxx
> > > Subject: [isapros] Re: [ISAServer] Firewall database
> > corruption due to power
> > > outtage
> > >
> > > Log Failure Alert.
> > >
> > > Thomas W Shinder, M.D.
> > > Site: www.isaserver.org
> > > Blog: http://blogs.isaserver.org/shinder/
> > > Book: http://tinyurl.com/3xqb7
> > > MVP -- ISA Firewalls
> > >
> > >
> > >
> > >> -----Original Message-----
> > >> From: isapros-bounce@xxxxxxxxxxxxx
> > >> [mailto:isapros-bounce@xxxxxxxxxxxxx] On Behalf Of Thor
> (Hammer of
> > >> God)
> > >> Sent: Monday, July 10, 2006 9:24 AM
> > >> To: isapros@xxxxxxxxxxxxx
> > >> Subject: [isapros] Re: [ISAServer] Firewall database
> > corruption due to
> > >> power outtage
> > >>
> > >> What do you mean "the alert?" What alert?
> > >>
> > >> t
> > >>
> > >>
> > >> On 7/10/06 7:21 AM, "Thomas W Shinder"
> > <tshinder@xxxxxxxxxxx> spoketh
> > >> to
> > >> all:
> > >>
> > >>> Hi Tim,
> > >>>
> > >>> But you can do that by configuring the Alert to not send
> > >> the system into
> > >>> lockdown.
> > >>>
> > >>> Thomas W Shinder, M.D.
> > >>> Site: www.isaserver.org
> > >>> Blog: http://blogs.isaserver.org/shinder/
> > >>> Book: http://tinyurl.com/3xqb7
> > >>> MVP -- ISA Firewalls
> > >>>
> > >>>
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> > >>>> Sent: Monday, July 10, 2006 9:16 AM
> > >>>> To: Adar Greenshpon; ISA-MVP; Avi Sander; Nathan Bigman
> > >>>> Subject: Re: [ISAServer] Firewall database corruption
> > due to power
> > >>>> outtage
> > >>>>
> > >>>> I guess I misunderstood. The "mentioned below" items
> kept going
> > >>>> back to how to remove the fluff, rather than dealing
> with why ISA
> > >>>> goes into lockdown so readily. I've still not seen any
> > information
> > >>>> on how/when/why ISA goes into lockdown when logging to
> ISA. You
> > >>>> had indicated that the other information was
> incorrect, so I was
> > >>>> standing by for that info.
> > >>>>
> > >>>> It doesn't matter if you put it in a different database.
> > When the
> > >>>> optimization plan (you know, the "normal" plan that
> > >> rebuilds indexes,
> > >>>> insures integrity, etc) runs, ISA goes into lockdown. Even if I
> > >>>> waited a week in between times that I purged the data,
> I'll still
> > >>>> have to perform some maintenance, and the system would still go
> > >>>> into lockdown.
> > >>>>
> > >>>> The solution is to disable lockdown.
> > >>>>
> > >>>> Thanks!
> > >>>>
> > >>>> t
> > >>>>
> > >>>>
> > >>>> On 7/10/06 12:13 AM, "Adar Greenshpon"
> > >> <Adar.Greenshpon@xxxxxxxxxxxxx>
> > >>>> spoketh to all:
> > >>>>
> > >>>>> Thor: as I've indicated below, we plan on improving the
> > >> connectivity
> > >>>>> issue you raised. Given the current SQL logging
> > >> limitation and your
> > >>>>> deployment, might it be possible to split the webproxy
> > >> log into two
> > >>>>> databases: one for historical purposes and one for current
> > >>>> events. The
> > >>>>> heavy-weight maintenance could be done on the former one
> > >>>> while the later
> > >>>>> one would be available for ISA to dump logs into.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> -----Original Message-----
> > >>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> > >>>>> Sent: Monday, July 10, 2006 1:50 AM
> > >>>>> To: ISA-MVP; Avi Sander; Adar Greenshpon; Nathan Bigman
> > >>>>> Subject: Re: [ISAServer] Firewall database corruption
> > due to power
> > >>>>> outtage
> > >>>>>
> > >>>>>
> > >>>>> Yes, Jim... Thank you. Sorry that I got so short, but I
> > >> was getting
> > >>>>> quite
> > >>>>> frustrated trying to make the same point over and over again.
> > >>>>>
> > >>>>> The focus need not be on "how to trim the logs." That's
> > >>>> not the *real*
> > >>>>> issue. The focus needs to be: "Regarding lockdown, what
> > >>>> factors are in
> > >>>>> play
> > >>>>> when ISA logs to SQL, and under what circumstances does
> > ISA invoke
> > >>>>> lockdown when posting to a SQL DB?"
> > >>>>>
> > >>>>> Let's put the padding issues *totally* out of the picture.
> > >>>> Let's say
> > >>>>> that
> > >>>>> I've got TB's of storage and that I could not care less
> > >>>> about how much
> > >>>>> fluff
> > >>>>> is in my WebProxyLog-- let it get as big as it wants.
> > >>>> However, I must
> > >>>>> back
> > >>>>> it up. I've also experienced ISA going into lockdown
> > >>>> during classic db
> > >>>>> maintenance like backing up, re-indexing, optimizing size,
> > >>>> etc. These
> > >>>>> processes must be run regardless of the presence of
> > >>>> "padding issues" or
> > >>>>> not.
> > >>>>> In fact, even if the padding problem is solved in ISA2006, in
> > >>>>> combination with or exclusive of contributing options
> > in SQL 2005
> > >>>>> (which I'm migrating to as I type), ISA will still go into
> > >>>>> lockdown when the SQL server engages in processes
> which may lock
> > >>>>> the db, or delay response in some way.
> > >>>>>
> > >>>>> Again- (and hopefully someone will listen this time) the
> > >>>> concern speaks
> > >>>>> to
> > >>>>> the "real world" usability of "lockdown mode" within the
> > >>>> enterprise. If
> > >>>>> Microsoft intends to tout the lockdown feature of ISA
> as a tool
> > >>>>> enterprise clients can use to ensure evidentiary
> integrity, then
> > >>>>> you
> > >>>> must address
> > >>>>> the
> > >>>>> mode's "survivability" during standard, everyday,
> > >>>> real-world processes
> > >>>>> that
> > >>>>> must be run on the SQL servers the ISA box is logging to,
> > >> or it will
> > >>>>> simply
> > >>>>> not be used. This relates to processes that may or
> may not have
> > >>>>> anything to do with the log files themselves.
> > >>>>>
> > >>>>> If nobody cares if lockdown is utilized in the
> > >> enterprise, then this
> > >>>>> thread
> > >>>>> is done. If, however, the goal is supply enterprise
> > >> customers with a
> > >>>>> lockdown mode they can use, then user-defined connection
> > >>>> parameters must
> > >>>>> be
> > >>>>> provided in order to customize the lockdown threshold to our
> > >>>>> environments.
> > >>>>>
> > >>>>> T
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On 7/9/06 9:28 AM, "Jim Harrison (ISA)"
> > >> <Jim.Harrison@xxxxxxxxxxxxx>
> > >>>>> spoketh
> > >>>>> to all:
> > >>>>>
> > >>>>>> It may help in the short run, but the issue Tim raises
> > >>>> isn't so much
> > >>>>>> "how much data is being logged", but "how long ISA will
> > >>>> wait while the
> > >>>>>> logging process fails to respond before it raises an
> > >> alert" and "I
> > >>>>> wanna
> > >>>>>> set this trigger point".
> > >>>>>>
> > >>>>>> What he's asking for (as have more than a few customers)
> > >>>> is some way
> > >>>>> for
> > >>>>>> the user to specify a "almost choking on my own data"
> > >>>> alert limit that
> > >>>>>> could be used to move ISA logging to an alternate
> > >> destination (SQL,
> > >>>>>> file, bitbucket) until the primary logger responds once
> > >>>> more. What we
> > >>>>>> have now is a "choked on my own data" alert, which is a
> > >>>> bit too late.
> > >>>>>>
> > >>>>>> In fact, I've seen one unfavorable comparison to
> > >> <product deleted>
> > >>>>> where
> > >>>>>> they actually:
> > >>>>>> 1. fail over to a backup logging process 2. monitor
> the primary
> > >>>>>> logger 3. fail back when the primary logger
> reappears 4. import
> > >>>>>> the alternate-store log data into the primary
> > >> store as a
> > >>>>>> low-pri part of the fail-back.
> > >>>>>>
> > >>>>>> Techincally, this is scriptable, except for the fact
> > >> that our alert
> > >>>>>> happens too late (and isn't "tweakable"). If one were
> > willing to
> > >>>>>> tolerate the momentary traffic block that's created today,
> > >>>> a script is
> > >>>>> a
> > >>>>>> workable solution (except for the "pre-panic" factor).
> > >>>>>>
> > >>>>>> Jim Harrison
> > >>>>>> SASD (ISA SE)
> > >>>>>> If We Can't Fix It - It Ain't Broke!
> > >>>>>>
> > >>>>>> -----Original Message-----
> > >>>>>> From: Susan Bradley, CPA aka Ebitz - SBS Rocks [MVP]
> > >>>>>> [mailto:sbradcpa@xxxxxxxxxxx]
> > >>>>>> Sent: Sunday, July 09, 2006 2:22 AM
> > >>>>>> To: isaserver@xxxxxxxxxxxxxxx
> > >>>>>> Cc: Avi Sander; Adar Greenshpon; Nathan Bigman
> > >>>>>> Subject: Re: [ISAServer] Firewall database corruption
> > >> due to power
> > >>>>>> outtage
> > >>>>>>
> > >>>>>> Shouldn't depadding not cause the system to freak out
> > and go into
> > >>>>>> lockdown? That's his real point.. not necessarily the
> > >>>> fact he has to
> > >>>>> do
> > >>>>>>
> > >>>>>> it..but the fact that ISA is there locking itself down in
> > >>>> the process
> > >>>>>> ....
> > >>>>>>
> > >>>>>> Thor (Hammer of God) wrote:
> > >>>>>>
> > >>>>>>> Never mind. Not sure why I bother...
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> t
> > >>>>>>>
> > >>>>>>> On 7/9/06 12:01 AM, "Avi Sander" <asander@xxxxxxxxxxxxx>
> > >>>> spoketh to
> > >>>>>> all:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Hi Thor -
> > >>>>>>>>
> > >>>>>>>> Please see
> > >>>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/tsqlr
> > >>>>>> ef
> > >>>>>>>> /ts_set-set_2uw7.asp .
> > >>>>>>>>
> > >>>>>>>> It describes how one can toggle the ANSI_PADDING
> > >> option. This can
> > >>>>>>>> control how fields are padded\trimmed in SQL, but
> > should be set
> > >>>>> prior
> > >>>>>> to
> > >>>>>>>> tbl creation.
> > >>>>>>>>
> > >>>>>>>> - avi
> > >>>>>>>> -----Original Message-----
> > >>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> > >>>>>>>> Sent: Friday, July 07, 2006 6:23 PM
> > >>>>>>>> To: Avi Sander; Adar Greenshpon; ISA-MVP
> > >>>>>>>> Cc: Nathan Bigman
> > >>>>>>>> Subject: Re: [ISAServer] Firewall database corruption
> > >>>> due to power
> > >>>>>>>> outtage
> > >>>>>>>>
> > >>>>>>>> I think we're getting off track a little here...
> > >>>>>>>>
> > >>>>>>>> I appreciate the link to that "fix," but I've *already
> > >>>> solved* the
> > >>>>>>>> padding
> > >>>>>>>> issue. I've been including a script for that in
> my Blackhat
> > >>>>> trainings
> > >>>>>>>> for
> > >>>>>>>> years. To that effect, simply trimming the data in
> > the default
> > >>>>> table
> > >>>>>>>> isn't
> > >>>>>>>> the best way to go, IMO. The data structure is not very
> > >>>> efficient
> > >>>>> to
> > >>>>>>>> begin
> > >>>>>>>> with (for my needs). I've created my own (better designed)
> > >>>>> structure,
> > >>>>>>>> and
> > >>>>>>>> trim the data as I post into that table (and there is no
> > >>>> "trim data"
> > >>>>>> DSN
> > >>>>>>>> option that I know of). But that's OK-- as an
> > administrator, I
> > >>>>>> *expect*
> > >>>>>>>> that I will have to adjust logging processes to better
> > >>>> fit my needs
> > >>>>>>>> (just
> > >>>>>>>> like I do with IIS logs that I post to SQL). You guys
> > >> provide the
> > >>>>>>>> mechanism,
> > >>>>>>>> and I customize it to suit me. Like I said, my yearly
> > >>>> retained data
> > >>>>> to
> > >>>>>>>> date
> > >>>>>>>> is only 10 gig. No problems.
> > >>>>>>>>
> > >>>>>>>> What I'm saying is that *when I run the process to
> > >> trim the data*
> > >>>>> ISA
> > >>>>>>>> goes
> > >>>>>>>> into "lockdown." This whole thing got started when we
> > >>>> were talking
> > >>>>>>>> about
> > >>>>>>>> lockdown mode, and how "trigger happy" it is when one is
> > >>>> logging to
> > >>>>>> SQL.
> > >>>>>>>>
> > >>>>>>>> I only brought it up in an effort to identify this:
> > >>>>>>>> If a customer logs to SQL, there is a padding problem.
> > >>>> The padding
> > >>>>>>>> problem
> > >>>>>>>> *requires* DB maintenance processes to control.
> When the DB
> > >>>>>> maintenance
> > >>>>>>>> is
> > >>>>>>>> executed against any DB of consequence (just
> several thousand
> > >>>>> records
> > >>>>>>>> seems
> > >>>>>>>> to be enough), ISA goes into lockdown, presumably
> > >> because of the
> > >>>>> "lag"
> > >>>>>>>> created by server load when the SQL server goes about
> > >> parsing out
> > >>>>> gigs
> > >>>>>>>> of
> > >>>>>>>> fluff data. This is true even on my production SQL
> > >>>> cluster (which
> > >>>>> is
> > >>>>>>>> kickass, mind you ;) so it's not a "hardware" problem.
> > >>>> Thus, I was
> > >>>>>>>> forced
> > >>>>>>>> to disable lockdown.
> > >>>>>>>>
> > >>>>>>>> Therefore, IF a customer is logging to SQL, AND
> they want use
> > >>>>>> "lockdown"
> > >>>>>>>> THEN better user-definable timeout parameters need to be
> > >>>> available.
> > >>>>>>>> That's
> > >>>>>>>> all I was saying.
> > >>>>>>>>
> > >>>>>>>> Of course, since you have identified that ISA2004 EE,
> > >> ISA2006 SE,
> > >>>>> and
> > >>>>>>>> ISA2006 EE properly trim data before posting it to SQL,
> > >>>> then it all
> > >>>>>>>> seems to
> > >>>>>>>> be a moot point, right? ;)
> > >>>>>>>>
> > >>>>>>>> The main box posting web access is currently ISA 2004
> > >> SE. But I
> > >>>>> think
> > >>>>>>>> I've
> > >>>>>>>> got a copy of ISA 2004 EE that I picked up in Hong Kong
> > >>>> along with
> > >>>>> an
> > >>>>>>>> old
> > >>>>>>>> Rush CD that I'll throw on that guy and see what
> > >>>> happens. (jk;) I'll
> > >>>>>> let
> > >>>>>>>> you
> > >>>>>>>> guys know what happens.
> > >>>>>>>>
> > >>>>>>>> Thx
> > >>>>>>>> t
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 7/7/06 12:29 AM, "Avi Sander" <asander@xxxxxxxxxxxxx>
> > >>>> spoketh to
> > >>>>>> all:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> there's also an option on the ODBC DSN : 'trim
> fields' that
> > >>>>> partially
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> helps,
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> if i recall correctly.
> > >>>>>>>>> I'll look into this further
> > >>>>>>>>>
> > >>>>>>>>> ________________________________
> > >>>>>>>>>
> > >>>>>>>>> From: Adar Greenshpon
> > >>>>>>>>> Sent: Fri 07/07/2006 09:07
> > >>>>>>>>> To: Avi Sander; Thor (Hammer of God);
> > >> isaserver@xxxxxxxxxxxxxxx
> > >>>>>>>>> Cc: Nathan Bigman
> > >>>>>>>>> Subject: RE: [ISAServer] Firewall database corruption
> > >>>> due to power
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> outtage
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Thor,
> > >>>>>>>>>
> > >>>>>>>>> Per the excessive padding, see this thread:
> > >>>>>>>>> http://forums.isaserver.org/m_140009200/tm.htm - I
> > >> think that's
> > >>>>> what
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> you're
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> looking for as we already heard this a few times (as
> > >> Avi pointed
> > >>>>> out,
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> the
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> problem does not exist in ISA 2004EE or ISA 2006
> > >>>> versions). Nathan
> > >>>>> -
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> will an
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> official KB help here?
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Adar.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> ________________________________
> > >>>>>>>>>
> > >>>>>>>>> From: Avi Sander
> > >>>>>>>>> Sent: Friday, July 07, 2006 12:27 AM
> > >>>>>>>>> To: Thor (Hammer of God)
> > >>>>>>>>> Cc: Adar Greenshpon
> > >>>>>>>>> Subject: RE: [ISAServer] Firewall database corruption
> > >>>> due to power
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> outtage
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> OK - that explains it.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> This is a known SQL ODBC interfacing issue.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> ISA 2004 EE and ISA2006 SE & EE moved away from ODBC
> > >>>> and use oledb
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> interfaces
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> for SQL server logging instead. This issue does not
> > >> exist there
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> anymore as it
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> is all trimmed by design (and throughput is also greatly
> > >>>>> increased).
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> The 2 potential causes of errors i mentioned earlier in
> > >>>> the thread
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> (buffers
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> full, 4 retries) are only relevant to the OLEDB
> > >>>> logging, and not to
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> your ISA -
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> my bad.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> let me check with the ISA2004 SE code and get back to
> > >> you with a
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> better
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> picture...
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> -avi
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> ________________________________
> > >>>>>>>>>
> > >>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> > >>>>>>>>> Sent: Thu 06/07/2006 23:12
> > >>>>>>>>> To: Avi Sander
> > >>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption
> > >>>> due to power
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> outtage
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> These are ISA2004 SE boxes. I haven't tried with
> > >>>> ISA2006 yet, but
> > >>>>>> I'm
> > >>>>>>>>> running it here at The Yeti and can check it out
> (unless you
> > >>>>> already
> > >>>>>>>>> know...)
> > >>>>>>>>>
> > >>>>>>>>> t
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On 7/6/06 2:08 PM, "Avi Sander" <asander@xxxxxxxxxxxxx>
> > >>>> spoketh to
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> all:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> Hey Thor -
> > >>>>>>>>>>
> > >>>>>>>>>> What ISA version are you using? and whichSKU: SE or EE?
> > >>>>>>>>>>
> > >>>>>>>>>> -avi
> > >>>>>>>>>>
> > >>>>>>>>>> ________________________________
> > >>>>>>>>>>
> > >>>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> > >>>>>>>>>> Sent: Thu 06/07/2006 18:39
> > >>>>>>>>>> To: Adar Greenshpon; ISA-MVP; Avi Sander
> > >>>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption
> > >>>> due to power
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> outtage
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> Hi Adar-
> > >>>>>>>>>>
> > >>>>>>>>>> Yes, logs going to an external SQL 2000 cluster via
> > >>>> direct gigabit
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> switched
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> link. And log retention is permanent. Though I've
> > >>>> only got 120
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> employees,
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> we
> > >>>>>>>>>> have tight policies on Internet usage and abuse and I,
> > >>>> like many,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> must keep
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> access logs to ensure these policies are being
> > >>>> honored, as well as
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> for any
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> number of potential legal issues that might arise. I
> > >>>> trust that
> > >>>>> you
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> are not
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> suggesting that ISA log retention via SQL should be
> > >> measured in
> > >>>>>> days,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> right?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> ;)
> > >>>>>>>>>>
> > >>>>>>>>>> Let me go into more detail about the "fluff" I
> > >> mentioned- it is
> > >>>>> not
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> so much
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> the "gigs of logs" as it would relate to real data- it
> > >>>> is actual
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> "fluff:"
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> Extra, padded spacing in the fields posted to
> SQL from ISA.
> > >>>>> Though
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> you guys
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> have defined the fields in SQL as nvarchar data, there
> > >>>> is some bad
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> mojo in
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> the
> > >>>>>>>>>> logging process. Cursory review of the data in
> > >>>> something like SQL
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> Query
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> Analyzer may not immediately reveal this, but binary
> > >> analyze of
> > >>>>> the
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> data
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> files
> > >>>>>>>>>> do.
> > >>>>>>>>>>
> > >>>>>>>>>> I've created some example text files to simplify what
> > >>>> I am talking
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> about
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> (Attached as SQL1.txt and SQL2.txt).
> > >>>>>>>>>>
> > >>>>>>>>>> Consider the following simple query to retrieve a
> > >> single row of
> > >>>>>> basic
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> data
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> from the WebProxyLog as stored by ISA:
> > >>>>>>>>>>
> > >>>>>>>>>> "select
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>
> > >>
> > ClientUserName,ClientAgent,DestHost,DestHostIP,SrcNetwork,DstNetwork
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> from WebProxyLog where LogID=10133541"
> > >>>>>>>>>>
> > >>>>>>>>>> (SQL1.txt contains the results, but I'll paste them
> > >>>> here as well)
> > >>>>>>>>>> <begin paste>
> > >>>>>>>>>> ANCHORSIGN\yomomma
> > >>>>>>>>>> PGP
> > >>>>>>>>>> update.pgp.com
> > >>>>>>>>>> 63.251.255.18 VPN Clients
> > >>>>>>>>>> External
> > >>>>>>>>>>
> > >>>>>>>>>> </end paste>
> > >>>>>>>>>>
> > >>>>>>>>>> Notice all the extra spacing; again, even though the
> > >> fields are
> > >>>>> type
> > >>>>>>>>>> nvarchar-- ISA is padding the data. Now, let's trim
> > >>>> it up and see
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> what
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> happens in this query:
> > >>>>>>>>>>
> > >>>>>>>>>> "select
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> rtrim(ClientUserName),rtrim(ClientAgent),rtrim(DestHost),rtrim(DestHos
> > >>>>>> tI
> > >>>>>>>> P),rt>>
> > >>>>>>>> r
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> im(SrcNetwork),rtrim(DstNetwork) from WebProxyLog where
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> LogID=10133541"
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> (SQL2.txt contains the results, but again I'll
> paste them)
> > >>>>>>>>>> <begin paste>
> > >>>>>>>>>> ANCHORSIGN\yomomma PGP update.pgp.com
> > 63.251.255.18
> > >>>>> VPN
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> Clients
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> External
> > >>>>>>>>>> </end paste>
> > >>>>>>>>>>
> > >>>>>>>>>> The original query results are 934 bytes. The
> > trimmed query
> > >>>>> results
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> are 74
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> bytes. Just for this simple query, with selected
> > >>>> fields from one
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> record, the
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> storage requirements for this data is 12.62 times
> > >>>> greater due to
> > >>>>> the
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> "fluff."
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> Extend that out to a full logged record, and then
> > >> multiply that
> > >>>>>> times
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> tens of
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> thousands. *THIS* is the 1-2 gigs of "fluff" I'm
> > >>>> talking about on
> > >>>>> a
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> *daily*
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> basis.
> > >>>>>>>>>>
> > >>>>>>>>>> The "required maintenance" I'm talking about is where
> > >>>> I take daily
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> raw data
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> from the WebProxyLog, trim it up, and then post it
> > >> into my own
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> tables. In my
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> case, the _yearly_ web proxy log data for my users is
> > >>>> only 10 gig.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> _Weekly_
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> "raw" data in the original ISA log format averages 12
> > >>>> gig. When
> > >>>>> I
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> run this
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> process, I am able to retrieve data via SQA, Access
> > >>>> front end, or
> > >>>>>> via
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> ADODB
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> objects, yet ISA will go into lockdown mode while
> > >> logging when
> > >>>>> this
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> job runs
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> at night. That's why I have to turn it off.
> > >>>>>>>>>>
> > >>>>>>>>>> I hope that better defines the issue as I see it.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanksl
> > >>>>>>>>>> T
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> "Tom Shinder pities Mr. T"
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 7/5/06 2:01 AM, "Adar Greenshpon"
> > >>>>> <Adar.Greenshpon@xxxxxxxxxxxxx>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> spoketh
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> to all:
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Hi Thor,
> > >>>>>>>>>>
> > >>>>>>>>>> Just so we'll be on the same page: are you directing
> > >>>> your ISA logs
> > >>>>>> to
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> an
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> external machine with SQL Server (SQL 2000 or 2005)?.
> > >>>> If so, how
> > >>>>> do
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> you do
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> the maintenance? What's your desired log retention
> > >>>> policy (e.g. 7
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> days)?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> Per the gigs of logs you're doing, have you considered
> > >>>> reducing it
> > >>>>>> by
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> not
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> logging images http requests?
> > >>>>>>>>>> Adar.
> > >>>>>>>>>> __________________________________________
> > >>>>>>>>>> Adar Greenshpon | Program Manager | Microsoft ISA Server
> > >>>>>>>>>> adarg@xxxxxxxxxxxxx <mailto:adarg@xxxxxxxxxxxxx>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> <mailto:adarg@xxxxxxxxxxxxx>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> | Tel: +972.4.856.1077 | SMS/Cell: +972.54.666.4579
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> ________________________________
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> > >>>>>>>>>> Sent: Tuesday, July 04, 2006 8:55 PM
> > >>>>>>>>>> To: Avi Sander
> > >>>>>>>>>> Cc: Adar Greenshpon; ISA-MVP
> > >>>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption
> > >>>> due to power
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> outtage
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> OK- the either/or situation makes more sense. I'm
> > >>>> sure that I'm
> > >>>>>> not
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> hitting
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> the log buffer- the 9 second minimum window is what is
> > >>>> getting me.
> > >>>>>>>>>>
> > >>>>>>>>>> How exactly do you define a "failure to commit?" Are
> > >>>> you actively
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> looking
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> for
> > >>>>>>>>>> a "failure" code from SQL, or is it the lack of
> a committed
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> transaction
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> within
> > >>>>>>>>>> the time span? I ask because if I am in the middle of DB
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> maintenance, I can
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> execute a transaction that might have to wait a
> > >> while to get a
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> committal, but
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> SQL knows about the transaction pending and won't
> > error out.
> > >>>>>>>>>>
> > >>>>>>>>>> The SQL table structure included in 2004 results in
> > >> HUGE table
> > >>>>> sizes
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> for web
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> proxy logging alone. I'm talking gigs worth of fluff
> > >>>> per day, so
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> daily DB
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> maintenance is an absolute requirement. I have
> never had a
> > >>>>> logging
> > >>>>>>>>>> transaction "live" through a maintenance period
> > >>>> without going into
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> lockdown,
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> hence it being disabled on all my web proxy listener
> > >> machines.
> > >>>>>>>>>>
> > >>>>>>>>>> I understand that the reasoning behind "lockdown" is for
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> "evidentiary"
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> reasons, but that seems counter intuitive to me. The
> > >>>> actual log
> > >>>>> is
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> more
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> important to me than the absence of a logged event
> > >>>> when the system
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> locks
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> down.
> > >>>>>>>>>> So, not only are the services not available (though a
> > >>>>> "fail-safe"),
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> but I
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> loose any log of any attempt in the meantime. Are we
> > >>>> going to see
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> some more
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> "user definable" setting in regard to logging and
> > >> lockdown? If
> > >>>>> not,
> > >>>>>> I
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> don't
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> see how even mid-size businesses will be able to
> log to SQL
> > >>>>> without
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> disabling
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> the feature.
> > >>>>>>>>>>
> > >>>>>>>>>> t
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 7/4/06 5:51 AM, "Avi Sander"
> > >>>> <asander@xxxxxxxxxxxxx> spoketh to
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> all:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> ISA will lockdown if either of the following occurs
> > >> (whichever
> > >>>>> comes
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> 1st):
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> - Four consecutive failures to commit the
> > >> data into the
> > >>>>> DB.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> Our
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> default COMMIT timeout is 30 secs. We'll retry every 3
> > >>>> secs. So at
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> minimum
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> you've got 9 secs here.
> > >>>>>>>>>> - LOG buffers are exhausted\out of memory. By
> > >>>> default on a
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> 1GB RAM
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> machine we'll accumulate tens of thousands of recs. Is
> > >>>> there a lot
> > >>>>>> of
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> traffic
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> going through when this happens? How much RAM does
> > >> this machine
> > >>>>>> have?
> > >>>>>>>>>>
> > >>>>>>>>>> What is the event description you see in the EventViewer?
> > >>>>>>>>>>
> > >>>>>>>>>> The Log buffers can be extended some more, though not
> > >>>> recommended
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> since can
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> use up a lot of memory.
> > >>>>>>>>>>
> > >>>>>>>>>> -avi
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> ________________________________
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> From: Adar Greenshpon
> > >>>>>>>>>> Sent: Tuesday, July 04, 2006 2:03 PM
> > >>>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx
> > >>>>>>>>>> Cc: Avi Sander
> > >>>>>>>>>> Subject: RE: [ISAServer] Firewall database corruption
> > >>>> due to power
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> outtage
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> Avi - wanna play logging diva?
> > >>>>>>>>>> Generally speaking, there are four internal commit
> > >>>> retries until
> > >>>>> ISA
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> enters
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> lockdown. We have seen a few thousand log records in
> > >>>> the buffers
> > >>>>> in
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> these
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> temporary moments in our stress labs (nothing you
> > >> could really
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> witness on a
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> normal SBS box though).
> > >>>>>>>>>>
> > >>>>>>>>>> Adar.
> > >>>>>>>>>> __________________________________________
> > >>>>>>>>>> Adar Greenshpon | Program Manager | Microsoft ISA Server
> > >>>>>>>>>> adarg@xxxxxxxxxxxxx <mailto:adarg@xxxxxxxxxxxxx>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> <mailto:adarg@xxxxxxxxxxxxx>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> | Tel: +972.4.856.1077 | SMS/Cell: +972.54.666.4579
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> -----Original Message-----
> > >>>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> > >>>>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> > >>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> > >>>>>>>>>> Sent: Wednesday, June 28, 2006 8:26 PM
> > >>>>>>>>>> To: ISA-MVP
> > >>>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption
> > >>>> due to power
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> outtage
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> If that is what you guys are expecting to happen, I
> > >>>> can tell you,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> that
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> *ain't* the case - not when logging to SQL on another
> > >>>> box, anyway.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> There's
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> no "using as much memory as it can" going on. It
> > >>>> seems to be more
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> like "I
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> did not get an acknowledgement that the last
> > transaction was
> > >>>>>> written,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> and
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> it's been 2 seconds, so I'm going into lockdown."
> > >>>>>>>>>>
> > >>>>>>>>>> I'm looking forward to what info you can get from
> > >> your "logging
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> diva."
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> t
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 6/28/06 8:43 AM, "Jim Harrison (ISA)"
> > >>>>>> <Jim.Harrison@xxxxxxxxxxxxx>
> > >>>>>>>>>> spoketh to all:
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> Actually, it's not a fixed value (or
> > >>>> user-changeable), other than
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> being
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> self-regulating.
> > >>>>>>>>>>> ISA logging will use as much memory as it can until
> > >>>> it determines
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> that
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> growth is unchecked. Thus, how quickly ISA will cry
> > >>>> foul depends
> > >>>>>> in
> > >>>>>>>>>>> part on how much free memory is available.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I'll ask our logging diva for details, but this sums
> > >>>> it up pretty
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> well.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> Jim Harrison
> > >>>>>>>>>>> SASD (ISA SE)
> > >>>>>>>>>>> If We Can't Fix It - It Ain't Broke!
> > >>>>>>>>>>>
> > >>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> > >>>>>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> > >>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> > >>>>>>>>>>> Sent: Wednesday, June 28, 2006 8:09 AM
> > >>>>>>>>>>> To: ISA-MVP
> > >>>>>>>>>>> Subject: Re: [ISAServer] Firewall database
> > corruption due to
> > >>>>> power
> > >>>>>>>>>>> outtage
> > >>>>>>>>>>>
> > >>>>>>>>>>> It must be about a 1k buffer, then. I repeatedly
> > put ISA in
> > >>>>>> lockdown
> > >>>>>>>>>>> within
> > >>>>>>>>>>> seconds of maintenance jobs being run, or other "data
> > >>>> intensive"
> > >>>>>>>>>>> operations
> > >>>>>>>>>>> on the DB. I would start the job, in in a few
> > seconds, get
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> disconnected
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> form my TS session because of it-- It wasn't a "one
> > >>>> time" thing-
> > >>>>> it
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> was
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> consistent and repeatable. That's the only reason I
> > >> went ahead
> > >>>>> and
> > >>>>>>>>>>> disabled
> > >>>>>>>>>>> lockdown.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Is the buffer user definable somewhere? What exactly
> > >>>> does your
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> script
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> do?
> > >>>>>>>>>>>
> > >>>>>>>>>>> t
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 6/28/06 7:54 AM, "Jim Harrison (ISA)"
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> <Jim.Harrison@xxxxxxxxxxxxx>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> spoketh to all:
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Actually, that's not true; ISA doesn't go into
> > >> lockdown until
> > >>>>> the
> > >>>>>>>>>>>> logging buffer is filled.
> > >>>>>>>>>>>> I agree that a "backup logging" concept is useful.
> > >>>>>>>>>>>> I have a script that works for ISA 2000; it
> shouldn't be
> > >>>>> difficult
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> to
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> "agnosticize" it.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jim Harrison
> > >>>>>>>>>>>> Security Platform Group (ISA SE)
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> If We Can't Fix It - It Ain't Broke!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>>> From: Thor (Hammer of God)
> [mailto:thor@xxxxxxxxxxxxxxx]
> > >>>>>>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> > >>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> > >>>>>>>>>>>> Sent: Wednesday, June 28, 2006 7:34 AM
> > >>>>>>>>>>>> To: ISA-MVP
> > >>>>>>>>>>>> Subject: Re: [ISAServer] Firewall database
> > >> corruption due to
> > >>>>> power
> > >>>>>>>>>>>> outtage
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'd like to see some way to at least "buffer"
> the logging
> > >>>>> requests
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>> when
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> logging to an off-box ODBC destination, like SQL.
> > >>>> ISA is quite
> > >>>>>>>>>>>> intolerant
> > >>>>>>>>>>>> of even the slightest interruption in logging.
> > >> But, I guess
> > >>>>> that
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>> makes
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> sense if the goal is to have "evidentiary credibility."
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> It wouldn't have to be something l337 like a
> > >> secondary local
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> logging
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> option
> > >>>>>>>>>>>> or anything like that-- I think being able to
> > set a timeout
> > >>>>> value
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>> would
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> would. With larger db's, a maintenance job
> > would kick the
> > >>>>> server
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> into
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> lockdown, even though the SQL service is still
> running...
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jim, I know you wanted me to get my infrastructure
> > >>>> layout to you
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> again
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>> my solution to the SQL db size problems, but I
> > >> haven't had a
> > >>>>>> chance
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> to
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> find
> > >>>>>>>>>>>> that old email. I should just re-create it ;)
> > >>>> Regardless, even
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> with
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>> "solution" in place, I've had to disable
> > lockdown mode when
> > >>>>>> logging
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> to
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> SQL
> > >>>>>>>>>>>> for that very reason.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> t
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 6/28/06 7:19 AM, "Thomas W Shinder"
> > >> <tshinder@xxxxxxxxxxx>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> spoketh
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> to
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> all:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> And it's a really good security and forensics decision.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thomas W Shinder, M.D.
> > >>>>>>>>>>>> Site: www.isaserver.org
> > >>>>>>>>>>>> Blog: http://blogs.isaserver.org/shinder/
> > >>>>>>>>>>>> Book: http://tinyurl.com/3xqb7 MVP -- ISA Firewalls
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>>> From: Amy Babinchak
> > >> [mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx]
> > >>>>>>>>>>>> <mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx%5d>
> > >>>>>>>>>>>> <mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx%5d>
> > >>>>>>>>>>>> Sent: Wednesday, June 28, 2006 6:28 AM
> > >>>>>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> Subject: RE: [ISAServer] Firewall database
> > >>>> corruption due to
> > >>>>>>>>>>>> power outtage
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I like that ISA shuts down if it can't log. It's
> > >>>> an indicator
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> that
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> something has gone wrong.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Amy
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> -----Original Message-----
> > >>>>>>>>>>>> From: Susan Bradley, CPA aka Ebitz - SBS Rocks [MVP]
> > >>>>>>>>>>>> [mailto:sbradcpa@xxxxxxxxxxx]
> > >>>> <mailto:sbradcpa@xxxxxxxxxxx%5d>
> > >>>>>>>>>>>> <mailto:sbradcpa@xxxxxxxxxxx%5d>
> > >>>>>>>>>>>> Sent: Wednesday, June 28, 2006 12:12 AM
> > >>>>>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> Subject: [ISAServer] Firewall database corruption
> > >>>> due to power
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> outtage
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> With the caveat that we all know we need a good
> > >>>> UPS but like
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>> stuff
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>>> happens.. I've seen two folks recently have issues
> > >>>> getting a
> > >>>>>>>>>>>> server back
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> up have a 'dirty shutdown' due to power outtage
> > >>>> that ended up
> > >>>>>>>>>>>> corrupting
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> the ISA firewall database.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Short of uninstalling and reinstalling... short of
> > >>>> adjusting
> > >>>>>>>>>>>> ISA so it
> > >>>>>>>>>>>> won't shut down if it can't log... short of a
> > >> better UPS...
> > >>>>> any
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>> other
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> recommendations on handling this better?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>> To subscribe to the list - send an email to
> > >>>>> list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in JOIN
> > >> isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in LEAVE
> > >> isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Don't forget the comma!
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>> To subscribe to the list - send an email to
> > >>>>> list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in JOIN
> > >> isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in LEAVE
> > >> isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Don't forget the comma!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>> To subscribe to the list - send an email to
> > >>>>> list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in JOIN
> > isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in LEAVE
> > >> isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Don't forget the comma!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>> To subscribe to the list - send an email to
> > >>>> list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in JOIN
> > isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in LEAVE
> > isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Don't forget the comma!
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>> To subscribe to the list - send an email to
> > >>>> list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in JOIN
> > isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>>> In the subject line put in LEAVE
> > isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Don't forget the comma!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>> ---
> > >>>>>>>>>>> To subscribe to the list - send an email to
> > >>>> list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>> In the subject line put in JOIN
> > isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>
> > >>>>>>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx In
> > >>>>>>>>>>> the subject line put in LEAVE
> isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>> youremailaddress
> > >>>>>>>>>>>
> > >>>>>>>>>>> Don't forget the comma!
> > >>>>>>>>>>> ---
> > >>>>>>>>>>> To subscribe to the list - send an email to
> > >>>> list@xxxxxxxxxxxxxxx
> > >>>>>>>>>>> In the subject line put in JOIN
> isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> youremailaddress
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx In
> > >>>>>>>>>>> the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>> youremailaddress
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>>> Don't forget the comma!
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>> ---
> > >>>>>>>>>> To subscribe to the list - send an email to
> > >>>> list@xxxxxxxxxxxxxxx
> > >>>>>>>>>> In the subject line put in JOIN
> isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> youremailaddress
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx In
> > >>>>>>>>>> the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> youremailaddress
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> Don't forget the comma!
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> ---
> > >>>>>>>>>> To subscribe to the list - send an email to
> > >>>> list@xxxxxxxxxxxxxxx
> > >>>>>>>>>> In the subject line put in JOIN
> isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> youremailaddress
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx In
> > >>>>>>>>>> the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>> youremailaddress
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>>> Don't forget the comma!
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> ---
> > >>>>>>> To subscribe to the list - send an email to
> > list@xxxxxxxxxxxxxxx
> > >>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> > >>>>>> youremailaddress
> > >>>>>>>
> > >>>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx In the
> > >>>>>>> subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> > >>>>>> youremailaddress
> > >>>>>>>
> > >>>>>>> Don't forget the comma!
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>> ---
> > >>>>>> To subscribe to the list - send an email to
> > list@xxxxxxxxxxxxxxx
> > >>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> > >>>>>> youremailaddress
> > >>>>>>
> > >>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx In the
> > >>>>>> subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> > >>>>>> youremailaddress
> > >>>>>>
> > >>>>>> Don't forget the comma!
> > >>>>>> ---
> > >>>>>> To subscribe to the list - send an email to
> > list@xxxxxxxxxxxxxxx
> > >>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> > >>>>> youremailaddress
> > >>>>>>
> > >>>>>> To leave the list - send an email to
> > list@xxxxxxxxxxxxxxx In the
> > >>>>>> subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> > >>>>> youremailaddress
> > >>>>>>
> > >>>>>> Don't forget the comma!
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> ---
> > >>>> To subscribe to the list - send an email to
> > list@xxxxxxxxxxxxxxx In
> > >>>> the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> > >>>> youremailaddress
> > >>>>
> > >>>> To leave the list - send an email to
> list@xxxxxxxxxxxxxxx In the
> > >>>> subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> > >>>> youremailaddress
> > >>>>
> > >>>> Don't forget the comma!
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> > > All mail to and from this domain is GFI-scanned.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
> All mail to and from this domain is GFI-scanned.
>
>
>
Other related posts:
