[isapros] Re: [ISAServer] Firewall database corruption due to power outtage
- From: "Thomas W Shinder" <tshinder@xxxxxxxxxxx>
- To: <isapros@xxxxxxxxxxxxx>
- Date: Mon, 10 Jul 2006 09:21:03 -0500
Hi Tim,
But you can do that by configuring the Alert to not send the system into
lockdown.
Thomas W Shinder, M.D.
Site: www.isaserver.org
Blog: http://blogs.isaserver.org/shinder/
Book: http://tinyurl.com/3xqb7
MVP -- ISA Firewalls
> -----Original Message-----
> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> Sent: Monday, July 10, 2006 9:16 AM
> To: Adar Greenshpon; ISA-MVP; Avi Sander; Nathan Bigman
> Subject: Re: [ISAServer] Firewall database corruption due to
> power outtage
>
> I guess I misunderstood. The "mentioned below" items kept
> going back to how
> to remove the fluff, rather than dealing with why ISA goes
> into lockdown so
> readily. I've still not seen any information on how/when/why
> ISA goes into
> lockdown when logging to ISA. You had indicated that the
> other information
> was incorrect, so I was standing by for that info.
>
> It doesn't matter if you put it in a different database. When the
> optimization plan (you know, the "normal" plan that rebuilds indexes,
> insures integrity, etc) runs, ISA goes into lockdown. Even if
> I waited a
> week in between times that I purged the data, I'll still have
> to perform
> some maintenance, and the system would still go into lockdown.
>
> The solution is to disable lockdown.
>
> Thanks!
>
> t
>
>
> On 7/10/06 12:13 AM, "Adar Greenshpon" <Adar.Greenshpon@xxxxxxxxxxxxx>
> spoketh to all:
>
> > Thor: as I've indicated below, we plan on improving the connectivity
> > issue you raised. Given the current SQL logging limitation and your
> > deployment, might it be possible to split the webproxy log into two
> > databases: one for historical purposes and one for current
> events. The
> > heavy-weight maintenance could be done on the former one
> while the later
> > one would be available for ISA to dump logs into.
> >
> >
> >
> > -----Original Message-----
> > From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> > Sent: Monday, July 10, 2006 1:50 AM
> > To: ISA-MVP; Avi Sander; Adar Greenshpon; Nathan Bigman
> > Subject: Re: [ISAServer] Firewall database corruption due to power
> > outtage
> >
> >
> > Yes, Jim... Thank you. Sorry that I got so short, but I was getting
> > quite
> > frustrated trying to make the same point over and over again.
> >
> > The focus need not be on "how to trim the logs." That's
> not the *real*
> > issue. The focus needs to be: "Regarding lockdown, what
> factors are in
> > play
> > when ISA logs to SQL, and under what circumstances does ISA invoke
> > lockdown
> > when posting to a SQL DB?"
> >
> > Let's put the padding issues *totally* out of the picture.
> Let's say
> > that
> > I've got TB's of storage and that I could not care less
> about how much
> > fluff
> > is in my WebProxyLog-- let it get as big as it wants.
> However, I must
> > back
> > it up. I've also experienced ISA going into lockdown
> during classic db
> > maintenance like backing up, re-indexing, optimizing size,
> etc. These
> > processes must be run regardless of the presence of
> "padding issues" or
> > not.
> > In fact, even if the padding problem is solved in ISA2006, in
> > combination
> > with or exclusive of contributing options in SQL 2005 (which I'm
> > migrating
> > to as I type), ISA will still go into lockdown when the SQL server
> > engages
> > in processes which may lock the db, or delay response in some way.
> >
> > Again- (and hopefully someone will listen this time) the
> concern speaks
> > to
> > the "real world" usability of "lockdown mode" within the
> enterprise. If
> > Microsoft intends to tout the lockdown feature of ISA as a tool
> > enterprise
> > clients can use to ensure evidentiary integrity, then you
> must address
> > the
> > mode's "survivability" during standard, everyday,
> real-world processes
> > that
> > must be run on the SQL servers the ISA box is logging to, or it will
> > simply
> > not be used. This relates to processes that may or may not have
> > anything to
> > do with the log files themselves.
> >
> > If nobody cares if lockdown is utilized in the enterprise, then this
> > thread
> > is done. If, however, the goal is supply enterprise customers with a
> > lockdown mode they can use, then user-defined connection
> parameters must
> > be
> > provided in order to customize the lockdown threshold to our
> > environments.
> >
> > T
> >
> >
> >
> >
> >
> >
> >
> > On 7/9/06 9:28 AM, "Jim Harrison (ISA)" <Jim.Harrison@xxxxxxxxxxxxx>
> > spoketh
> > to all:
> >
> >> It may help in the short run, but the issue Tim raises
> isn't so much
> >> "how much data is being logged", but "how long ISA will
> wait while the
> >> logging process fails to respond before it raises an alert" and "I
> > wanna
> >> set this trigger point".
> >>
> >> What he's asking for (as have more than a few customers)
> is some way
> > for
> >> the user to specify a "almost choking on my own data"
> alert limit that
> >> could be used to move ISA logging to an alternate destination (SQL,
> >> file, bitbucket) until the primary logger responds once
> more. What we
> >> have now is a "choked on my own data" alert, which is a
> bit too late.
> >>
> >> In fact, I've seen one unfavorable comparison to <product deleted>
> > where
> >> they actually:
> >> 1. fail over to a backup logging process
> >> 2. monitor the primary logger
> >> 3. fail back when the primary logger reappears
> >> 4. import the alternate-store log data into the primary store as a
> >> low-pri part of the fail-back.
> >>
> >> Techincally, this is scriptable, except for the fact that our alert
> >> happens too late (and isn't "tweakable"). If one were willing to
> >> tolerate the momentary traffic block that's created today,
> a script is
> > a
> >> workable solution (except for the "pre-panic" factor).
> >>
> >> Jim Harrison
> >> SASD (ISA SE)
> >> If We Can't Fix It - It Ain't Broke!
> >>
> >> -----Original Message-----
> >> From: Susan Bradley, CPA aka Ebitz - SBS Rocks [MVP]
> >> [mailto:sbradcpa@xxxxxxxxxxx]
> >> Sent: Sunday, July 09, 2006 2:22 AM
> >> To: isaserver@xxxxxxxxxxxxxxx
> >> Cc: Avi Sander; Adar Greenshpon; Nathan Bigman
> >> Subject: Re: [ISAServer] Firewall database corruption due to power
> >> outtage
> >>
> >> Shouldn't depadding not cause the system to freak out and go into
> >> lockdown? That's his real point.. not necessarily the
> fact he has to
> > do
> >>
> >> it..but the fact that ISA is there locking itself down in
> the process
> >> ....
> >>
> >> Thor (Hammer of God) wrote:
> >>
> >>> Never mind. Not sure why I bother...
> >>>
> >>>
> >>> t
> >>>
> >>> On 7/9/06 12:01 AM, "Avi Sander" <asander@xxxxxxxxxxxxx>
> spoketh to
> >> all:
> >>>
> >>>
> >>>
> >>>> Hi Thor -
> >>>>
> >>>> Please see
> >>>>
> >
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/tsqlr
> >> ef
> >>>> /ts_set-set_2uw7.asp .
> >>>>
> >>>> It describes how one can toggle the ANSI_PADDING option. This can
> >>>> control how fields are padded\trimmed in SQL, but should be set
> > prior
> >> to
> >>>> tbl creation.
> >>>>
> >>>> - avi
> >>>> -----Original Message-----
> >>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> >>>> Sent: Friday, July 07, 2006 6:23 PM
> >>>> To: Avi Sander; Adar Greenshpon; ISA-MVP
> >>>> Cc: Nathan Bigman
> >>>> Subject: Re: [ISAServer] Firewall database corruption
> due to power
> >>>> outtage
> >>>>
> >>>> I think we're getting off track a little here...
> >>>>
> >>>> I appreciate the link to that "fix," but I've *already
> solved* the
> >>>> padding
> >>>> issue. I've been including a script for that in my Blackhat
> > trainings
> >>>> for
> >>>> years. To that effect, simply trimming the data in the default
> > table
> >>>> isn't
> >>>> the best way to go, IMO. The data structure is not very
> efficient
> > to
> >>>> begin
> >>>> with (for my needs). I've created my own (better designed)
> > structure,
> >>>> and
> >>>> trim the data as I post into that table (and there is no
> "trim data"
> >> DSN
> >>>> option that I know of). But that's OK-- as an administrator, I
> >> *expect*
> >>>> that I will have to adjust logging processes to better
> fit my needs
> >>>> (just
> >>>> like I do with IIS logs that I post to SQL). You guys provide the
> >>>> mechanism,
> >>>> and I customize it to suit me. Like I said, my yearly
> retained data
> > to
> >>>> date
> >>>> is only 10 gig. No problems.
> >>>>
> >>>> What I'm saying is that *when I run the process to trim the data*
> > ISA
> >>>> goes
> >>>> into "lockdown." This whole thing got started when we
> were talking
> >>>> about
> >>>> lockdown mode, and how "trigger happy" it is when one is
> logging to
> >> SQL.
> >>>>
> >>>> I only brought it up in an effort to identify this:
> >>>> If a customer logs to SQL, there is a padding problem.
> The padding
> >>>> problem
> >>>> *requires* DB maintenance processes to control. When the DB
> >> maintenance
> >>>> is
> >>>> executed against any DB of consequence (just several thousand
> > records
> >>>> seems
> >>>> to be enough), ISA goes into lockdown, presumably because of the
> > "lag"
> >>>> created by server load when the SQL server goes about parsing out
> > gigs
> >>>> of
> >>>> fluff data. This is true even on my production SQL
> cluster (which
> > is
> >>>> kickass, mind you ;) so it's not a "hardware" problem.
> Thus, I was
> >>>> forced
> >>>> to disable lockdown.
> >>>>
> >>>> Therefore, IF a customer is logging to SQL, AND they want use
> >> "lockdown"
> >>>> THEN better user-definable timeout parameters need to be
> available.
> >>>> That's
> >>>> all I was saying.
> >>>>
> >>>> Of course, since you have identified that ISA2004 EE, ISA2006 SE,
> > and
> >>>> ISA2006 EE properly trim data before posting it to SQL,
> then it all
> >>>> seems to
> >>>> be a moot point, right? ;)
> >>>>
> >>>> The main box posting web access is currently ISA 2004 SE. But I
> > think
> >>>> I've
> >>>> got a copy of ISA 2004 EE that I picked up in Hong Kong
> along with
> > an
> >>>> old
> >>>> Rush CD that I'll throw on that guy and see what
> happens. (jk;) I'll
> >> let
> >>>> you
> >>>> guys know what happens.
> >>>>
> >>>> Thx
> >>>> t
> >>>>
> >>>>
> >>>>
> >>>> On 7/7/06 12:29 AM, "Avi Sander" <asander@xxxxxxxxxxxxx>
> spoketh to
> >> all:
> >>>>
> >>>>
> >>>>
> >>>>> there's also an option on the ODBC DSN : 'trim fields' that
> > partially
> >>>>>
> >>>>>
> >>>> helps,
> >>>>
> >>>>
> >>>>> if i recall correctly.
> >>>>> I'll look into this further
> >>>>>
> >>>>> ________________________________
> >>>>>
> >>>>> From: Adar Greenshpon
> >>>>> Sent: Fri 07/07/2006 09:07
> >>>>> To: Avi Sander; Thor (Hammer of God); isaserver@xxxxxxxxxxxxxxx
> >>>>> Cc: Nathan Bigman
> >>>>> Subject: RE: [ISAServer] Firewall database corruption
> due to power
> >>>>>
> >>>>>
> >>>> outtage
> >>>>
> >>>>
> >>>>>
> >>>>> Thor,
> >>>>>
> >>>>> Per the excessive padding, see this thread:
> >>>>> http://forums.isaserver.org/m_140009200/tm.htm - I think that's
> > what
> >>>>>
> >>>>>
> >>>> you're
> >>>>
> >>>>
> >>>>> looking for as we already heard this a few times (as Avi pointed
> > out,
> >>>>>
> >>>>>
> >>>> the
> >>>>
> >>>>
> >>>>> problem does not exist in ISA 2004EE or ISA 2006
> versions). Nathan
> > -
> >>>>>
> >>>>>
> >>>> will an
> >>>>
> >>>>
> >>>>> official KB help here?
> >>>>>
> >>>>>
> >>>>>
> >>>>> Adar.
> >>>>>
> >>>>>
> >>>>>
> >>>>> ________________________________
> >>>>>
> >>>>> From: Avi Sander
> >>>>> Sent: Friday, July 07, 2006 12:27 AM
> >>>>> To: Thor (Hammer of God)
> >>>>> Cc: Adar Greenshpon
> >>>>> Subject: RE: [ISAServer] Firewall database corruption
> due to power
> >>>>>
> >>>>>
> >>>> outtage
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>> OK - that explains it.
> >>>>>
> >>>>>
> >>>>>
> >>>>> This is a known SQL ODBC interfacing issue.
> >>>>>
> >>>>>
> >>>>>
> >>>>> ISA 2004 EE and ISA2006 SE & EE moved away from ODBC
> and use oledb
> >>>>>
> >>>>>
> >>>> interfaces
> >>>>
> >>>>
> >>>>> for SQL server logging instead. This issue does not exist there
> >>>>>
> >>>>>
> >>>> anymore as it
> >>>>
> >>>>
> >>>>> is all trimmed by design (and throughput is also greatly
> > increased).
> >>>>>
> >>>>>
> >>>>>
> >>>>> The 2 potential causes of errors i mentioned earlier in
> the thread
> >>>>>
> >>>>>
> >>>> (buffers
> >>>>
> >>>>
> >>>>> full, 4 retries) are only relevant to the OLEDB
> logging, and not to
> >>>>>
> >>>>>
> >>>> your ISA -
> >>>>
> >>>>
> >>>>> my bad.
> >>>>>
> >>>>>
> >>>>>
> >>>>> let me check with the ISA2004 SE code and get back to you with a
> >>>>>
> >>>>>
> >>>> better
> >>>>
> >>>>
> >>>>> picture...
> >>>>>
> >>>>>
> >>>>>
> >>>>> -avi
> >>>>>
> >>>>>
> >>>>>
> >>>>> ________________________________
> >>>>>
> >>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> >>>>> Sent: Thu 06/07/2006 23:12
> >>>>> To: Avi Sander
> >>>>> Subject: Re: [ISAServer] Firewall database corruption
> due to power
> >>>>>
> >>>>>
> >>>> outtage
> >>>>
> >>>>
> >>>>> These are ISA2004 SE boxes. I haven't tried with
> ISA2006 yet, but
> >> I'm
> >>>>> running it here at The Yeti and can check it out (unless you
> > already
> >>>>> know...)
> >>>>>
> >>>>> t
> >>>>>
> >>>>>
> >>>>> On 7/6/06 2:08 PM, "Avi Sander" <asander@xxxxxxxxxxxxx>
> spoketh to
> >>>>>
> >>>>>
> >>>> all:
> >>>>
> >>>>
> >>>>>> Hey Thor -
> >>>>>>
> >>>>>> What ISA version are you using? and whichSKU: SE or EE?
> >>>>>>
> >>>>>> -avi
> >>>>>>
> >>>>>> ________________________________
> >>>>>>
> >>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> >>>>>> Sent: Thu 06/07/2006 18:39
> >>>>>> To: Adar Greenshpon; ISA-MVP; Avi Sander
> >>>>>> Subject: Re: [ISAServer] Firewall database corruption
> due to power
> >>>>>>
> >>>>>>
> >>>> outtage
> >>>>
> >>>>
> >>>>>> Hi Adar-
> >>>>>>
> >>>>>> Yes, logs going to an external SQL 2000 cluster via
> direct gigabit
> >>>>>>
> >>>>>>
> >>>> switched
> >>>>
> >>>>
> >>>>>> link. And log retention is permanent. Though I've
> only got 120
> >>>>>>
> >>>>>>
> >>>> employees,
> >>>>
> >>>>
> >>>>>> we
> >>>>>> have tight policies on Internet usage and abuse and I,
> like many,
> >>>>>>
> >>>>>>
> >>>> must keep
> >>>>
> >>>>
> >>>>>> access logs to ensure these policies are being
> honored, as well as
> >>>>>>
> >>>>>>
> >>>> for any
> >>>>
> >>>>
> >>>>>> number of potential legal issues that might arise. I
> trust that
> > you
> >>>>>>
> >>>>>>
> >>>> are not
> >>>>
> >>>>
> >>>>>> suggesting that ISA log retention via SQL should be measured in
> >> days,
> >>>>>>
> >>>>>>
> >>>> right?
> >>>>
> >>>>
> >>>>>> ;)
> >>>>>>
> >>>>>> Let me go into more detail about the "fluff" I mentioned- it is
> > not
> >>>>>>
> >>>>>>
> >>>> so much
> >>>>
> >>>>
> >>>>>> the "gigs of logs" as it would relate to real data- it
> is actual
> >>>>>>
> >>>>>>
> >>>> "fluff:"
> >>>>
> >>>>
> >>>>>> Extra, padded spacing in the fields posted to SQL from ISA.
> > Though
> >>>>>>
> >>>>>>
> >>>> you guys
> >>>>
> >>>>
> >>>>>> have defined the fields in SQL as nvarchar data, there
> is some bad
> >>>>>>
> >>>>>>
> >>>> mojo in
> >>>>
> >>>>
> >>>>>> the
> >>>>>> logging process. Cursory review of the data in
> something like SQL
> >>>>>>
> >>>>>>
> >>>> Query
> >>>>
> >>>>
> >>>>>> Analyzer may not immediately reveal this, but binary analyze of
> > the
> >>>>>>
> >>>>>>
> >>>> data
> >>>>
> >>>>
> >>>>>> files
> >>>>>> do.
> >>>>>>
> >>>>>> I've created some example text files to simplify what
> I am talking
> >>>>>>
> >>>>>>
> >>>> about
> >>>>
> >>>>
> >>>>>> (Attached as SQL1.txt and SQL2.txt).
> >>>>>>
> >>>>>> Consider the following simple query to retrieve a single row of
> >> basic
> >>>>>>
> >>>>>>
> >>>> data
> >>>>
> >>>>
> >>>>>> from the WebProxyLog as stored by ISA:
> >>>>>>
> >>>>>> "select
> >>>>>>
> >>>>>>
> >>>>
> ClientUserName,ClientAgent,DestHost,DestHostIP,SrcNetwork,DstNetwork
> >>>>
> >>>>
> >>>>>> from WebProxyLog where LogID=10133541"
> >>>>>>
> >>>>>> (SQL1.txt contains the results, but I'll paste them
> here as well)
> >>>>>> <begin paste>
> >>>>>> ANCHORSIGN\yomomma
> >>>>>> PGP
> >>>>>> update.pgp.com
> >>>>>> 63.251.255.18 VPN Clients
> >>>>>> External
> >>>>>>
> >>>>>> </end paste>
> >>>>>>
> >>>>>> Notice all the extra spacing; again, even though the fields are
> > type
> >>>>>> nvarchar-- ISA is padding the data. Now, let's trim
> it up and see
> >>>>>>
> >>>>>>
> >>>> what
> >>>>
> >>>>
> >>>>>> happens in this query:
> >>>>>>
> >>>>>> "select
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >
> rtrim(ClientUserName),rtrim(ClientAgent),rtrim(DestHost),rtrim(DestHos
> >> tI
> >>>> P),rt>>
> >>>> r
> >>>>
> >>>>
> >>>>>> im(SrcNetwork),rtrim(DstNetwork) from WebProxyLog where
> >>>>>>
> >>>>>>
> >>>> LogID=10133541"
> >>>>
> >>>>
> >>>>>> (SQL2.txt contains the results, but again I'll paste them)
> >>>>>> <begin paste>
> >>>>>> ANCHORSIGN\yomomma PGP update.pgp.com 63.251.255.18
> > VPN
> >>>>>>
> >>>>>>
> >>>> Clients
> >>>>
> >>>>
> >>>>>> External
> >>>>>> </end paste>
> >>>>>>
> >>>>>> The original query results are 934 bytes. The trimmed query
> > results
> >>>>>>
> >>>>>>
> >>>> are 74
> >>>>
> >>>>
> >>>>>> bytes. Just for this simple query, with selected
> fields from one
> >>>>>>
> >>>>>>
> >>>> record, the
> >>>>
> >>>>
> >>>>>> storage requirements for this data is 12.62 times
> greater due to
> > the
> >>>>>>
> >>>>>>
> >>>> "fluff."
> >>>>
> >>>>
> >>>>>> Extend that out to a full logged record, and then multiply that
> >> times
> >>>>>>
> >>>>>>
> >>>> tens of
> >>>>
> >>>>
> >>>>>> thousands. *THIS* is the 1-2 gigs of "fluff" I'm
> talking about on
> > a
> >>>>>>
> >>>>>>
> >>>> *daily*
> >>>>
> >>>>
> >>>>>> basis.
> >>>>>>
> >>>>>> The "required maintenance" I'm talking about is where
> I take daily
> >>>>>>
> >>>>>>
> >>>> raw data
> >>>>
> >>>>
> >>>>>> from the WebProxyLog, trim it up, and then post it into my own
> >>>>>>
> >>>>>>
> >>>> tables. In my
> >>>>
> >>>>
> >>>>>> case, the _yearly_ web proxy log data for my users is
> only 10 gig.
> >>>>>>
> >>>>>>
> >>>> _Weekly_
> >>>>
> >>>>
> >>>>>> "raw" data in the original ISA log format averages 12
> gig. When
> > I
> >>>>>>
> >>>>>>
> >>>> run this
> >>>>
> >>>>
> >>>>>> process, I am able to retrieve data via SQA, Access
> front end, or
> >> via
> >>>>>>
> >>>>>>
> >>>> ADODB
> >>>>
> >>>>
> >>>>>> objects, yet ISA will go into lockdown mode while logging when
> > this
> >>>>>>
> >>>>>>
> >>>> job runs
> >>>>
> >>>>
> >>>>>> at night. That's why I have to turn it off.
> >>>>>>
> >>>>>> I hope that better defines the issue as I see it.
> >>>>>>
> >>>>>> Thanksl
> >>>>>> T
> >>>>>>
> >>>>>> --
> >>>>>> "Tom Shinder pities Mr. T"
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 7/5/06 2:01 AM, "Adar Greenshpon"
> > <Adar.Greenshpon@xxxxxxxxxxxxx>
> >>>>>>
> >>>>>>
> >>>> spoketh
> >>>>
> >>>>
> >>>>>> to all:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Hi Thor,
> >>>>>>
> >>>>>> Just so we'll be on the same page: are you directing
> your ISA logs
> >> to
> >>>>>>
> >>>>>>
> >>>> an
> >>>>
> >>>>
> >>>>>> external machine with SQL Server (SQL 2000 or 2005)?.
> If so, how
> > do
> >>>>>>
> >>>>>>
> >>>> you do
> >>>>
> >>>>
> >>>>>> the maintenance? What's your desired log retention
> policy (e.g. 7
> >>>>>>
> >>>>>>
> >>>> days)?
> >>>>
> >>>>
> >>>>>> Per the gigs of logs you're doing, have you considered
> reducing it
> >> by
> >>>>>>
> >>>>>>
> >>>> not
> >>>>
> >>>>
> >>>>>> logging images http requests?
> >>>>>> Adar.
> >>>>>> __________________________________________
> >>>>>> Adar Greenshpon | Program Manager | Microsoft ISA Server
> >>>>>> adarg@xxxxxxxxxxxxx <mailto:adarg@xxxxxxxxxxxxx>
> >>>>>>
> >>>>>>
> >>>> <mailto:adarg@xxxxxxxxxxxxx>
> >>>>
> >>>>
> >>>>>> | Tel: +972.4.856.1077 | SMS/Cell: +972.54.666.4579
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ________________________________
> >>>>>>
> >>>>>>
> >>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> >>>>>> Sent: Tuesday, July 04, 2006 8:55 PM
> >>>>>> To: Avi Sander
> >>>>>> Cc: Adar Greenshpon; ISA-MVP
> >>>>>> Subject: Re: [ISAServer] Firewall database corruption
> due to power
> >>>>>>
> >>>>>>
> >>>> outtage
> >>>>
> >>>>
> >>>>>> OK- the either/or situation makes more sense. I'm
> sure that I'm
> >> not
> >>>>>>
> >>>>>>
> >>>> hitting
> >>>>
> >>>>
> >>>>>> the log buffer- the 9 second minimum window is what is
> getting me.
> >>>>>>
> >>>>>> How exactly do you define a "failure to commit?" Are
> you actively
> >>>>>>
> >>>>>>
> >>>> looking
> >>>>
> >>>>
> >>>>>> for
> >>>>>> a "failure" code from SQL, or is it the lack of a committed
> >>>>>>
> >>>>>>
> >>>> transaction
> >>>>
> >>>>
> >>>>>> within
> >>>>>> the time span? I ask because if I am in the middle of DB
> >>>>>>
> >>>>>>
> >>>> maintenance, I can
> >>>>
> >>>>
> >>>>>> execute a transaction that might have to wait a while to get a
> >>>>>>
> >>>>>>
> >>>> committal, but
> >>>>
> >>>>
> >>>>>> SQL knows about the transaction pending and won't error out.
> >>>>>>
> >>>>>> The SQL table structure included in 2004 results in HUGE table
> > sizes
> >>>>>>
> >>>>>>
> >>>> for web
> >>>>
> >>>>
> >>>>>> proxy logging alone. I'm talking gigs worth of fluff
> per day, so
> >>>>>>
> >>>>>>
> >>>> daily DB
> >>>>
> >>>>
> >>>>>> maintenance is an absolute requirement. I have never had a
> > logging
> >>>>>> transaction "live" through a maintenance period
> without going into
> >>>>>>
> >>>>>>
> >>>> lockdown,
> >>>>
> >>>>
> >>>>>> hence it being disabled on all my web proxy listener machines.
> >>>>>>
> >>>>>> I understand that the reasoning behind "lockdown" is for
> >>>>>>
> >>>>>>
> >>>> "evidentiary"
> >>>>
> >>>>
> >>>>>> reasons, but that seems counter intuitive to me. The
> actual log
> > is
> >>>>>>
> >>>>>>
> >>>> more
> >>>>
> >>>>
> >>>>>> important to me than the absence of a logged event
> when the system
> >>>>>>
> >>>>>>
> >>>> locks
> >>>>
> >>>>
> >>>>>> down.
> >>>>>> So, not only are the services not available (though a
> > "fail-safe"),
> >>>>>>
> >>>>>>
> >>>> but I
> >>>>
> >>>>
> >>>>>> loose any log of any attempt in the meantime. Are we
> going to see
> >>>>>>
> >>>>>>
> >>>> some more
> >>>>
> >>>>
> >>>>>> "user definable" setting in regard to logging and lockdown? If
> > not,
> >> I
> >>>>>>
> >>>>>>
> >>>> don't
> >>>>
> >>>>
> >>>>>> see how even mid-size businesses will be able to log to SQL
> > without
> >>>>>>
> >>>>>>
> >>>> disabling
> >>>>
> >>>>
> >>>>>> the feature.
> >>>>>>
> >>>>>> t
> >>>>>>
> >>>>>>
> >>>>>> On 7/4/06 5:51 AM, "Avi Sander"
> <asander@xxxxxxxxxxxxx> spoketh to
> >>>>>>
> >>>>>>
> >>>> all:
> >>>>
> >>>>
> >>>>>> ISA will lockdown if either of the following occurs (whichever
> > comes
> >>>>>>
> >>>>>>
> >>>> 1st):
> >>>>
> >>>>
> >>>>>> - Four consecutive failures to commit the data into the
> > DB.
> >>>>>>
> >>>>>>
> >>>> Our
> >>>>
> >>>>
> >>>>>> default COMMIT timeout is 30 secs. We'll retry every 3
> secs. So at
> >>>>>>
> >>>>>>
> >>>> minimum
> >>>>
> >>>>
> >>>>>> you've got 9 secs here.
> >>>>>> - LOG buffers are exhausted\out of memory. By
> default on a
> >>>>>>
> >>>>>>
> >>>> 1GB RAM
> >>>>
> >>>>
> >>>>>> machine we'll accumulate tens of thousands of recs. Is
> there a lot
> >> of
> >>>>>>
> >>>>>>
> >>>> traffic
> >>>>
> >>>>
> >>>>>> going through when this happens? How much RAM does this machine
> >> have?
> >>>>>>
> >>>>>> What is the event description you see in the EventViewer?
> >>>>>>
> >>>>>> The Log buffers can be extended some more, though not
> recommended
> >>>>>>
> >>>>>>
> >>>> since can
> >>>>
> >>>>
> >>>>>> use up a lot of memory.
> >>>>>>
> >>>>>> -avi
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ________________________________
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> From: Adar Greenshpon
> >>>>>> Sent: Tuesday, July 04, 2006 2:03 PM
> >>>>>> To: isaserver@xxxxxxxxxxxxxxx
> >>>>>> Cc: Avi Sander
> >>>>>> Subject: RE: [ISAServer] Firewall database corruption
> due to power
> >>>>>>
> >>>>>>
> >>>> outtage
> >>>>
> >>>>
> >>>>>> Avi - wanna play logging diva?
> >>>>>> Generally speaking, there are four internal commit
> retries until
> > ISA
> >>>>>>
> >>>>>>
> >>>> enters
> >>>>
> >>>>
> >>>>>> lockdown. We have seen a few thousand log records in
> the buffers
> > in
> >>>>>>
> >>>>>>
> >>>> these
> >>>>
> >>>>
> >>>>>> temporary moments in our stress labs (nothing you could really
> >>>>>>
> >>>>>>
> >>>> witness on a
> >>>>
> >>>>
> >>>>>> normal SBS box though).
> >>>>>>
> >>>>>> Adar.
> >>>>>> __________________________________________
> >>>>>> Adar Greenshpon | Program Manager | Microsoft ISA Server
> >>>>>> adarg@xxxxxxxxxxxxx <mailto:adarg@xxxxxxxxxxxxx>
> >>>>>>
> >>>>>>
> >>>> <mailto:adarg@xxxxxxxxxxxxx>
> >>>>
> >>>>
> >>>>>> | Tel: +972.4.856.1077 | SMS/Cell: +972.54.666.4579
> >>>>>>
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> >>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> >>>>>> Sent: Wednesday, June 28, 2006 8:26 PM
> >>>>>> To: ISA-MVP
> >>>>>> Subject: Re: [ISAServer] Firewall database corruption
> due to power
> >>>>>>
> >>>>>>
> >>>> outtage
> >>>>
> >>>>
> >>>>>> If that is what you guys are expecting to happen, I
> can tell you,
> >>>>>>
> >>>>>>
> >>>> that
> >>>>
> >>>>
> >>>>>> *ain't* the case - not when logging to SQL on another
> box, anyway.
> >>>>>>
> >>>>>>
> >>>> There's
> >>>>
> >>>>
> >>>>>> no "using as much memory as it can" going on. It
> seems to be more
> >>>>>>
> >>>>>>
> >>>> like "I
> >>>>
> >>>>
> >>>>>> did not get an acknowledgement that the last transaction was
> >> written,
> >>>>>>
> >>>>>>
> >>>> and
> >>>>
> >>>>
> >>>>>> it's been 2 seconds, so I'm going into lockdown."
> >>>>>>
> >>>>>> I'm looking forward to what info you can get from your "logging
> >>>>>>
> >>>>>>
> >>>> diva."
> >>>>
> >>>>
> >>>>>> t
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 6/28/06 8:43 AM, "Jim Harrison (ISA)"
> >> <Jim.Harrison@xxxxxxxxxxxxx>
> >>>>>> spoketh to all:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Actually, it's not a fixed value (or
> user-changeable), other than
> >>>>>>>
> >>>>>>>
> >>>> being
> >>>>
> >>>>
> >>>>>>> self-regulating.
> >>>>>>> ISA logging will use as much memory as it can until
> it determines
> >>>>>>>
> >>>>>>>
> >>>> that
> >>>>
> >>>>
> >>>>>>> growth is unchecked. Thus, how quickly ISA will cry
> foul depends
> >> in
> >>>>>>> part on how much free memory is available.
> >>>>>>>
> >>>>>>> I'll ask our logging diva for details, but this sums
> it up pretty
> >>>>>>>
> >>>>>>>
> >>>> well.
> >>>>
> >>>>
> >>>>>>> Jim Harrison
> >>>>>>> SASD (ISA SE)
> >>>>>>> If We Can't Fix It - It Ain't Broke!
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> >>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> >>>>>>> Sent: Wednesday, June 28, 2006 8:09 AM
> >>>>>>> To: ISA-MVP
> >>>>>>> Subject: Re: [ISAServer] Firewall database corruption due to
> > power
> >>>>>>> outtage
> >>>>>>>
> >>>>>>> It must be about a 1k buffer, then. I repeatedly put ISA in
> >> lockdown
> >>>>>>> within
> >>>>>>> seconds of maintenance jobs being run, or other "data
> intensive"
> >>>>>>> operations
> >>>>>>> on the DB. I would start the job, in in a few seconds, get
> >>>>>>>
> >>>>>>>
> >>>> disconnected
> >>>>
> >>>>
> >>>>>>> form my TS session because of it-- It wasn't a "one
> time" thing-
> > it
> >>>>>>>
> >>>>>>>
> >>>> was
> >>>>
> >>>>
> >>>>>>> consistent and repeatable. That's the only reason I went ahead
> > and
> >>>>>>> disabled
> >>>>>>> lockdown.
> >>>>>>>
> >>>>>>> Is the buffer user definable somewhere? What exactly
> does your
> >>>>>>>
> >>>>>>>
> >>>> script
> >>>>
> >>>>
> >>>>>>> do?
> >>>>>>>
> >>>>>>> t
> >>>>>>>
> >>>>>>>
> >>>>>>> On 6/28/06 7:54 AM, "Jim Harrison (ISA)"
> >>>>>>>
> >>>>>>>
> >>>> <Jim.Harrison@xxxxxxxxxxxxx>
> >>>>
> >>>>
> >>>>>>> spoketh to all:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Actually, that's not true; ISA doesn't go into lockdown until
> > the
> >>>>>>>> logging buffer is filled.
> >>>>>>>> I agree that a "backup logging" concept is useful.
> >>>>>>>> I have a script that works for ISA 2000; it shouldn't be
> > difficult
> >>>>>>>>
> >>>>>>>>
> >>>> to
> >>>>
> >>>>
> >>>>>>>> "agnosticize" it.
> >>>>>>>>
> >>>>>>>> Jim Harrison
> >>>>>>>> Security Platform Group (ISA SE)
> >>>>>>>>
> >>>>>>>> If We Can't Fix It - It Ain't Broke!
> >>>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
> >>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
> > <mailto:thor@xxxxxxxxxxxxxxx%5d>
> >>>>>>>> Sent: Wednesday, June 28, 2006 7:34 AM
> >>>>>>>> To: ISA-MVP
> >>>>>>>> Subject: Re: [ISAServer] Firewall database corruption due to
> > power
> >>>>>>>> outtage
> >>>>>>>>
> >>>>>>>> I'd like to see some way to at least "buffer" the logging
> > requests
> >>>>>>>>
> >>>>>>>>
> >>>>>>> when
> >>>>>>>
> >>>>>>>
> >>>>>>>> logging to an off-box ODBC destination, like SQL.
> ISA is quite
> >>>>>>>> intolerant
> >>>>>>>> of even the slightest interruption in logging. But, I guess
> > that
> >>>>>>>>
> >>>>>>>>
> >>>>>>> makes
> >>>>>>>
> >>>>>>>
> >>>>>>>> sense if the goal is to have "evidentiary credibility."
> >>>>>>>>
> >>>>>>>> It wouldn't have to be something l337 like a secondary local
> >>>>>>>>
> >>>>>>>>
> >>>> logging
> >>>>
> >>>>
> >>>>>>>> option
> >>>>>>>> or anything like that-- I think being able to set a timeout
> > value
> >>>>>>>>
> >>>>>>>>
> >>>>>>> would
> >>>>>>>
> >>>>>>>
> >>>>>>>> would. With larger db's, a maintenance job would kick the
> > server
> >>>>>>>>
> >>>>>>>>
> >>>> into
> >>>>
> >>>>
> >>>>>>>> lockdown, even though the SQL service is still running...
> >>>>>>>>
> >>>>>>>> Jim, I know you wanted me to get my infrastructure
> layout to you
> >>>>>>>>
> >>>>>>>>
> >>>> again
> >>>>
> >>>>
> >>>>>>>> for
> >>>>>>>> my solution to the SQL db size problems, but I haven't had a
> >> chance
> >>>>>>>>
> >>>>>>>>
> >>>> to
> >>>>
> >>>>
> >>>>>>>> find
> >>>>>>>> that old email. I should just re-create it ;)
> Regardless, even
> >>>>>>>>
> >>>>>>>>
> >>>> with
> >>>>
> >>>>
> >>>>>>>> that
> >>>>>>>> "solution" in place, I've had to disable lockdown mode when
> >> logging
> >>>>>>>>
> >>>>>>>>
> >>>> to
> >>>>
> >>>>
> >>>>>>>> SQL
> >>>>>>>> for that very reason.
> >>>>>>>>
> >>>>>>>> t
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 6/28/06 7:19 AM, "Thomas W Shinder" <tshinder@xxxxxxxxxxx>
> >>>>>>>>
> >>>>>>>>
> >>>> spoketh
> >>>>
> >>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>>
> >>>>>>>> all:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> And it's a really good security and forensics decision.
> >>>>>>>>>
> >>>>>>>>> Thomas W Shinder, M.D.
> >>>>>>>>> Site: www.isaserver.org
> >>>>>>>>> Blog: http://blogs.isaserver.org/shinder/
> >>>>>>>>> Book: http://tinyurl.com/3xqb7
> >>>>>>>>> MVP -- ISA Firewalls
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Amy Babinchak [mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx]
> >>>>>>>>>> <mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx%5d>
> >>>>>>>>>> <mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx%5d>
> >>>>>>>>>> Sent: Wednesday, June 28, 2006 6:28 AM
> >>>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx
> >>>>>>>>>> Subject: RE: [ISAServer] Firewall database
> corruption due to
> >>>>>>>>>> power outtage
> >>>>>>>>>>
> >>>>>>>>>> I like that ISA shuts down if it can't log. It's
> an indicator
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> that
> >>>>
> >>>>
> >>>>>>>>>> something has gone wrong.
> >>>>>>>>>>
> >>>>>>>>>> Amy
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -----Original Message-----
> >>>>>>>>>> From: Susan Bradley, CPA aka Ebitz - SBS Rocks [MVP]
> >>>>>>>>>> [mailto:sbradcpa@xxxxxxxxxxx]
> <mailto:sbradcpa@xxxxxxxxxxx%5d>
> >>>>>>>>>> <mailto:sbradcpa@xxxxxxxxxxx%5d>
> >>>>>>>>>> Sent: Wednesday, June 28, 2006 12:12 AM
> >>>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx
> >>>>>>>>>> Subject: [ISAServer] Firewall database corruption
> due to power
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>> outtage
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>> With the caveat that we all know we need a good
> UPS but like
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> stuff
> >>>>
> >>>>
> >>>>>>>>>> happens.. I've seen two folks recently have issues
> getting a
> >>>>>>>>>> server back
> >>>>>>>>>>
> >>>>>>>>>> up have a 'dirty shutdown' due to power outtage
> that ended up
> >>>>>>>>>> corrupting
> >>>>>>>>>>
> >>>>>>>>>> the ISA firewall database.
> >>>>>>>>>>
> >>>>>>>>>> Short of uninstalling and reinstalling... short of
> adjusting
> >>>>>>>>>> ISA so it
> >>>>>>>>>> won't shut down if it can't log... short of a better UPS...
> > any
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>> other
> >>>>>>>
> >>>>>>>
> >>>>>>>>>> recommendations on handling this better?
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> To subscribe to the list - send an email to
> > list@xxxxxxxxxxxxxxx
> >>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>>>> youremailaddress
> >>>>>>>>>>
> >>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>>>> youremailaddress
> >>>>>>>>>>
> >>>>>>>>>> Don't forget the comma!
> >>>>>>>>>> ---
> >>>>>>>>>> To subscribe to the list - send an email to
> > list@xxxxxxxxxxxxxxx
> >>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>>>> youremailaddress
> >>>>>>>>>>
> >>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>>>> youremailaddress
> >>>>>>>>>>
> >>>>>>>>>> Don't forget the comma!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> ---
> >>>>>>>>> To subscribe to the list - send an email to
> > list@xxxxxxxxxxxxxxx
> >>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> youremailaddress
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> youremailaddress
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Don't forget the comma!
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> ---
> >>>>>>>> To subscribe to the list - send an email to
> list@xxxxxxxxxxxxxxx
> >>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>> youremailaddress
> >>>>>>>>
> >>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>> youremailaddress
> >>>>>>>>
> >>>>>>>> Don't forget the comma!
> >>>>>>>> ---
> >>>>>>>> To subscribe to the list - send an email to
> list@xxxxxxxxxxxxxxx
> >>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>>
> >>>>>>>>
> >>>>>>> youremailaddress
> >>>>>>>
> >>>>>>>
> >>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>>
> >>>>>>>>
> >>>>>>> youremailaddress
> >>>>>>>
> >>>>>>>
> >>>>>>>> Don't forget the comma!
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>> ---
> >>>>>>> To subscribe to the list - send an email to
> list@xxxxxxxxxxxxxxx
> >>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >>>>>>> youremailaddress
> >>>>>>>
> >>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >>>>>>> youremailaddress
> >>>>>>>
> >>>>>>> Don't forget the comma!
> >>>>>>> ---
> >>>>>>> To subscribe to the list - send an email to
> list@xxxxxxxxxxxxxxx
> >>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>
> >>>>>>>
> >>>> youremailaddress
> >>>>
> >>>>
> >>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >>>>>>>
> >>>>>>>
> >>>> youremailaddress
> >>>>
> >>>>
> >>>>>>> Don't forget the comma!
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> ---
> >>>>>> To subscribe to the list - send an email to
> list@xxxxxxxxxxxxxxx
> >>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >>>>>>
> >>>>>>
> >>>> youremailaddress
> >>>>
> >>>>
> >>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >>>>>>
> >>>>>>
> >>>> youremailaddress
> >>>>
> >>>>
> >>>>>> Don't forget the comma!
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---
> >>>>>> To subscribe to the list - send an email to
> list@xxxxxxxxxxxxxxx
> >>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >>>>>>
> >>>>>>
> >>>> youremailaddress
> >>>>
> >>>>
> >>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >>>>>>
> >>>>>>
> >>>> youremailaddress
> >>>>
> >>>>
> >>>>>> Don't forget the comma!
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> ---
> >>> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx
> >>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >> youremailaddress
> >>>
> >>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >> youremailaddress
> >>>
> >>> Don't forget the comma!
> >>>
> >>>
> >>>
> >> ---
> >> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx
> >> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> >> youremailaddress
> >>
> >> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> >> youremailaddress
> >>
> >> Don't forget the comma!
> >> ---
> >> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx
> >> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> > youremailaddress
> >>
> >> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> >> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> > youremailaddress
> >>
> >> Don't forget the comma!
> >>
> >>
> >
> >
> >
> >
>
>
> ---
> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx
> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
> youremailaddress
>
> To leave the list - send an email to list@xxxxxxxxxxxxxxx
> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
> youremailaddress
>
> Don't forget the comma!
>
>
Other related posts:
