[isapros] Re: [ISAServer] Firewall database corruption due to power outtage
- From: "Thor (Hammer of God)" <thor@xxxxxxxxxxxxxxx>
- To: "isapros@xxxxxxxxxxxxx" <isapros@xxxxxxxxxxxxx>
- Date: Mon, 10 Jul 2006 07:24:24 -0700
What do you mean "the alert?" What alert?
t
On 7/10/06 7:21 AM, "Thomas W Shinder" <tshinder@xxxxxxxxxxx> spoketh to
all:
> Hi Tim,
>
> But you can do that by configuring the Alert to not send the system into
> lockdown.
>
> Thomas W Shinder, M.D.
> Site: www.isaserver.org
> Blog: http://blogs.isaserver.org/shinder/
> Book: http://tinyurl.com/3xqb7
> MVP -- ISA Firewalls
>
>
>
>> -----Original Message-----
>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
>> Sent: Monday, July 10, 2006 9:16 AM
>> To: Adar Greenshpon; ISA-MVP; Avi Sander; Nathan Bigman
>> Subject: Re: [ISAServer] Firewall database corruption due to
>> power outtage
>>
>> I guess I misunderstood. The "mentioned below" items kept
>> going back to how
>> to remove the fluff, rather than dealing with why ISA goes
>> into lockdown so
>> readily. I've still not seen any information on how/when/why
>> ISA goes into
>> lockdown when logging to ISA. You had indicated that the
>> other information
>> was incorrect, so I was standing by for that info.
>>
>> It doesn't matter if you put it in a different database. When the
>> optimization plan (you know, the "normal" plan that rebuilds indexes,
>> insures integrity, etc) runs, ISA goes into lockdown. Even if
>> I waited a
>> week in between times that I purged the data, I'll still have
>> to perform
>> some maintenance, and the system would still go into lockdown.
>>
>> The solution is to disable lockdown.
>>
>> Thanks!
>>
>> t
>>
>>
>> On 7/10/06 12:13 AM, "Adar Greenshpon" <Adar.Greenshpon@xxxxxxxxxxxxx>
>> spoketh to all:
>>
>>> Thor: as I've indicated below, we plan on improving the connectivity
>>> issue you raised. Given the current SQL logging limitation and your
>>> deployment, might it be possible to split the webproxy log into two
>>> databases: one for historical purposes and one for current
>> events. The
>>> heavy-weight maintenance could be done on the former one
>> while the later
>>> one would be available for ISA to dump logs into.
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
>>> Sent: Monday, July 10, 2006 1:50 AM
>>> To: ISA-MVP; Avi Sander; Adar Greenshpon; Nathan Bigman
>>> Subject: Re: [ISAServer] Firewall database corruption due to power
>>> outtage
>>>
>>>
>>> Yes, Jim... Thank you. Sorry that I got so short, but I was getting
>>> quite
>>> frustrated trying to make the same point over and over again.
>>>
>>> The focus need not be on "how to trim the logs." That's
>> not the *real*
>>> issue. The focus needs to be: "Regarding lockdown, what
>> factors are in
>>> play
>>> when ISA logs to SQL, and under what circumstances does ISA invoke
>>> lockdown
>>> when posting to a SQL DB?"
>>>
>>> Let's put the padding issues *totally* out of the picture.
>> Let's say
>>> that
>>> I've got TB's of storage and that I could not care less
>> about how much
>>> fluff
>>> is in my WebProxyLog-- let it get as big as it wants.
>> However, I must
>>> back
>>> it up. I've also experienced ISA going into lockdown
>> during classic db
>>> maintenance like backing up, re-indexing, optimizing size,
>> etc. These
>>> processes must be run regardless of the presence of
>> "padding issues" or
>>> not.
>>> In fact, even if the padding problem is solved in ISA2006, in
>>> combination
>>> with or exclusive of contributing options in SQL 2005 (which I'm
>>> migrating
>>> to as I type), ISA will still go into lockdown when the SQL server
>>> engages
>>> in processes which may lock the db, or delay response in some way.
>>>
>>> Again- (and hopefully someone will listen this time) the
>> concern speaks
>>> to
>>> the "real world" usability of "lockdown mode" within the
>> enterprise. If
>>> Microsoft intends to tout the lockdown feature of ISA as a tool
>>> enterprise
>>> clients can use to ensure evidentiary integrity, then you
>> must address
>>> the
>>> mode's "survivability" during standard, everyday,
>> real-world processes
>>> that
>>> must be run on the SQL servers the ISA box is logging to, or it will
>>> simply
>>> not be used. This relates to processes that may or may not have
>>> anything to
>>> do with the log files themselves.
>>>
>>> If nobody cares if lockdown is utilized in the enterprise, then this
>>> thread
>>> is done. If, however, the goal is supply enterprise customers with a
>>> lockdown mode they can use, then user-defined connection
>> parameters must
>>> be
>>> provided in order to customize the lockdown threshold to our
>>> environments.
>>>
>>> T
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 7/9/06 9:28 AM, "Jim Harrison (ISA)" <Jim.Harrison@xxxxxxxxxxxxx>
>>> spoketh
>>> to all:
>>>
>>>> It may help in the short run, but the issue Tim raises
>> isn't so much
>>>> "how much data is being logged", but "how long ISA will
>> wait while the
>>>> logging process fails to respond before it raises an alert" and "I
>>> wanna
>>>> set this trigger point".
>>>>
>>>> What he's asking for (as have more than a few customers)
>> is some way
>>> for
>>>> the user to specify a "almost choking on my own data"
>> alert limit that
>>>> could be used to move ISA logging to an alternate destination (SQL,
>>>> file, bitbucket) until the primary logger responds once
>> more. What we
>>>> have now is a "choked on my own data" alert, which is a
>> bit too late.
>>>>
>>>> In fact, I've seen one unfavorable comparison to <product deleted>
>>> where
>>>> they actually:
>>>> 1. fail over to a backup logging process
>>>> 2. monitor the primary logger
>>>> 3. fail back when the primary logger reappears
>>>> 4. import the alternate-store log data into the primary store as a
>>>> low-pri part of the fail-back.
>>>>
>>>> Techincally, this is scriptable, except for the fact that our alert
>>>> happens too late (and isn't "tweakable"). If one were willing to
>>>> tolerate the momentary traffic block that's created today,
>> a script is
>>> a
>>>> workable solution (except for the "pre-panic" factor).
>>>>
>>>> Jim Harrison
>>>> SASD (ISA SE)
>>>> If We Can't Fix It - It Ain't Broke!
>>>>
>>>> -----Original Message-----
>>>> From: Susan Bradley, CPA aka Ebitz - SBS Rocks [MVP]
>>>> [mailto:sbradcpa@xxxxxxxxxxx]
>>>> Sent: Sunday, July 09, 2006 2:22 AM
>>>> To: isaserver@xxxxxxxxxxxxxxx
>>>> Cc: Avi Sander; Adar Greenshpon; Nathan Bigman
>>>> Subject: Re: [ISAServer] Firewall database corruption due to power
>>>> outtage
>>>>
>>>> Shouldn't depadding not cause the system to freak out and go into
>>>> lockdown? That's his real point.. not necessarily the
>> fact he has to
>>> do
>>>>
>>>> it..but the fact that ISA is there locking itself down in
>> the process
>>>> ....
>>>>
>>>> Thor (Hammer of God) wrote:
>>>>
>>>>> Never mind. Not sure why I bother...
>>>>>
>>>>>
>>>>> t
>>>>>
>>>>> On 7/9/06 12:01 AM, "Avi Sander" <asander@xxxxxxxxxxxxx>
>> spoketh to
>>>> all:
>>>>>
>>>>>
>>>>>
>>>>>> Hi Thor -
>>>>>>
>>>>>> Please see
>>>>>>
>>>
>> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/tsqlr
>>>> ef
>>>>>> /ts_set-set_2uw7.asp .
>>>>>>
>>>>>> It describes how one can toggle the ANSI_PADDING option. This can
>>>>>> control how fields are padded\trimmed in SQL, but should be set
>>> prior
>>>> to
>>>>>> tbl creation.
>>>>>>
>>>>>> - avi
>>>>>> -----Original Message-----
>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
>>>>>> Sent: Friday, July 07, 2006 6:23 PM
>>>>>> To: Avi Sander; Adar Greenshpon; ISA-MVP
>>>>>> Cc: Nathan Bigman
>>>>>> Subject: Re: [ISAServer] Firewall database corruption
>> due to power
>>>>>> outtage
>>>>>>
>>>>>> I think we're getting off track a little here...
>>>>>>
>>>>>> I appreciate the link to that "fix," but I've *already
>> solved* the
>>>>>> padding
>>>>>> issue. I've been including a script for that in my Blackhat
>>> trainings
>>>>>> for
>>>>>> years. To that effect, simply trimming the data in the default
>>> table
>>>>>> isn't
>>>>>> the best way to go, IMO. The data structure is not very
>> efficient
>>> to
>>>>>> begin
>>>>>> with (for my needs). I've created my own (better designed)
>>> structure,
>>>>>> and
>>>>>> trim the data as I post into that table (and there is no
>> "trim data"
>>>> DSN
>>>>>> option that I know of). But that's OK-- as an administrator, I
>>>> *expect*
>>>>>> that I will have to adjust logging processes to better
>> fit my needs
>>>>>> (just
>>>>>> like I do with IIS logs that I post to SQL). You guys provide the
>>>>>> mechanism,
>>>>>> and I customize it to suit me. Like I said, my yearly
>> retained data
>>> to
>>>>>> date
>>>>>> is only 10 gig. No problems.
>>>>>>
>>>>>> What I'm saying is that *when I run the process to trim the data*
>>> ISA
>>>>>> goes
>>>>>> into "lockdown." This whole thing got started when we
>> were talking
>>>>>> about
>>>>>> lockdown mode, and how "trigger happy" it is when one is
>> logging to
>>>> SQL.
>>>>>>
>>>>>> I only brought it up in an effort to identify this:
>>>>>> If a customer logs to SQL, there is a padding problem.
>> The padding
>>>>>> problem
>>>>>> *requires* DB maintenance processes to control. When the DB
>>>> maintenance
>>>>>> is
>>>>>> executed against any DB of consequence (just several thousand
>>> records
>>>>>> seems
>>>>>> to be enough), ISA goes into lockdown, presumably because of the
>>> "lag"
>>>>>> created by server load when the SQL server goes about parsing out
>>> gigs
>>>>>> of
>>>>>> fluff data. This is true even on my production SQL
>> cluster (which
>>> is
>>>>>> kickass, mind you ;) so it's not a "hardware" problem.
>> Thus, I was
>>>>>> forced
>>>>>> to disable lockdown.
>>>>>>
>>>>>> Therefore, IF a customer is logging to SQL, AND they want use
>>>> "lockdown"
>>>>>> THEN better user-definable timeout parameters need to be
>> available.
>>>>>> That's
>>>>>> all I was saying.
>>>>>>
>>>>>> Of course, since you have identified that ISA2004 EE, ISA2006 SE,
>>> and
>>>>>> ISA2006 EE properly trim data before posting it to SQL,
>> then it all
>>>>>> seems to
>>>>>> be a moot point, right? ;)
>>>>>>
>>>>>> The main box posting web access is currently ISA 2004 SE. But I
>>> think
>>>>>> I've
>>>>>> got a copy of ISA 2004 EE that I picked up in Hong Kong
>> along with
>>> an
>>>>>> old
>>>>>> Rush CD that I'll throw on that guy and see what
>> happens. (jk;) I'll
>>>> let
>>>>>> you
>>>>>> guys know what happens.
>>>>>>
>>>>>> Thx
>>>>>> t
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 7/7/06 12:29 AM, "Avi Sander" <asander@xxxxxxxxxxxxx>
>> spoketh to
>>>> all:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> there's also an option on the ODBC DSN : 'trim fields' that
>>> partially
>>>>>>>
>>>>>>>
>>>>>> helps,
>>>>>>
>>>>>>
>>>>>>> if i recall correctly.
>>>>>>> I'll look into this further
>>>>>>>
>>>>>>> ________________________________
>>>>>>>
>>>>>>> From: Adar Greenshpon
>>>>>>> Sent: Fri 07/07/2006 09:07
>>>>>>> To: Avi Sander; Thor (Hammer of God); isaserver@xxxxxxxxxxxxxxx
>>>>>>> Cc: Nathan Bigman
>>>>>>> Subject: RE: [ISAServer] Firewall database corruption
>> due to power
>>>>>>>
>>>>>>>
>>>>>> outtage
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thor,
>>>>>>>
>>>>>>> Per the excessive padding, see this thread:
>>>>>>> http://forums.isaserver.org/m_140009200/tm.htm - I think that's
>>> what
>>>>>>>
>>>>>>>
>>>>>> you're
>>>>>>
>>>>>>
>>>>>>> looking for as we already heard this a few times (as Avi pointed
>>> out,
>>>>>>>
>>>>>>>
>>>>>> the
>>>>>>
>>>>>>
>>>>>>> problem does not exist in ISA 2004EE or ISA 2006
>> versions). Nathan
>>> -
>>>>>>>
>>>>>>>
>>>>>> will an
>>>>>>
>>>>>>
>>>>>>> official KB help here?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Adar.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ________________________________
>>>>>>>
>>>>>>> From: Avi Sander
>>>>>>> Sent: Friday, July 07, 2006 12:27 AM
>>>>>>> To: Thor (Hammer of God)
>>>>>>> Cc: Adar Greenshpon
>>>>>>> Subject: RE: [ISAServer] Firewall database corruption
>> due to power
>>>>>>>
>>>>>>>
>>>>>> outtage
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> OK - that explains it.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This is a known SQL ODBC interfacing issue.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ISA 2004 EE and ISA2006 SE & EE moved away from ODBC
>> and use oledb
>>>>>>>
>>>>>>>
>>>>>> interfaces
>>>>>>
>>>>>>
>>>>>>> for SQL server logging instead. This issue does not exist there
>>>>>>>
>>>>>>>
>>>>>> anymore as it
>>>>>>
>>>>>>
>>>>>>> is all trimmed by design (and throughput is also greatly
>>> increased).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The 2 potential causes of errors i mentioned earlier in
>> the thread
>>>>>>>
>>>>>>>
>>>>>> (buffers
>>>>>>
>>>>>>
>>>>>>> full, 4 retries) are only relevant to the OLEDB
>> logging, and not to
>>>>>>>
>>>>>>>
>>>>>> your ISA -
>>>>>>
>>>>>>
>>>>>>> my bad.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> let me check with the ISA2004 SE code and get back to you with a
>>>>>>>
>>>>>>>
>>>>>> better
>>>>>>
>>>>>>
>>>>>>> picture...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -avi
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ________________________________
>>>>>>>
>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
>>>>>>> Sent: Thu 06/07/2006 23:12
>>>>>>> To: Avi Sander
>>>>>>> Subject: Re: [ISAServer] Firewall database corruption
>> due to power
>>>>>>>
>>>>>>>
>>>>>> outtage
>>>>>>
>>>>>>
>>>>>>> These are ISA2004 SE boxes. I haven't tried with
>> ISA2006 yet, but
>>>> I'm
>>>>>>> running it here at The Yeti and can check it out (unless you
>>> already
>>>>>>> know...)
>>>>>>>
>>>>>>> t
>>>>>>>
>>>>>>>
>>>>>>> On 7/6/06 2:08 PM, "Avi Sander" <asander@xxxxxxxxxxxxx>
>> spoketh to
>>>>>>>
>>>>>>>
>>>>>> all:
>>>>>>
>>>>>>
>>>>>>>> Hey Thor -
>>>>>>>>
>>>>>>>> What ISA version are you using? and whichSKU: SE or EE?
>>>>>>>>
>>>>>>>> -avi
>>>>>>>>
>>>>>>>> ________________________________
>>>>>>>>
>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
>>>>>>>> Sent: Thu 06/07/2006 18:39
>>>>>>>> To: Adar Greenshpon; ISA-MVP; Avi Sander
>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption
>> due to power
>>>>>>>>
>>>>>>>>
>>>>>> outtage
>>>>>>
>>>>>>
>>>>>>>> Hi Adar-
>>>>>>>>
>>>>>>>> Yes, logs going to an external SQL 2000 cluster via
>> direct gigabit
>>>>>>>>
>>>>>>>>
>>>>>> switched
>>>>>>
>>>>>>
>>>>>>>> link. And log retention is permanent. Though I've
>> only got 120
>>>>>>>>
>>>>>>>>
>>>>>> employees,
>>>>>>
>>>>>>
>>>>>>>> we
>>>>>>>> have tight policies on Internet usage and abuse and I,
>> like many,
>>>>>>>>
>>>>>>>>
>>>>>> must keep
>>>>>>
>>>>>>
>>>>>>>> access logs to ensure these policies are being
>> honored, as well as
>>>>>>>>
>>>>>>>>
>>>>>> for any
>>>>>>
>>>>>>
>>>>>>>> number of potential legal issues that might arise. I
>> trust that
>>> you
>>>>>>>>
>>>>>>>>
>>>>>> are not
>>>>>>
>>>>>>
>>>>>>>> suggesting that ISA log retention via SQL should be measured in
>>>> days,
>>>>>>>>
>>>>>>>>
>>>>>> right?
>>>>>>
>>>>>>
>>>>>>>> ;)
>>>>>>>>
>>>>>>>> Let me go into more detail about the "fluff" I mentioned- it is
>>> not
>>>>>>>>
>>>>>>>>
>>>>>> so much
>>>>>>
>>>>>>
>>>>>>>> the "gigs of logs" as it would relate to real data- it
>> is actual
>>>>>>>>
>>>>>>>>
>>>>>> "fluff:"
>>>>>>
>>>>>>
>>>>>>>> Extra, padded spacing in the fields posted to SQL from ISA.
>>> Though
>>>>>>>>
>>>>>>>>
>>>>>> you guys
>>>>>>
>>>>>>
>>>>>>>> have defined the fields in SQL as nvarchar data, there
>> is some bad
>>>>>>>>
>>>>>>>>
>>>>>> mojo in
>>>>>>
>>>>>>
>>>>>>>> the
>>>>>>>> logging process. Cursory review of the data in
>> something like SQL
>>>>>>>>
>>>>>>>>
>>>>>> Query
>>>>>>
>>>>>>
>>>>>>>> Analyzer may not immediately reveal this, but binary analyze of
>>> the
>>>>>>>>
>>>>>>>>
>>>>>> data
>>>>>>
>>>>>>
>>>>>>>> files
>>>>>>>> do.
>>>>>>>>
>>>>>>>> I've created some example text files to simplify what
>> I am talking
>>>>>>>>
>>>>>>>>
>>>>>> about
>>>>>>
>>>>>>
>>>>>>>> (Attached as SQL1.txt and SQL2.txt).
>>>>>>>>
>>>>>>>> Consider the following simple query to retrieve a single row of
>>>> basic
>>>>>>>>
>>>>>>>>
>>>>>> data
>>>>>>
>>>>>>
>>>>>>>> from the WebProxyLog as stored by ISA:
>>>>>>>>
>>>>>>>> "select
>>>>>>>>
>>>>>>>>
>>>>>>
>> ClientUserName,ClientAgent,DestHost,DestHostIP,SrcNetwork,DstNetwork
>>>>>>
>>>>>>
>>>>>>>> from WebProxyLog where LogID=10133541"
>>>>>>>>
>>>>>>>> (SQL1.txt contains the results, but I'll paste them
>> here as well)
>>>>>>>> <begin paste>
>>>>>>>> ANCHORSIGN\yomomma
>>>>>>>> PGP
>>>>>>>> update.pgp.com
>>>>>>>> 63.251.255.18 VPN Clients
>>>>>>>> External
>>>>>>>>
>>>>>>>> </end paste>
>>>>>>>>
>>>>>>>> Notice all the extra spacing; again, even though the fields are
>>> type
>>>>>>>> nvarchar-- ISA is padding the data. Now, let's trim
>> it up and see
>>>>>>>>
>>>>>>>>
>>>>>> what
>>>>>>
>>>>>>
>>>>>>>> happens in this query:
>>>>>>>>
>>>>>>>> "select
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>
>> rtrim(ClientUserName),rtrim(ClientAgent),rtrim(DestHost),rtrim(DestHos
>>>> tI
>>>>>> P),rt>>
>>>>>> r
>>>>>>
>>>>>>
>>>>>>>> im(SrcNetwork),rtrim(DstNetwork) from WebProxyLog where
>>>>>>>>
>>>>>>>>
>>>>>> LogID=10133541"
>>>>>>
>>>>>>
>>>>>>>> (SQL2.txt contains the results, but again I'll paste them)
>>>>>>>> <begin paste>
>>>>>>>> ANCHORSIGN\yomomma PGP update.pgp.com 63.251.255.18
>>> VPN
>>>>>>>>
>>>>>>>>
>>>>>> Clients
>>>>>>
>>>>>>
>>>>>>>> External
>>>>>>>> </end paste>
>>>>>>>>
>>>>>>>> The original query results are 934 bytes. The trimmed query
>>> results
>>>>>>>>
>>>>>>>>
>>>>>> are 74
>>>>>>
>>>>>>
>>>>>>>> bytes. Just for this simple query, with selected
>> fields from one
>>>>>>>>
>>>>>>>>
>>>>>> record, the
>>>>>>
>>>>>>
>>>>>>>> storage requirements for this data is 12.62 times
>> greater due to
>>> the
>>>>>>>>
>>>>>>>>
>>>>>> "fluff."
>>>>>>
>>>>>>
>>>>>>>> Extend that out to a full logged record, and then multiply that
>>>> times
>>>>>>>>
>>>>>>>>
>>>>>> tens of
>>>>>>
>>>>>>
>>>>>>>> thousands. *THIS* is the 1-2 gigs of "fluff" I'm
>> talking about on
>>> a
>>>>>>>>
>>>>>>>>
>>>>>> *daily*
>>>>>>
>>>>>>
>>>>>>>> basis.
>>>>>>>>
>>>>>>>> The "required maintenance" I'm talking about is where
>> I take daily
>>>>>>>>
>>>>>>>>
>>>>>> raw data
>>>>>>
>>>>>>
>>>>>>>> from the WebProxyLog, trim it up, and then post it into my own
>>>>>>>>
>>>>>>>>
>>>>>> tables. In my
>>>>>>
>>>>>>
>>>>>>>> case, the _yearly_ web proxy log data for my users is
>> only 10 gig.
>>>>>>>>
>>>>>>>>
>>>>>> _Weekly_
>>>>>>
>>>>>>
>>>>>>>> "raw" data in the original ISA log format averages 12
>> gig. When
>>> I
>>>>>>>>
>>>>>>>>
>>>>>> run this
>>>>>>
>>>>>>
>>>>>>>> process, I am able to retrieve data via SQA, Access
>> front end, or
>>>> via
>>>>>>>>
>>>>>>>>
>>>>>> ADODB
>>>>>>
>>>>>>
>>>>>>>> objects, yet ISA will go into lockdown mode while logging when
>>> this
>>>>>>>>
>>>>>>>>
>>>>>> job runs
>>>>>>
>>>>>>
>>>>>>>> at night. That's why I have to turn it off.
>>>>>>>>
>>>>>>>> I hope that better defines the issue as I see it.
>>>>>>>>
>>>>>>>> Thanksl
>>>>>>>> T
>>>>>>>>
>>>>>>>> --
>>>>>>>> "Tom Shinder pities Mr. T"
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 7/5/06 2:01 AM, "Adar Greenshpon"
>>> <Adar.Greenshpon@xxxxxxxxxxxxx>
>>>>>>>>
>>>>>>>>
>>>>>> spoketh
>>>>>>
>>>>>>
>>>>>>>> to all:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Thor,
>>>>>>>>
>>>>>>>> Just so we'll be on the same page: are you directing
>> your ISA logs
>>>> to
>>>>>>>>
>>>>>>>>
>>>>>> an
>>>>>>
>>>>>>
>>>>>>>> external machine with SQL Server (SQL 2000 or 2005)?.
>> If so, how
>>> do
>>>>>>>>
>>>>>>>>
>>>>>> you do
>>>>>>
>>>>>>
>>>>>>>> the maintenance? What's your desired log retention
>> policy (e.g. 7
>>>>>>>>
>>>>>>>>
>>>>>> days)?
>>>>>>
>>>>>>
>>>>>>>> Per the gigs of logs you're doing, have you considered
>> reducing it
>>>> by
>>>>>>>>
>>>>>>>>
>>>>>> not
>>>>>>
>>>>>>
>>>>>>>> logging images http requests?
>>>>>>>> Adar.
>>>>>>>> __________________________________________
>>>>>>>> Adar Greenshpon | Program Manager | Microsoft ISA Server
>>>>>>>> adarg@xxxxxxxxxxxxx <mailto:adarg@xxxxxxxxxxxxx>
>>>>>>>>
>>>>>>>>
>>>>>> <mailto:adarg@xxxxxxxxxxxxx>
>>>>>>
>>>>>>
>>>>>>>> | Tel: +972.4.856.1077 | SMS/Cell: +972.54.666.4579
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ________________________________
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
>>>>>>>> Sent: Tuesday, July 04, 2006 8:55 PM
>>>>>>>> To: Avi Sander
>>>>>>>> Cc: Adar Greenshpon; ISA-MVP
>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption
>> due to power
>>>>>>>>
>>>>>>>>
>>>>>> outtage
>>>>>>
>>>>>>
>>>>>>>> OK- the either/or situation makes more sense. I'm
>> sure that I'm
>>>> not
>>>>>>>>
>>>>>>>>
>>>>>> hitting
>>>>>>
>>>>>>
>>>>>>>> the log buffer- the 9 second minimum window is what is
>> getting me.
>>>>>>>>
>>>>>>>> How exactly do you define a "failure to commit?" Are
>> you actively
>>>>>>>>
>>>>>>>>
>>>>>> looking
>>>>>>
>>>>>>
>>>>>>>> for
>>>>>>>> a "failure" code from SQL, or is it the lack of a committed
>>>>>>>>
>>>>>>>>
>>>>>> transaction
>>>>>>
>>>>>>
>>>>>>>> within
>>>>>>>> the time span? I ask because if I am in the middle of DB
>>>>>>>>
>>>>>>>>
>>>>>> maintenance, I can
>>>>>>
>>>>>>
>>>>>>>> execute a transaction that might have to wait a while to get a
>>>>>>>>
>>>>>>>>
>>>>>> committal, but
>>>>>>
>>>>>>
>>>>>>>> SQL knows about the transaction pending and won't error out.
>>>>>>>>
>>>>>>>> The SQL table structure included in 2004 results in HUGE table
>>> sizes
>>>>>>>>
>>>>>>>>
>>>>>> for web
>>>>>>
>>>>>>
>>>>>>>> proxy logging alone. I'm talking gigs worth of fluff
>> per day, so
>>>>>>>>
>>>>>>>>
>>>>>> daily DB
>>>>>>
>>>>>>
>>>>>>>> maintenance is an absolute requirement. I have never had a
>>> logging
>>>>>>>> transaction "live" through a maintenance period
>> without going into
>>>>>>>>
>>>>>>>>
>>>>>> lockdown,
>>>>>>
>>>>>>
>>>>>>>> hence it being disabled on all my web proxy listener machines.
>>>>>>>>
>>>>>>>> I understand that the reasoning behind "lockdown" is for
>>>>>>>>
>>>>>>>>
>>>>>> "evidentiary"
>>>>>>
>>>>>>
>>>>>>>> reasons, but that seems counter intuitive to me. The
>> actual log
>>> is
>>>>>>>>
>>>>>>>>
>>>>>> more
>>>>>>
>>>>>>
>>>>>>>> important to me than the absence of a logged event
>> when the system
>>>>>>>>
>>>>>>>>
>>>>>> locks
>>>>>>
>>>>>>
>>>>>>>> down.
>>>>>>>> So, not only are the services not available (though a
>>> "fail-safe"),
>>>>>>>>
>>>>>>>>
>>>>>> but I
>>>>>>
>>>>>>
>>>>>>>> loose any log of any attempt in the meantime. Are we
>> going to see
>>>>>>>>
>>>>>>>>
>>>>>> some more
>>>>>>
>>>>>>
>>>>>>>> "user definable" setting in regard to logging and lockdown? If
>>> not,
>>>> I
>>>>>>>>
>>>>>>>>
>>>>>> don't
>>>>>>
>>>>>>
>>>>>>>> see how even mid-size businesses will be able to log to SQL
>>> without
>>>>>>>>
>>>>>>>>
>>>>>> disabling
>>>>>>
>>>>>>
>>>>>>>> the feature.
>>>>>>>>
>>>>>>>> t
>>>>>>>>
>>>>>>>>
>>>>>>>> On 7/4/06 5:51 AM, "Avi Sander"
>> <asander@xxxxxxxxxxxxx> spoketh to
>>>>>>>>
>>>>>>>>
>>>>>> all:
>>>>>>
>>>>>>
>>>>>>>> ISA will lockdown if either of the following occurs (whichever
>>> comes
>>>>>>>>
>>>>>>>>
>>>>>> 1st):
>>>>>>
>>>>>>
>>>>>>>> - Four consecutive failures to commit the data into the
>>> DB.
>>>>>>>>
>>>>>>>>
>>>>>> Our
>>>>>>
>>>>>>
>>>>>>>> default COMMIT timeout is 30 secs. We'll retry every 3
>> secs. So at
>>>>>>>>
>>>>>>>>
>>>>>> minimum
>>>>>>
>>>>>>
>>>>>>>> you've got 9 secs here.
>>>>>>>> - LOG buffers are exhausted\out of memory. By
>> default on a
>>>>>>>>
>>>>>>>>
>>>>>> 1GB RAM
>>>>>>
>>>>>>
>>>>>>>> machine we'll accumulate tens of thousands of recs. Is
>> there a lot
>>>> of
>>>>>>>>
>>>>>>>>
>>>>>> traffic
>>>>>>
>>>>>>
>>>>>>>> going through when this happens? How much RAM does this machine
>>>> have?
>>>>>>>>
>>>>>>>> What is the event description you see in the EventViewer?
>>>>>>>>
>>>>>>>> The Log buffers can be extended some more, though not
>> recommended
>>>>>>>>
>>>>>>>>
>>>>>> since can
>>>>>>
>>>>>>
>>>>>>>> use up a lot of memory.
>>>>>>>>
>>>>>>>> -avi
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ________________________________
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> From: Adar Greenshpon
>>>>>>>> Sent: Tuesday, July 04, 2006 2:03 PM
>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx
>>>>>>>> Cc: Avi Sander
>>>>>>>> Subject: RE: [ISAServer] Firewall database corruption
>> due to power
>>>>>>>>
>>>>>>>>
>>>>>> outtage
>>>>>>
>>>>>>
>>>>>>>> Avi - wanna play logging diva?
>>>>>>>> Generally speaking, there are four internal commit
>> retries until
>>> ISA
>>>>>>>>
>>>>>>>>
>>>>>> enters
>>>>>>
>>>>>>
>>>>>>>> lockdown. We have seen a few thousand log records in
>> the buffers
>>> in
>>>>>>>>
>>>>>>>>
>>>>>> these
>>>>>>
>>>>>>
>>>>>>>> temporary moments in our stress labs (nothing you could really
>>>>>>>>
>>>>>>>>
>>>>>> witness on a
>>>>>>
>>>>>>
>>>>>>>> normal SBS box though).
>>>>>>>>
>>>>>>>> Adar.
>>>>>>>> __________________________________________
>>>>>>>> Adar Greenshpon | Program Manager | Microsoft ISA Server
>>>>>>>> adarg@xxxxxxxxxxxxx <mailto:adarg@xxxxxxxxxxxxx>
>>>>>>>>
>>>>>>>>
>>>>>> <mailto:adarg@xxxxxxxxxxxxx>
>>>>>>
>>>>>>
>>>>>>>> | Tel: +972.4.856.1077 | SMS/Cell: +972.54.666.4579
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
>>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
>>>>>>>> Sent: Wednesday, June 28, 2006 8:26 PM
>>>>>>>> To: ISA-MVP
>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption
>> due to power
>>>>>>>>
>>>>>>>>
>>>>>> outtage
>>>>>>
>>>>>>
>>>>>>>> If that is what you guys are expecting to happen, I
>> can tell you,
>>>>>>>>
>>>>>>>>
>>>>>> that
>>>>>>
>>>>>>
>>>>>>>> *ain't* the case - not when logging to SQL on another
>> box, anyway.
>>>>>>>>
>>>>>>>>
>>>>>> There's
>>>>>>
>>>>>>
>>>>>>>> no "using as much memory as it can" going on. It
>> seems to be more
>>>>>>>>
>>>>>>>>
>>>>>> like "I
>>>>>>
>>>>>>
>>>>>>>> did not get an acknowledgement that the last transaction was
>>>> written,
>>>>>>>>
>>>>>>>>
>>>>>> and
>>>>>>
>>>>>>
>>>>>>>> it's been 2 seconds, so I'm going into lockdown."
>>>>>>>>
>>>>>>>> I'm looking forward to what info you can get from your "logging
>>>>>>>>
>>>>>>>>
>>>>>> diva."
>>>>>>
>>>>>>
>>>>>>>> t
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 6/28/06 8:43 AM, "Jim Harrison (ISA)"
>>>> <Jim.Harrison@xxxxxxxxxxxxx>
>>>>>>>> spoketh to all:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Actually, it's not a fixed value (or
>> user-changeable), other than
>>>>>>>>>
>>>>>>>>>
>>>>>> being
>>>>>>
>>>>>>
>>>>>>>>> self-regulating.
>>>>>>>>> ISA logging will use as much memory as it can until
>> it determines
>>>>>>>>>
>>>>>>>>>
>>>>>> that
>>>>>>
>>>>>>
>>>>>>>>> growth is unchecked. Thus, how quickly ISA will cry
>> foul depends
>>>> in
>>>>>>>>> part on how much free memory is available.
>>>>>>>>>
>>>>>>>>> I'll ask our logging diva for details, but this sums
>> it up pretty
>>>>>>>>>
>>>>>>>>>
>>>>>> well.
>>>>>>
>>>>>>
>>>>>>>>> Jim Harrison
>>>>>>>>> SASD (ISA SE)
>>>>>>>>> If We Can't Fix It - It Ain't Broke!
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
>>>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
>>>>>>>>> Sent: Wednesday, June 28, 2006 8:09 AM
>>>>>>>>> To: ISA-MVP
>>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption due to
>>> power
>>>>>>>>> outtage
>>>>>>>>>
>>>>>>>>> It must be about a 1k buffer, then. I repeatedly put ISA in
>>>> lockdown
>>>>>>>>> within
>>>>>>>>> seconds of maintenance jobs being run, or other "data
>> intensive"
>>>>>>>>> operations
>>>>>>>>> on the DB. I would start the job, in in a few seconds, get
>>>>>>>>>
>>>>>>>>>
>>>>>> disconnected
>>>>>>
>>>>>>
>>>>>>>>> form my TS session because of it-- It wasn't a "one
>> time" thing-
>>> it
>>>>>>>>>
>>>>>>>>>
>>>>>> was
>>>>>>
>>>>>>
>>>>>>>>> consistent and repeatable. That's the only reason I went ahead
>>> and
>>>>>>>>> disabled
>>>>>>>>> lockdown.
>>>>>>>>>
>>>>>>>>> Is the buffer user definable somewhere? What exactly
>> does your
>>>>>>>>>
>>>>>>>>>
>>>>>> script
>>>>>>
>>>>>>
>>>>>>>>> do?
>>>>>>>>>
>>>>>>>>> t
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 6/28/06 7:54 AM, "Jim Harrison (ISA)"
>>>>>>>>>
>>>>>>>>>
>>>>>> <Jim.Harrison@xxxxxxxxxxxxx>
>>>>>>
>>>>>>
>>>>>>>>> spoketh to all:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Actually, that's not true; ISA doesn't go into lockdown until
>>> the
>>>>>>>>>> logging buffer is filled.
>>>>>>>>>> I agree that a "backup logging" concept is useful.
>>>>>>>>>> I have a script that works for ISA 2000; it shouldn't be
>>> difficult
>>>>>>>>>>
>>>>>>>>>>
>>>>>> to
>>>>>>
>>>>>>
>>>>>>>>>> "agnosticize" it.
>>>>>>>>>>
>>>>>>>>>> Jim Harrison
>>>>>>>>>> Security Platform Group (ISA SE)
>>>>>>>>>>
>>>>>>>>>> If We Can't Fix It - It Ain't Broke!
>>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx]
>>>>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
>>> <mailto:thor@xxxxxxxxxxxxxxx%5d>
>>>>>>>>>> Sent: Wednesday, June 28, 2006 7:34 AM
>>>>>>>>>> To: ISA-MVP
>>>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption due to
>>> power
>>>>>>>>>> outtage
>>>>>>>>>>
>>>>>>>>>> I'd like to see some way to at least "buffer" the logging
>>> requests
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> when
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> logging to an off-box ODBC destination, like SQL.
>> ISA is quite
>>>>>>>>>> intolerant
>>>>>>>>>> of even the slightest interruption in logging. But, I guess
>>> that
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> makes
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> sense if the goal is to have "evidentiary credibility."
>>>>>>>>>>
>>>>>>>>>> It wouldn't have to be something l337 like a secondary local
>>>>>>>>>>
>>>>>>>>>>
>>>>>> logging
>>>>>>
>>>>>>
>>>>>>>>>> option
>>>>>>>>>> or anything like that-- I think being able to set a timeout
>>> value
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> would
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> would. With larger db's, a maintenance job would kick the
>>> server
>>>>>>>>>>
>>>>>>>>>>
>>>>>> into
>>>>>>
>>>>>>
>>>>>>>>>> lockdown, even though the SQL service is still running...
>>>>>>>>>>
>>>>>>>>>> Jim, I know you wanted me to get my infrastructure
>> layout to you
>>>>>>>>>>
>>>>>>>>>>
>>>>>> again
>>>>>>
>>>>>>
>>>>>>>>>> for
>>>>>>>>>> my solution to the SQL db size problems, but I haven't had a
>>>> chance
>>>>>>>>>>
>>>>>>>>>>
>>>>>> to
>>>>>>
>>>>>>
>>>>>>>>>> find
>>>>>>>>>> that old email. I should just re-create it ;)
>> Regardless, even
>>>>>>>>>>
>>>>>>>>>>
>>>>>> with
>>>>>>
>>>>>>
>>>>>>>>>> that
>>>>>>>>>> "solution" in place, I've had to disable lockdown mode when
>>>> logging
>>>>>>>>>>
>>>>>>>>>>
>>>>>> to
>>>>>>
>>>>>>
>>>>>>>>>> SQL
>>>>>>>>>> for that very reason.
>>>>>>>>>>
>>>>>>>>>> t
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 6/28/06 7:19 AM, "Thomas W Shinder" <tshinder@xxxxxxxxxxx>
>>>>>>>>>>
>>>>>>>>>>
>>>>>> spoketh
>>>>>>
>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> all:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> And it's a really good security and forensics decision.
>>>>>>>>>>>
>>>>>>>>>>> Thomas W Shinder, M.D.
>>>>>>>>>>> Site: www.isaserver.org
>>>>>>>>>>> Blog: http://blogs.isaserver.org/shinder/
>>>>>>>>>>> Book: http://tinyurl.com/3xqb7
>>>>>>>>>>> MVP -- ISA Firewalls
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Amy Babinchak [mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx]
>>>>>>>>>>>> <mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx%5d>
>>>>>>>>>>>> <mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx%5d>
>>>>>>>>>>>> Sent: Wednesday, June 28, 2006 6:28 AM
>>>>>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx
>>>>>>>>>>>> Subject: RE: [ISAServer] Firewall database
>> corruption due to
>>>>>>>>>>>> power outtage
>>>>>>>>>>>>
>>>>>>>>>>>> I like that ISA shuts down if it can't log. It's
>> an indicator
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>> that
>>>>>>
>>>>>>
>>>>>>>>>>>> something has gone wrong.
>>>>>>>>>>>>
>>>>>>>>>>>> Amy
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>> From: Susan Bradley, CPA aka Ebitz - SBS Rocks [MVP]
>>>>>>>>>>>> [mailto:sbradcpa@xxxxxxxxxxx]
>> <mailto:sbradcpa@xxxxxxxxxxx%5d>
>>>>>>>>>>>> <mailto:sbradcpa@xxxxxxxxxxx%5d>
>>>>>>>>>>>> Sent: Wednesday, June 28, 2006 12:12 AM
>>>>>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx
>>>>>>>>>>>> Subject: [ISAServer] Firewall database corruption
>> due to power
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> outtage
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> With the caveat that we all know we need a good
>> UPS but like
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>> stuff
>>>>>>
>>>>>>
>>>>>>>>>>>> happens.. I've seen two folks recently have issues
>> getting a
>>>>>>>>>>>> server back
>>>>>>>>>>>>
>>>>>>>>>>>> up have a 'dirty shutdown' due to power outtage
>> that ended up
>>>>>>>>>>>> corrupting
>>>>>>>>>>>>
>>>>>>>>>>>> the ISA firewall database.
>>>>>>>>>>>>
>>>>>>>>>>>> Short of uninstalling and reinstalling... short of
>> adjusting
>>>>>>>>>>>> ISA so it
>>>>>>>>>>>> won't shut down if it can't log... short of a better UPS...
>>> any
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> other
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> recommendations on handling this better?
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> To subscribe to the list - send an email to
>>> list@xxxxxxxxxxxxxxx
>>>>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>>>> youremailaddress
>>>>>>>>>>>>
>>>>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>>>> youremailaddress
>>>>>>>>>>>>
>>>>>>>>>>>> Don't forget the comma!
>>>>>>>>>>>> ---
>>>>>>>>>>>> To subscribe to the list - send an email to
>>> list@xxxxxxxxxxxxxxx
>>>>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>>>> youremailaddress
>>>>>>>>>>>>
>>>>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>>>> youremailaddress
>>>>>>>>>>>>
>>>>>>>>>>>> Don't forget the comma!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> To subscribe to the list - send an email to
>>> list@xxxxxxxxxxxxxxx
>>>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> youremailaddress
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> youremailaddress
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Don't forget the comma!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> To subscribe to the list - send an email to
>> list@xxxxxxxxxxxxxxx
>>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>> youremailaddress
>>>>>>>>>>
>>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>> youremailaddress
>>>>>>>>>>
>>>>>>>>>> Don't forget the comma!
>>>>>>>>>> ---
>>>>>>>>>> To subscribe to the list - send an email to
>> list@xxxxxxxxxxxxxxx
>>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> youremailaddress
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> youremailaddress
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Don't forget the comma!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> To subscribe to the list - send an email to
>> list@xxxxxxxxxxxxxxx
>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>> youremailaddress
>>>>>>>>>
>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>> youremailaddress
>>>>>>>>>
>>>>>>>>> Don't forget the comma!
>>>>>>>>> ---
>>>>>>>>> To subscribe to the list - send an email to
>> list@xxxxxxxxxxxxxxx
>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>
>>>>>>>>>
>>>>>> youremailaddress
>>>>>>
>>>>>>
>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>>
>>>>>>>>>
>>>>>> youremailaddress
>>>>>>
>>>>>>
>>>>>>>>> Don't forget the comma!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ---
>>>>>>>> To subscribe to the list - send an email to
>> list@xxxxxxxxxxxxxxx
>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>
>>>>>>>>
>>>>>> youremailaddress
>>>>>>
>>>>>>
>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>
>>>>>>>>
>>>>>> youremailaddress
>>>>>>
>>>>>>
>>>>>>>> Don't forget the comma!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> To subscribe to the list - send an email to
>> list@xxxxxxxxxxxxxxx
>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>
>>>>>>>>
>>>>>> youremailaddress
>>>>>>
>>>>>>
>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>>>>>>
>>>>>>>>
>>>>>> youremailaddress
>>>>>>
>>>>>>
>>>>>>>> Don't forget the comma!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx
>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>> youremailaddress
>>>>>
>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>> youremailaddress
>>>>>
>>>>> Don't forget the comma!
>>>>>
>>>>>
>>>>>
>>>> ---
>>>> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx
>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>>> youremailaddress
>>>>
>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>>> youremailaddress
>>>>
>>>> Don't forget the comma!
>>>> ---
>>>> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx
>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>>> youremailaddress
>>>>
>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>>> youremailaddress
>>>>
>>>> Don't forget the comma!
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>> ---
>> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx
>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx,
>> youremailaddress
>>
>> To leave the list - send an email to list@xxxxxxxxxxxxxxx
>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx,
>> youremailaddress
>>
>> Don't forget the comma!
>>
>>
>
>
>
Other related posts:
