Like unto thusly: http://www.microsoft.com/technet/prodtechnol/isa/2004/plan/disablelockdownonlogfailure.mspx ------------------------------------------------------- Jim Harrison MCP(NT4, W2K), A+, Network+, PCG http://isaserver.org/Jim_Harrison/ http://isatools.org Read the help / books / articles! ------------------------------------------------------- -----Original Message----- From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On Behalf Of Thomas W Shinder Sent: Monday, July 10, 2006 08:10 To: isapros@xxxxxxxxxxxxx Subject: [isapros] Re: [ISAServer] Firewall database corruption due to power outtage Log Failure Alert. Thomas W Shinder, M.D. Site: www.isaserver.org Blog: http://blogs.isaserver.org/shinder/ Book: http://tinyurl.com/3xqb7 MVP -- ISA Firewalls > -----Original Message----- > From: isapros-bounce@xxxxxxxxxxxxx > [mailto:isapros-bounce@xxxxxxxxxxxxx] On Behalf Of Thor (Hammer of > God) > Sent: Monday, July 10, 2006 9:24 AM > To: isapros@xxxxxxxxxxxxx > Subject: [isapros] Re: [ISAServer] Firewall database corruption due to > power outtage > > What do you mean "the alert?" What alert? > > t > > > On 7/10/06 7:21 AM, "Thomas W Shinder" <tshinder@xxxxxxxxxxx> spoketh > to > all: > > > Hi Tim, > > > > But you can do that by configuring the Alert to not send > the system into > > lockdown. > > > > Thomas W Shinder, M.D. > > Site: www.isaserver.org > > Blog: http://blogs.isaserver.org/shinder/ > > Book: http://tinyurl.com/3xqb7 > > MVP -- ISA Firewalls > > > > > > > >> -----Original Message----- > >> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx] > >> Sent: Monday, July 10, 2006 9:16 AM > >> To: Adar Greenshpon; ISA-MVP; Avi Sander; Nathan Bigman > >> Subject: Re: [ISAServer] Firewall database corruption due to power > >> outtage > >> > >> I guess I misunderstood. The "mentioned below" items kept going > >> back to how to remove the fluff, rather than dealing with why ISA > >> goes into lockdown so readily. I've still not seen any information > >> on how/when/why ISA goes into lockdown when logging to ISA. You > >> had indicated that the other information was incorrect, so I was > >> standing by for that info. > >> > >> It doesn't matter if you put it in a different database. When the > >> optimization plan (you know, the "normal" plan that > rebuilds indexes, > >> insures integrity, etc) runs, ISA goes into lockdown. Even if I > >> waited a week in between times that I purged the data, I'll still > >> have to perform some maintenance, and the system would still go > >> into lockdown. > >> > >> The solution is to disable lockdown. > >> > >> Thanks! > >> > >> t > >> > >> > >> On 7/10/06 12:13 AM, "Adar Greenshpon" > <Adar.Greenshpon@xxxxxxxxxxxxx> > >> spoketh to all: > >> > >>> Thor: as I've indicated below, we plan on improving the > connectivity > >>> issue you raised. Given the current SQL logging > limitation and your > >>> deployment, might it be possible to split the webproxy > log into two > >>> databases: one for historical purposes and one for current > >> events. The > >>> heavy-weight maintenance could be done on the former one > >> while the later > >>> one would be available for ISA to dump logs into. > >>> > >>> > >>> > >>> -----Original Message----- > >>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx] > >>> Sent: Monday, July 10, 2006 1:50 AM > >>> To: ISA-MVP; Avi Sander; Adar Greenshpon; Nathan Bigman > >>> Subject: Re: [ISAServer] Firewall database corruption due to power > >>> outtage > >>> > >>> > >>> Yes, Jim... Thank you. Sorry that I got so short, but I > was getting > >>> quite > >>> frustrated trying to make the same point over and over again. > >>> > >>> The focus need not be on "how to trim the logs." That's > >> not the *real* > >>> issue. The focus needs to be: "Regarding lockdown, what > >> factors are in > >>> play > >>> when ISA logs to SQL, and under what circumstances does ISA invoke > >>> lockdown when posting to a SQL DB?" > >>> > >>> Let's put the padding issues *totally* out of the picture. > >> Let's say > >>> that > >>> I've got TB's of storage and that I could not care less > >> about how much > >>> fluff > >>> is in my WebProxyLog-- let it get as big as it wants. > >> However, I must > >>> back > >>> it up. I've also experienced ISA going into lockdown > >> during classic db > >>> maintenance like backing up, re-indexing, optimizing size, > >> etc. These > >>> processes must be run regardless of the presence of > >> "padding issues" or > >>> not. > >>> In fact, even if the padding problem is solved in ISA2006, in > >>> combination with or exclusive of contributing options in SQL 2005 > >>> (which I'm migrating to as I type), ISA will still go into > >>> lockdown when the SQL server engages in processes which may lock > >>> the db, or delay response in some way. > >>> > >>> Again- (and hopefully someone will listen this time) the > >> concern speaks > >>> to > >>> the "real world" usability of "lockdown mode" within the > >> enterprise. If > >>> Microsoft intends to tout the lockdown feature of ISA as a tool > >>> enterprise clients can use to ensure evidentiary integrity, then > >>> you > >> must address > >>> the > >>> mode's "survivability" during standard, everyday, > >> real-world processes > >>> that > >>> must be run on the SQL servers the ISA box is logging to, > or it will > >>> simply > >>> not be used. This relates to processes that may or may not have > >>> anything to do with the log files themselves. > >>> > >>> If nobody cares if lockdown is utilized in the > enterprise, then this > >>> thread > >>> is done. If, however, the goal is supply enterprise > customers with a > >>> lockdown mode they can use, then user-defined connection > >> parameters must > >>> be > >>> provided in order to customize the lockdown threshold to our > >>> environments. > >>> > >>> T > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On 7/9/06 9:28 AM, "Jim Harrison (ISA)" > <Jim.Harrison@xxxxxxxxxxxxx> > >>> spoketh > >>> to all: > >>> > >>>> It may help in the short run, but the issue Tim raises > >> isn't so much > >>>> "how much data is being logged", but "how long ISA will > >> wait while the > >>>> logging process fails to respond before it raises an > alert" and "I > >>> wanna > >>>> set this trigger point". > >>>> > >>>> What he's asking for (as have more than a few customers) > >> is some way > >>> for > >>>> the user to specify a "almost choking on my own data" > >> alert limit that > >>>> could be used to move ISA logging to an alternate > destination (SQL, > >>>> file, bitbucket) until the primary logger responds once > >> more. What we > >>>> have now is a "choked on my own data" alert, which is a > >> bit too late. > >>>> > >>>> In fact, I've seen one unfavorable comparison to > <product deleted> > >>> where > >>>> they actually: > >>>> 1. fail over to a backup logging process 2. monitor the primary > >>>> logger 3. fail back when the primary logger reappears 4. import > >>>> the alternate-store log data into the primary > store as a > >>>> low-pri part of the fail-back. > >>>> > >>>> Techincally, this is scriptable, except for the fact > that our alert > >>>> happens too late (and isn't "tweakable"). If one were willing to > >>>> tolerate the momentary traffic block that's created today, > >> a script is > >>> a > >>>> workable solution (except for the "pre-panic" factor). > >>>> > >>>> Jim Harrison > >>>> SASD (ISA SE) > >>>> If We Can't Fix It - It Ain't Broke! > >>>> > >>>> -----Original Message----- > >>>> From: Susan Bradley, CPA aka Ebitz - SBS Rocks [MVP] > >>>> [mailto:sbradcpa@xxxxxxxxxxx] > >>>> Sent: Sunday, July 09, 2006 2:22 AM > >>>> To: isaserver@xxxxxxxxxxxxxxx > >>>> Cc: Avi Sander; Adar Greenshpon; Nathan Bigman > >>>> Subject: Re: [ISAServer] Firewall database corruption > due to power > >>>> outtage > >>>> > >>>> Shouldn't depadding not cause the system to freak out and go into > >>>> lockdown? That's his real point.. not necessarily the > >> fact he has to > >>> do > >>>> > >>>> it..but the fact that ISA is there locking itself down in > >> the process > >>>> .... > >>>> > >>>> Thor (Hammer of God) wrote: > >>>> > >>>>> Never mind. Not sure why I bother... > >>>>> > >>>>> > >>>>> t > >>>>> > >>>>> On 7/9/06 12:01 AM, "Avi Sander" <asander@xxxxxxxxxxxxx> > >> spoketh to > >>>> all: > >>>>> > >>>>> > >>>>> > >>>>>> Hi Thor - > >>>>>> > >>>>>> Please see > >>>>>> > >>> > >> > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/tsqlr > >>>> ef > >>>>>> /ts_set-set_2uw7.asp . > >>>>>> > >>>>>> It describes how one can toggle the ANSI_PADDING > option. This can > >>>>>> control how fields are padded\trimmed in SQL, but should be set > >>> prior > >>>> to > >>>>>> tbl creation. > >>>>>> > >>>>>> - avi > >>>>>> -----Original Message----- > >>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx] > >>>>>> Sent: Friday, July 07, 2006 6:23 PM > >>>>>> To: Avi Sander; Adar Greenshpon; ISA-MVP > >>>>>> Cc: Nathan Bigman > >>>>>> Subject: Re: [ISAServer] Firewall database corruption > >> due to power > >>>>>> outtage > >>>>>> > >>>>>> I think we're getting off track a little here... > >>>>>> > >>>>>> I appreciate the link to that "fix," but I've *already > >> solved* the > >>>>>> padding > >>>>>> issue. I've been including a script for that in my Blackhat > >>> trainings > >>>>>> for > >>>>>> years. To that effect, simply trimming the data in the default > >>> table > >>>>>> isn't > >>>>>> the best way to go, IMO. The data structure is not very > >> efficient > >>> to > >>>>>> begin > >>>>>> with (for my needs). I've created my own (better designed) > >>> structure, > >>>>>> and > >>>>>> trim the data as I post into that table (and there is no > >> "trim data" > >>>> DSN > >>>>>> option that I know of). But that's OK-- as an administrator, I > >>>> *expect* > >>>>>> that I will have to adjust logging processes to better > >> fit my needs > >>>>>> (just > >>>>>> like I do with IIS logs that I post to SQL). You guys > provide the > >>>>>> mechanism, > >>>>>> and I customize it to suit me. Like I said, my yearly > >> retained data > >>> to > >>>>>> date > >>>>>> is only 10 gig. No problems. > >>>>>> > >>>>>> What I'm saying is that *when I run the process to > trim the data* > >>> ISA > >>>>>> goes > >>>>>> into "lockdown." This whole thing got started when we > >> were talking > >>>>>> about > >>>>>> lockdown mode, and how "trigger happy" it is when one is > >> logging to > >>>> SQL. > >>>>>> > >>>>>> I only brought it up in an effort to identify this: > >>>>>> If a customer logs to SQL, there is a padding problem. > >> The padding > >>>>>> problem > >>>>>> *requires* DB maintenance processes to control. When the DB > >>>> maintenance > >>>>>> is > >>>>>> executed against any DB of consequence (just several thousand > >>> records > >>>>>> seems > >>>>>> to be enough), ISA goes into lockdown, presumably > because of the > >>> "lag" > >>>>>> created by server load when the SQL server goes about > parsing out > >>> gigs > >>>>>> of > >>>>>> fluff data. This is true even on my production SQL > >> cluster (which > >>> is > >>>>>> kickass, mind you ;) so it's not a "hardware" problem. > >> Thus, I was > >>>>>> forced > >>>>>> to disable lockdown. > >>>>>> > >>>>>> Therefore, IF a customer is logging to SQL, AND they want use > >>>> "lockdown" > >>>>>> THEN better user-definable timeout parameters need to be > >> available. > >>>>>> That's > >>>>>> all I was saying. > >>>>>> > >>>>>> Of course, since you have identified that ISA2004 EE, > ISA2006 SE, > >>> and > >>>>>> ISA2006 EE properly trim data before posting it to SQL, > >> then it all > >>>>>> seems to > >>>>>> be a moot point, right? ;) > >>>>>> > >>>>>> The main box posting web access is currently ISA 2004 > SE. But I > >>> think > >>>>>> I've > >>>>>> got a copy of ISA 2004 EE that I picked up in Hong Kong > >> along with > >>> an > >>>>>> old > >>>>>> Rush CD that I'll throw on that guy and see what > >> happens. (jk;) I'll > >>>> let > >>>>>> you > >>>>>> guys know what happens. > >>>>>> > >>>>>> Thx > >>>>>> t > >>>>>> > >>>>>> > >>>>>> > >>>>>> On 7/7/06 12:29 AM, "Avi Sander" <asander@xxxxxxxxxxxxx> > >> spoketh to > >>>> all: > >>>>>> > >>>>>> > >>>>>> > >>>>>>> there's also an option on the ODBC DSN : 'trim fields' that > >>> partially > >>>>>>> > >>>>>>> > >>>>>> helps, > >>>>>> > >>>>>> > >>>>>>> if i recall correctly. > >>>>>>> I'll look into this further > >>>>>>> > >>>>>>> ________________________________ > >>>>>>> > >>>>>>> From: Adar Greenshpon > >>>>>>> Sent: Fri 07/07/2006 09:07 > >>>>>>> To: Avi Sander; Thor (Hammer of God); > isaserver@xxxxxxxxxxxxxxx > >>>>>>> Cc: Nathan Bigman > >>>>>>> Subject: RE: [ISAServer] Firewall database corruption > >> due to power > >>>>>>> > >>>>>>> > >>>>>> outtage > >>>>>> > >>>>>> > >>>>>>> > >>>>>>> Thor, > >>>>>>> > >>>>>>> Per the excessive padding, see this thread: > >>>>>>> http://forums.isaserver.org/m_140009200/tm.htm - I > think that's > >>> what > >>>>>>> > >>>>>>> > >>>>>> you're > >>>>>> > >>>>>> > >>>>>>> looking for as we already heard this a few times (as > Avi pointed > >>> out, > >>>>>>> > >>>>>>> > >>>>>> the > >>>>>> > >>>>>> > >>>>>>> problem does not exist in ISA 2004EE or ISA 2006 > >> versions). Nathan > >>> - > >>>>>>> > >>>>>>> > >>>>>> will an > >>>>>> > >>>>>> > >>>>>>> official KB help here? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Adar. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ________________________________ > >>>>>>> > >>>>>>> From: Avi Sander > >>>>>>> Sent: Friday, July 07, 2006 12:27 AM > >>>>>>> To: Thor (Hammer of God) > >>>>>>> Cc: Adar Greenshpon > >>>>>>> Subject: RE: [ISAServer] Firewall database corruption > >> due to power > >>>>>>> > >>>>>>> > >>>>>> outtage > >>>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> OK - that explains it. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> This is a known SQL ODBC interfacing issue. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ISA 2004 EE and ISA2006 SE & EE moved away from ODBC > >> and use oledb > >>>>>>> > >>>>>>> > >>>>>> interfaces > >>>>>> > >>>>>> > >>>>>>> for SQL server logging instead. This issue does not > exist there > >>>>>>> > >>>>>>> > >>>>>> anymore as it > >>>>>> > >>>>>> > >>>>>>> is all trimmed by design (and throughput is also greatly > >>> increased). > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> The 2 potential causes of errors i mentioned earlier in > >> the thread > >>>>>>> > >>>>>>> > >>>>>> (buffers > >>>>>> > >>>>>> > >>>>>>> full, 4 retries) are only relevant to the OLEDB > >> logging, and not to > >>>>>>> > >>>>>>> > >>>>>> your ISA - > >>>>>> > >>>>>> > >>>>>>> my bad. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> let me check with the ISA2004 SE code and get back to > you with a > >>>>>>> > >>>>>>> > >>>>>> better > >>>>>> > >>>>>> > >>>>>>> picture... > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -avi > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ________________________________ > >>>>>>> > >>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx] > >>>>>>> Sent: Thu 06/07/2006 23:12 > >>>>>>> To: Avi Sander > >>>>>>> Subject: Re: [ISAServer] Firewall database corruption > >> due to power > >>>>>>> > >>>>>>> > >>>>>> outtage > >>>>>> > >>>>>> > >>>>>>> These are ISA2004 SE boxes. I haven't tried with > >> ISA2006 yet, but > >>>> I'm > >>>>>>> running it here at The Yeti and can check it out (unless you > >>> already > >>>>>>> know...) > >>>>>>> > >>>>>>> t > >>>>>>> > >>>>>>> > >>>>>>> On 7/6/06 2:08 PM, "Avi Sander" <asander@xxxxxxxxxxxxx> > >> spoketh to > >>>>>>> > >>>>>>> > >>>>>> all: > >>>>>> > >>>>>> > >>>>>>>> Hey Thor - > >>>>>>>> > >>>>>>>> What ISA version are you using? and whichSKU: SE or EE? > >>>>>>>> > >>>>>>>> -avi > >>>>>>>> > >>>>>>>> ________________________________ > >>>>>>>> > >>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx] > >>>>>>>> Sent: Thu 06/07/2006 18:39 > >>>>>>>> To: Adar Greenshpon; ISA-MVP; Avi Sander > >>>>>>>> Subject: Re: [ISAServer] Firewall database corruption > >> due to power > >>>>>>>> > >>>>>>>> > >>>>>> outtage > >>>>>> > >>>>>> > >>>>>>>> Hi Adar- > >>>>>>>> > >>>>>>>> Yes, logs going to an external SQL 2000 cluster via > >> direct gigabit > >>>>>>>> > >>>>>>>> > >>>>>> switched > >>>>>> > >>>>>> > >>>>>>>> link. And log retention is permanent. Though I've > >> only got 120 > >>>>>>>> > >>>>>>>> > >>>>>> employees, > >>>>>> > >>>>>> > >>>>>>>> we > >>>>>>>> have tight policies on Internet usage and abuse and I, > >> like many, > >>>>>>>> > >>>>>>>> > >>>>>> must keep > >>>>>> > >>>>>> > >>>>>>>> access logs to ensure these policies are being > >> honored, as well as > >>>>>>>> > >>>>>>>> > >>>>>> for any > >>>>>> > >>>>>> > >>>>>>>> number of potential legal issues that might arise. I > >> trust that > >>> you > >>>>>>>> > >>>>>>>> > >>>>>> are not > >>>>>> > >>>>>> > >>>>>>>> suggesting that ISA log retention via SQL should be > measured in > >>>> days, > >>>>>>>> > >>>>>>>> > >>>>>> right? > >>>>>> > >>>>>> > >>>>>>>> ;) > >>>>>>>> > >>>>>>>> Let me go into more detail about the "fluff" I > mentioned- it is > >>> not > >>>>>>>> > >>>>>>>> > >>>>>> so much > >>>>>> > >>>>>> > >>>>>>>> the "gigs of logs" as it would relate to real data- it > >> is actual > >>>>>>>> > >>>>>>>> > >>>>>> "fluff:" > >>>>>> > >>>>>> > >>>>>>>> Extra, padded spacing in the fields posted to SQL from ISA. > >>> Though > >>>>>>>> > >>>>>>>> > >>>>>> you guys > >>>>>> > >>>>>> > >>>>>>>> have defined the fields in SQL as nvarchar data, there > >> is some bad > >>>>>>>> > >>>>>>>> > >>>>>> mojo in > >>>>>> > >>>>>> > >>>>>>>> the > >>>>>>>> logging process. Cursory review of the data in > >> something like SQL > >>>>>>>> > >>>>>>>> > >>>>>> Query > >>>>>> > >>>>>> > >>>>>>>> Analyzer may not immediately reveal this, but binary > analyze of > >>> the > >>>>>>>> > >>>>>>>> > >>>>>> data > >>>>>> > >>>>>> > >>>>>>>> files > >>>>>>>> do. > >>>>>>>> > >>>>>>>> I've created some example text files to simplify what > >> I am talking > >>>>>>>> > >>>>>>>> > >>>>>> about > >>>>>> > >>>>>> > >>>>>>>> (Attached as SQL1.txt and SQL2.txt). > >>>>>>>> > >>>>>>>> Consider the following simple query to retrieve a > single row of > >>>> basic > >>>>>>>> > >>>>>>>> > >>>>>> data > >>>>>> > >>>>>> > >>>>>>>> from the WebProxyLog as stored by ISA: > >>>>>>>> > >>>>>>>> "select > >>>>>>>> > >>>>>>>> > >>>>>> > >> > ClientUserName,ClientAgent,DestHost,DestHostIP,SrcNetwork,DstNetwork > >>>>>> > >>>>>> > >>>>>>>> from WebProxyLog where LogID=10133541" > >>>>>>>> > >>>>>>>> (SQL1.txt contains the results, but I'll paste them > >> here as well) > >>>>>>>> <begin paste> > >>>>>>>> ANCHORSIGN\yomomma > >>>>>>>> PGP > >>>>>>>> update.pgp.com > >>>>>>>> 63.251.255.18 VPN Clients > >>>>>>>> External > >>>>>>>> > >>>>>>>> </end paste> > >>>>>>>> > >>>>>>>> Notice all the extra spacing; again, even though the > fields are > >>> type > >>>>>>>> nvarchar-- ISA is padding the data. Now, let's trim > >> it up and see > >>>>>>>> > >>>>>>>> > >>>>>> what > >>>>>> > >>>>>> > >>>>>>>> happens in this query: > >>>>>>>> > >>>>>>>> "select > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> > >>> > >> > rtrim(ClientUserName),rtrim(ClientAgent),rtrim(DestHost),rtrim(DestHos > >>>> tI > >>>>>> P),rt>> > >>>>>> r > >>>>>> > >>>>>> > >>>>>>>> im(SrcNetwork),rtrim(DstNetwork) from WebProxyLog where > >>>>>>>> > >>>>>>>> > >>>>>> LogID=10133541" > >>>>>> > >>>>>> > >>>>>>>> (SQL2.txt contains the results, but again I'll paste them) > >>>>>>>> <begin paste> > >>>>>>>> ANCHORSIGN\yomomma PGP update.pgp.com 63.251.255.18 > >>> VPN > >>>>>>>> > >>>>>>>> > >>>>>> Clients > >>>>>> > >>>>>> > >>>>>>>> External > >>>>>>>> </end paste> > >>>>>>>> > >>>>>>>> The original query results are 934 bytes. The trimmed query > >>> results > >>>>>>>> > >>>>>>>> > >>>>>> are 74 > >>>>>> > >>>>>> > >>>>>>>> bytes. Just for this simple query, with selected > >> fields from one > >>>>>>>> > >>>>>>>> > >>>>>> record, the > >>>>>> > >>>>>> > >>>>>>>> storage requirements for this data is 12.62 times > >> greater due to > >>> the > >>>>>>>> > >>>>>>>> > >>>>>> "fluff." > >>>>>> > >>>>>> > >>>>>>>> Extend that out to a full logged record, and then > multiply that > >>>> times > >>>>>>>> > >>>>>>>> > >>>>>> tens of > >>>>>> > >>>>>> > >>>>>>>> thousands. *THIS* is the 1-2 gigs of "fluff" I'm > >> talking about on > >>> a > >>>>>>>> > >>>>>>>> > >>>>>> *daily* > >>>>>> > >>>>>> > >>>>>>>> basis. > >>>>>>>> > >>>>>>>> The "required maintenance" I'm talking about is where > >> I take daily > >>>>>>>> > >>>>>>>> > >>>>>> raw data > >>>>>> > >>>>>> > >>>>>>>> from the WebProxyLog, trim it up, and then post it > into my own > >>>>>>>> > >>>>>>>> > >>>>>> tables. In my > >>>>>> > >>>>>> > >>>>>>>> case, the _yearly_ web proxy log data for my users is > >> only 10 gig. > >>>>>>>> > >>>>>>>> > >>>>>> _Weekly_ > >>>>>> > >>>>>> > >>>>>>>> "raw" data in the original ISA log format averages 12 > >> gig. When > >>> I > >>>>>>>> > >>>>>>>> > >>>>>> run this > >>>>>> > >>>>>> > >>>>>>>> process, I am able to retrieve data via SQA, Access > >> front end, or > >>>> via > >>>>>>>> > >>>>>>>> > >>>>>> ADODB > >>>>>> > >>>>>> > >>>>>>>> objects, yet ISA will go into lockdown mode while > logging when > >>> this > >>>>>>>> > >>>>>>>> > >>>>>> job runs > >>>>>> > >>>>>> > >>>>>>>> at night. That's why I have to turn it off. > >>>>>>>> > >>>>>>>> I hope that better defines the issue as I see it. > >>>>>>>> > >>>>>>>> Thanksl > >>>>>>>> T > >>>>>>>> > >>>>>>>> -- > >>>>>>>> "Tom Shinder pities Mr. T" > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On 7/5/06 2:01 AM, "Adar Greenshpon" > >>> <Adar.Greenshpon@xxxxxxxxxxxxx> > >>>>>>>> > >>>>>>>> > >>>>>> spoketh > >>>>>> > >>>>>> > >>>>>>>> to all: > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Hi Thor, > >>>>>>>> > >>>>>>>> Just so we'll be on the same page: are you directing > >> your ISA logs > >>>> to > >>>>>>>> > >>>>>>>> > >>>>>> an > >>>>>> > >>>>>> > >>>>>>>> external machine with SQL Server (SQL 2000 or 2005)?. > >> If so, how > >>> do > >>>>>>>> > >>>>>>>> > >>>>>> you do > >>>>>> > >>>>>> > >>>>>>>> the maintenance? What's your desired log retention > >> policy (e.g. 7 > >>>>>>>> > >>>>>>>> > >>>>>> days)? > >>>>>> > >>>>>> > >>>>>>>> Per the gigs of logs you're doing, have you considered > >> reducing it > >>>> by > >>>>>>>> > >>>>>>>> > >>>>>> not > >>>>>> > >>>>>> > >>>>>>>> logging images http requests? > >>>>>>>> Adar. > >>>>>>>> __________________________________________ > >>>>>>>> Adar Greenshpon | Program Manager | Microsoft ISA Server > >>>>>>>> adarg@xxxxxxxxxxxxx <mailto:adarg@xxxxxxxxxxxxx> > >>>>>>>> > >>>>>>>> > >>>>>> <mailto:adarg@xxxxxxxxxxxxx> > >>>>>> > >>>>>> > >>>>>>>> | Tel: +972.4.856.1077 | SMS/Cell: +972.54.666.4579 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> ________________________________ > >>>>>>>> > >>>>>>>> > >>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx] > >>>>>>>> Sent: Tuesday, July 04, 2006 8:55 PM > >>>>>>>> To: Avi Sander > >>>>>>>> Cc: Adar Greenshpon; ISA-MVP > >>>>>>>> Subject: Re: [ISAServer] Firewall database corruption > >> due to power > >>>>>>>> > >>>>>>>> > >>>>>> outtage > >>>>>> > >>>>>> > >>>>>>>> OK- the either/or situation makes more sense. I'm > >> sure that I'm > >>>> not > >>>>>>>> > >>>>>>>> > >>>>>> hitting > >>>>>> > >>>>>> > >>>>>>>> the log buffer- the 9 second minimum window is what is > >> getting me. > >>>>>>>> > >>>>>>>> How exactly do you define a "failure to commit?" Are > >> you actively > >>>>>>>> > >>>>>>>> > >>>>>> looking > >>>>>> > >>>>>> > >>>>>>>> for > >>>>>>>> a "failure" code from SQL, or is it the lack of a committed > >>>>>>>> > >>>>>>>> > >>>>>> transaction > >>>>>> > >>>>>> > >>>>>>>> within > >>>>>>>> the time span? I ask because if I am in the middle of DB > >>>>>>>> > >>>>>>>> > >>>>>> maintenance, I can > >>>>>> > >>>>>> > >>>>>>>> execute a transaction that might have to wait a > while to get a > >>>>>>>> > >>>>>>>> > >>>>>> committal, but > >>>>>> > >>>>>> > >>>>>>>> SQL knows about the transaction pending and won't error out. > >>>>>>>> > >>>>>>>> The SQL table structure included in 2004 results in > HUGE table > >>> sizes > >>>>>>>> > >>>>>>>> > >>>>>> for web > >>>>>> > >>>>>> > >>>>>>>> proxy logging alone. I'm talking gigs worth of fluff > >> per day, so > >>>>>>>> > >>>>>>>> > >>>>>> daily DB > >>>>>> > >>>>>> > >>>>>>>> maintenance is an absolute requirement. I have never had a > >>> logging > >>>>>>>> transaction "live" through a maintenance period > >> without going into > >>>>>>>> > >>>>>>>> > >>>>>> lockdown, > >>>>>> > >>>>>> > >>>>>>>> hence it being disabled on all my web proxy listener > machines. > >>>>>>>> > >>>>>>>> I understand that the reasoning behind "lockdown" is for > >>>>>>>> > >>>>>>>> > >>>>>> "evidentiary" > >>>>>> > >>>>>> > >>>>>>>> reasons, but that seems counter intuitive to me. The > >> actual log > >>> is > >>>>>>>> > >>>>>>>> > >>>>>> more > >>>>>> > >>>>>> > >>>>>>>> important to me than the absence of a logged event > >> when the system > >>>>>>>> > >>>>>>>> > >>>>>> locks > >>>>>> > >>>>>> > >>>>>>>> down. > >>>>>>>> So, not only are the services not available (though a > >>> "fail-safe"), > >>>>>>>> > >>>>>>>> > >>>>>> but I > >>>>>> > >>>>>> > >>>>>>>> loose any log of any attempt in the meantime. Are we > >> going to see > >>>>>>>> > >>>>>>>> > >>>>>> some more > >>>>>> > >>>>>> > >>>>>>>> "user definable" setting in regard to logging and > lockdown? If > >>> not, > >>>> I > >>>>>>>> > >>>>>>>> > >>>>>> don't > >>>>>> > >>>>>> > >>>>>>>> see how even mid-size businesses will be able to log to SQL > >>> without > >>>>>>>> > >>>>>>>> > >>>>>> disabling > >>>>>> > >>>>>> > >>>>>>>> the feature. > >>>>>>>> > >>>>>>>> t > >>>>>>>> > >>>>>>>> > >>>>>>>> On 7/4/06 5:51 AM, "Avi Sander" > >> <asander@xxxxxxxxxxxxx> spoketh to > >>>>>>>> > >>>>>>>> > >>>>>> all: > >>>>>> > >>>>>> > >>>>>>>> ISA will lockdown if either of the following occurs > (whichever > >>> comes > >>>>>>>> > >>>>>>>> > >>>>>> 1st): > >>>>>> > >>>>>> > >>>>>>>> - Four consecutive failures to commit the > data into the > >>> DB. > >>>>>>>> > >>>>>>>> > >>>>>> Our > >>>>>> > >>>>>> > >>>>>>>> default COMMIT timeout is 30 secs. We'll retry every 3 > >> secs. So at > >>>>>>>> > >>>>>>>> > >>>>>> minimum > >>>>>> > >>>>>> > >>>>>>>> you've got 9 secs here. > >>>>>>>> - LOG buffers are exhausted\out of memory. By > >> default on a > >>>>>>>> > >>>>>>>> > >>>>>> 1GB RAM > >>>>>> > >>>>>> > >>>>>>>> machine we'll accumulate tens of thousands of recs. Is > >> there a lot > >>>> of > >>>>>>>> > >>>>>>>> > >>>>>> traffic > >>>>>> > >>>>>> > >>>>>>>> going through when this happens? How much RAM does > this machine > >>>> have? > >>>>>>>> > >>>>>>>> What is the event description you see in the EventViewer? > >>>>>>>> > >>>>>>>> The Log buffers can be extended some more, though not > >> recommended > >>>>>>>> > >>>>>>>> > >>>>>> since can > >>>>>> > >>>>>> > >>>>>>>> use up a lot of memory. > >>>>>>>> > >>>>>>>> -avi > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> ________________________________ > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> From: Adar Greenshpon > >>>>>>>> Sent: Tuesday, July 04, 2006 2:03 PM > >>>>>>>> To: isaserver@xxxxxxxxxxxxxxx > >>>>>>>> Cc: Avi Sander > >>>>>>>> Subject: RE: [ISAServer] Firewall database corruption > >> due to power > >>>>>>>> > >>>>>>>> > >>>>>> outtage > >>>>>> > >>>>>> > >>>>>>>> Avi - wanna play logging diva? > >>>>>>>> Generally speaking, there are four internal commit > >> retries until > >>> ISA > >>>>>>>> > >>>>>>>> > >>>>>> enters > >>>>>> > >>>>>> > >>>>>>>> lockdown. We have seen a few thousand log records in > >> the buffers > >>> in > >>>>>>>> > >>>>>>>> > >>>>>> these > >>>>>> > >>>>>> > >>>>>>>> temporary moments in our stress labs (nothing you > could really > >>>>>>>> > >>>>>>>> > >>>>>> witness on a > >>>>>> > >>>>>> > >>>>>>>> normal SBS box though). > >>>>>>>> > >>>>>>>> Adar. > >>>>>>>> __________________________________________ > >>>>>>>> Adar Greenshpon | Program Manager | Microsoft ISA Server > >>>>>>>> adarg@xxxxxxxxxxxxx <mailto:adarg@xxxxxxxxxxxxx> > >>>>>>>> > >>>>>>>> > >>>>>> <mailto:adarg@xxxxxxxxxxxxx> > >>>>>> > >>>>>> > >>>>>>>> | Tel: +972.4.856.1077 | SMS/Cell: +972.54.666.4579 > >>>>>>>> > >>>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx] > >>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d> > >> <mailto:thor@xxxxxxxxxxxxxxx%5d> > >>>>>>>> Sent: Wednesday, June 28, 2006 8:26 PM > >>>>>>>> To: ISA-MVP > >>>>>>>> Subject: Re: [ISAServer] Firewall database corruption > >> due to power > >>>>>>>> > >>>>>>>> > >>>>>> outtage > >>>>>> > >>>>>> > >>>>>>>> If that is what you guys are expecting to happen, I > >> can tell you, > >>>>>>>> > >>>>>>>> > >>>>>> that > >>>>>> > >>>>>> > >>>>>>>> *ain't* the case - not when logging to SQL on another > >> box, anyway. > >>>>>>>> > >>>>>>>> > >>>>>> There's > >>>>>> > >>>>>> > >>>>>>>> no "using as much memory as it can" going on. It > >> seems to be more > >>>>>>>> > >>>>>>>> > >>>>>> like "I > >>>>>> > >>>>>> > >>>>>>>> did not get an acknowledgement that the last transaction was > >>>> written, > >>>>>>>> > >>>>>>>> > >>>>>> and > >>>>>> > >>>>>> > >>>>>>>> it's been 2 seconds, so I'm going into lockdown." > >>>>>>>> > >>>>>>>> I'm looking forward to what info you can get from > your "logging > >>>>>>>> > >>>>>>>> > >>>>>> diva." > >>>>>> > >>>>>> > >>>>>>>> t > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On 6/28/06 8:43 AM, "Jim Harrison (ISA)" > >>>> <Jim.Harrison@xxxxxxxxxxxxx> > >>>>>>>> spoketh to all: > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> Actually, it's not a fixed value (or > >> user-changeable), other than > >>>>>>>>> > >>>>>>>>> > >>>>>> being > >>>>>> > >>>>>> > >>>>>>>>> self-regulating. > >>>>>>>>> ISA logging will use as much memory as it can until > >> it determines > >>>>>>>>> > >>>>>>>>> > >>>>>> that > >>>>>> > >>>>>> > >>>>>>>>> growth is unchecked. Thus, how quickly ISA will cry > >> foul depends > >>>> in > >>>>>>>>> part on how much free memory is available. > >>>>>>>>> > >>>>>>>>> I'll ask our logging diva for details, but this sums > >> it up pretty > >>>>>>>>> > >>>>>>>>> > >>>>>> well. > >>>>>> > >>>>>> > >>>>>>>>> Jim Harrison > >>>>>>>>> SASD (ISA SE) > >>>>>>>>> If We Can't Fix It - It Ain't Broke! > >>>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx] > >>>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d> > >> <mailto:thor@xxxxxxxxxxxxxxx%5d> > >>>>>>>>> Sent: Wednesday, June 28, 2006 8:09 AM > >>>>>>>>> To: ISA-MVP > >>>>>>>>> Subject: Re: [ISAServer] Firewall database corruption due to > >>> power > >>>>>>>>> outtage > >>>>>>>>> > >>>>>>>>> It must be about a 1k buffer, then. I repeatedly put ISA in > >>>> lockdown > >>>>>>>>> within > >>>>>>>>> seconds of maintenance jobs being run, or other "data > >> intensive" > >>>>>>>>> operations > >>>>>>>>> on the DB. I would start the job, in in a few seconds, get > >>>>>>>>> > >>>>>>>>> > >>>>>> disconnected > >>>>>> > >>>>>> > >>>>>>>>> form my TS session because of it-- It wasn't a "one > >> time" thing- > >>> it > >>>>>>>>> > >>>>>>>>> > >>>>>> was > >>>>>> > >>>>>> > >>>>>>>>> consistent and repeatable. That's the only reason I > went ahead > >>> and > >>>>>>>>> disabled > >>>>>>>>> lockdown. > >>>>>>>>> > >>>>>>>>> Is the buffer user definable somewhere? What exactly > >> does your > >>>>>>>>> > >>>>>>>>> > >>>>>> script > >>>>>> > >>>>>> > >>>>>>>>> do? > >>>>>>>>> > >>>>>>>>> t > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On 6/28/06 7:54 AM, "Jim Harrison (ISA)" > >>>>>>>>> > >>>>>>>>> > >>>>>> <Jim.Harrison@xxxxxxxxxxxxx> > >>>>>> > >>>>>> > >>>>>>>>> spoketh to all: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Actually, that's not true; ISA doesn't go into > lockdown until > >>> the > >>>>>>>>>> logging buffer is filled. > >>>>>>>>>> I agree that a "backup logging" concept is useful. > >>>>>>>>>> I have a script that works for ISA 2000; it shouldn't be > >>> difficult > >>>>>>>>>> > >>>>>>>>>> > >>>>>> to > >>>>>> > >>>>>> > >>>>>>>>>> "agnosticize" it. > >>>>>>>>>> > >>>>>>>>>> Jim Harrison > >>>>>>>>>> Security Platform Group (ISA SE) > >>>>>>>>>> > >>>>>>>>>> If We Can't Fix It - It Ain't Broke! > >>>>>>>>>> > >>>>>>>>>> -----Original Message----- > >>>>>>>>>> From: Thor (Hammer of God) [mailto:thor@xxxxxxxxxxxxxxx] > >>>>>>>>>> <mailto:thor@xxxxxxxxxxxxxxx%5d> > >>> <mailto:thor@xxxxxxxxxxxxxxx%5d> > >>>>>>>>>> Sent: Wednesday, June 28, 2006 7:34 AM > >>>>>>>>>> To: ISA-MVP > >>>>>>>>>> Subject: Re: [ISAServer] Firewall database > corruption due to > >>> power > >>>>>>>>>> outtage > >>>>>>>>>> > >>>>>>>>>> I'd like to see some way to at least "buffer" the logging > >>> requests > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> when > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> logging to an off-box ODBC destination, like SQL. > >> ISA is quite > >>>>>>>>>> intolerant > >>>>>>>>>> of even the slightest interruption in logging. > But, I guess > >>> that > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> makes > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> sense if the goal is to have "evidentiary credibility." > >>>>>>>>>> > >>>>>>>>>> It wouldn't have to be something l337 like a > secondary local > >>>>>>>>>> > >>>>>>>>>> > >>>>>> logging > >>>>>> > >>>>>> > >>>>>>>>>> option > >>>>>>>>>> or anything like that-- I think being able to set a timeout > >>> value > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> would > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> would. With larger db's, a maintenance job would kick the > >>> server > >>>>>>>>>> > >>>>>>>>>> > >>>>>> into > >>>>>> > >>>>>> > >>>>>>>>>> lockdown, even though the SQL service is still running... > >>>>>>>>>> > >>>>>>>>>> Jim, I know you wanted me to get my infrastructure > >> layout to you > >>>>>>>>>> > >>>>>>>>>> > >>>>>> again > >>>>>> > >>>>>> > >>>>>>>>>> for > >>>>>>>>>> my solution to the SQL db size problems, but I > haven't had a > >>>> chance > >>>>>>>>>> > >>>>>>>>>> > >>>>>> to > >>>>>> > >>>>>> > >>>>>>>>>> find > >>>>>>>>>> that old email. I should just re-create it ;) > >> Regardless, even > >>>>>>>>>> > >>>>>>>>>> > >>>>>> with > >>>>>> > >>>>>> > >>>>>>>>>> that > >>>>>>>>>> "solution" in place, I've had to disable lockdown mode when > >>>> logging > >>>>>>>>>> > >>>>>>>>>> > >>>>>> to > >>>>>> > >>>>>> > >>>>>>>>>> SQL > >>>>>>>>>> for that very reason. > >>>>>>>>>> > >>>>>>>>>> t > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On 6/28/06 7:19 AM, "Thomas W Shinder" > <tshinder@xxxxxxxxxxx> > >>>>>>>>>> > >>>>>>>>>> > >>>>>> spoketh > >>>>>> > >>>>>> > >>>>>>>>> to > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> all: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> And it's a really good security and forensics decision. > >>>>>>>>>>> > >>>>>>>>>>> Thomas W Shinder, M.D. > >>>>>>>>>>> Site: www.isaserver.org > >>>>>>>>>>> Blog: http://blogs.isaserver.org/shinder/ > >>>>>>>>>>> Book: http://tinyurl.com/3xqb7 MVP -- ISA Firewalls > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>> From: Amy Babinchak > [mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx] > >>>>>>>>>>>> <mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx%5d> > >>>>>>>>>>>> <mailto:amy@xxxxxxxxxxxxxxxxxxxxxxxxxx%5d> > >>>>>>>>>>>> Sent: Wednesday, June 28, 2006 6:28 AM > >>>>>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx > >>>>>>>>>>>> Subject: RE: [ISAServer] Firewall database > >> corruption due to > >>>>>>>>>>>> power outtage > >>>>>>>>>>>> > >>>>>>>>>>>> I like that ISA shuts down if it can't log. It's > >> an indicator > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>> that > >>>>>> > >>>>>> > >>>>>>>>>>>> something has gone wrong. > >>>>>>>>>>>> > >>>>>>>>>>>> Amy > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>> From: Susan Bradley, CPA aka Ebitz - SBS Rocks [MVP] > >>>>>>>>>>>> [mailto:sbradcpa@xxxxxxxxxxx] > >> <mailto:sbradcpa@xxxxxxxxxxx%5d> > >>>>>>>>>>>> <mailto:sbradcpa@xxxxxxxxxxx%5d> > >>>>>>>>>>>> Sent: Wednesday, June 28, 2006 12:12 AM > >>>>>>>>>>>> To: isaserver@xxxxxxxxxxxxxxx > >>>>>>>>>>>> Subject: [ISAServer] Firewall database corruption > >> due to power > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> outtage > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>>> With the caveat that we all know we need a good > >> UPS but like > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>> stuff > >>>>>> > >>>>>> > >>>>>>>>>>>> happens.. I've seen two folks recently have issues > >> getting a > >>>>>>>>>>>> server back > >>>>>>>>>>>> > >>>>>>>>>>>> up have a 'dirty shutdown' due to power outtage > >> that ended up > >>>>>>>>>>>> corrupting > >>>>>>>>>>>> > >>>>>>>>>>>> the ISA firewall database. > >>>>>>>>>>>> > >>>>>>>>>>>> Short of uninstalling and reinstalling... short of > >> adjusting > >>>>>>>>>>>> ISA so it > >>>>>>>>>>>> won't shut down if it can't log... short of a > better UPS... > >>> any > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>> other > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>>>> recommendations on handling this better? > >>>>>>>>>>>> > >>>>>>>>>>>> --- > >>>>>>>>>>>> To subscribe to the list - send an email to > >>> list@xxxxxxxxxxxxxxx > >>>>>>>>>>>> In the subject line put in JOIN > isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>>>> youremailaddress > >>>>>>>>>>>> > >>>>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx > >>>>>>>>>>>> In the subject line put in LEAVE > isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>>>> youremailaddress > >>>>>>>>>>>> > >>>>>>>>>>>> Don't forget the comma! > >>>>>>>>>>>> --- > >>>>>>>>>>>> To subscribe to the list - send an email to > >>> list@xxxxxxxxxxxxxxx > >>>>>>>>>>>> In the subject line put in JOIN > isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>>>> youremailaddress > >>>>>>>>>>>> > >>>>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx > >>>>>>>>>>>> In the subject line put in LEAVE > isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>>>> youremailaddress > >>>>>>>>>>>> > >>>>>>>>>>>> Don't forget the comma! > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> --- > >>>>>>>>>>> To subscribe to the list - send an email to > >>> list@xxxxxxxxxxxxxxx > >>>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> youremailaddress > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx > >>>>>>>>>>> In the subject line put in LEAVE > isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> youremailaddress > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> Don't forget the comma! > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> --- > >>>>>>>>>> To subscribe to the list - send an email to > >> list@xxxxxxxxxxxxxxx > >>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>> youremailaddress > >>>>>>>>>> > >>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx > >>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>> youremailaddress > >>>>>>>>>> > >>>>>>>>>> Don't forget the comma! > >>>>>>>>>> --- > >>>>>>>>>> To subscribe to the list - send an email to > >> list@xxxxxxxxxxxxxxx > >>>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> youremailaddress > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx > >>>>>>>>>> In the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> youremailaddress > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Don't forget the comma! > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> --- > >>>>>>>>> To subscribe to the list - send an email to > >> list@xxxxxxxxxxxxxxx > >>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>> youremailaddress > >>>>>>>>> > >>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx In > >>>>>>>>> the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>> youremailaddress > >>>>>>>>> > >>>>>>>>> Don't forget the comma! > >>>>>>>>> --- > >>>>>>>>> To subscribe to the list - send an email to > >> list@xxxxxxxxxxxxxxx > >>>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>> > >>>>>>>>> > >>>>>> youremailaddress > >>>>>> > >>>>>> > >>>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx In > >>>>>>>>> the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >>>>>>>>> > >>>>>>>>> > >>>>>> youremailaddress > >>>>>> > >>>>>> > >>>>>>>>> Don't forget the comma! > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> --- > >>>>>>>> To subscribe to the list - send an email to > >> list@xxxxxxxxxxxxxxx > >>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>>>>>>> > >>>>>>>> > >>>>>> youremailaddress > >>>>>> > >>>>>> > >>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx In > >>>>>>>> the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >>>>>>>> > >>>>>>>> > >>>>>> youremailaddress > >>>>>> > >>>>>> > >>>>>>>> Don't forget the comma! > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> --- > >>>>>>>> To subscribe to the list - send an email to > >> list@xxxxxxxxxxxxxxx > >>>>>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>>>>>>> > >>>>>>>> > >>>>>> youremailaddress > >>>>>> > >>>>>> > >>>>>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx In > >>>>>>>> the subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >>>>>>>> > >>>>>>>> > >>>>>> youremailaddress > >>>>>> > >>>>>> > >>>>>>>> Don't forget the comma! > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>>> --- > >>>>> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx > >>>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>>> youremailaddress > >>>>> > >>>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx In the > >>>>> subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >>>> youremailaddress > >>>>> > >>>>> Don't forget the comma! > >>>>> > >>>>> > >>>>> > >>>> --- > >>>> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx > >>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>>> youremailaddress > >>>> > >>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx In the > >>>> subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >>>> youremailaddress > >>>> > >>>> Don't forget the comma! > >>>> --- > >>>> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx > >>>> In the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >>> youremailaddress > >>>> > >>>> To leave the list - send an email to list@xxxxxxxxxxxxxxx In the > >>>> subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >>> youremailaddress > >>>> > >>>> Don't forget the comma! > >>>> > >>>> > >>> > >>> > >>> > >>> > >> > >> > >> --- > >> To subscribe to the list - send an email to list@xxxxxxxxxxxxxxx In > >> the subject line put in JOIN isaserver@xxxxxxxxxxxxxxx, > >> youremailaddress > >> > >> To leave the list - send an email to list@xxxxxxxxxxxxxxx In the > >> subject line put in LEAVE isaserver@xxxxxxxxxxxxxxx, > >> youremailaddress > >> > >> Don't forget the comma! > >> > >> > > > > > > > > > > All mail to and from this domain is GFI-scanned.