Yeah, there are a ton of them. Took me a couple of months to narrow down my choices to Elron and WebSpy. It's always better to have your own custom app, but for those that don't have a development team or the resources.... I wrote an ASP app that sounds similar to what you've put together that worked for a little while, but our needs outgrew it and I wasn't being paid to bang out code at the time. It also depends on who will be generating the reports and what security you need to put into generating the reports. -Shawn ----- Shawn R. Quillman Robert Bosch Corporation RBNA/CSA1 38000 Hills Tech Drive Farmington Hills, MI 48331 (248) 553-1164 (P) (248) 848-6969 (F) shawn.quillman@xxxxxxxxxxxx -----Original Message----- From: cismic [mailto:cismic@xxxxxxx] Sent: Thursday, April 15, 2004 1:53 PM To: [ISAserver.org Discussion List] Subject: [isalist] Re: ISA Logs Problem http://www.ISAserver.org There are too many choices. I think as any developer who starts a product seeing his/her idea put into a program can somtime limit the big picture. I mean, to put all the beans in one basket. Sniffer, packet pusher, import/export, and then to analyze all that stuff. Me, I like pulling all that stuff into my sql box ( not every one will have that) and then do the churning there. I seperate the raw daily import into various daily summaries. Weekly the daily summaries are pulled into weekly summaries etc. An example report:(cust and paste into notepad and remove wordwrap) This an hourly report of status codes my version of cross tab for sql. SummaryYear SummaryMonth SummaryDay sc_Status Hour00 Hour01 Hour02 Hour03 Hour04 Hour05 Hour06 Hour07 Hour08 Hour09 Hour10 Hour11 Hour12 Hour13 Hour14 Hour15 Hour16 Hour17 Hour18 Hour19 Hour20 Hour21 Hour22 Hour23 ----------- ------------ ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- 2004 4 14 200 0 0 2 2 0 10 0 0 0 1 3 6 0 0 0 0 0 0 1 0 0 0 0 10 2004 4 14 302 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 2004 4 14 304 0 0 0 0 0 11 0 0 0 11 11 2 0 0 0 0 0 0 0 0 0 0 0 0 2004 4 14 404 0 7 12 12 3 4 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 3 3 8 2004 4 14 414 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 2004 4 14 500 0 0 0 0 0 1 0 0 0 0 2 0 0 0 0 2 0 0 1 3 1 0 3 2 2004 4 14 10054 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 2004 4 14 12202 0 1 0 0 0 5 6 2 5 7 2 3 3 7 0 3 4 0 4 3 3 0 0 0 and another example report Month c2000 c2001 c2002 c2003 c2004 ----------- ----------- ----------- ----------- ----------- ----------- 1 0 0 0 10455 85 2 0 0 0 18742 110 3 0 0 0 57478 3808 4 0 0 0 9498 4439 5 0 0 0 12983 0 6 0 0 0 5449 0 7 0 0 0 24344 0 8 0 0 0 131974 0 9 0 0 7833 27176 0 10 0 0 18759 94402 0 11 0 0 20958 150 0 12 0 0 61536 150 0 I can quickly show the month and year with the hits for each month. I've been working with logs for awhile and I keep learning new things. I'm hoping that once people use the vbscripts that I've created for importing logs that I'll get some feedback With that I'll start uploading the SQL that does all the parsing and functions that I've been working on to analyze the logs. I also have some C# programs that I'm testing that are faster then the vbscripts. I really appreciate all the great information. Joseph ----- Original Message ----- From: "Quillman Shawn (RBNA/CSA1)" <Shawn.Quillman@xxxxxxxxxxxx> To: "[ISAserver.org Discussion List]" <isalist@xxxxxxxxxxxxx> Sent: Thursday, April 15, 2004 10:25 AM Subject: [isalist] Re: ISA Logs Problem http://www.ISAserver.org WebSpy is, in my opinion, the best log analyzer out there. It was quite a bit quicker than WebTrends and had a nice interface. It was between WebSpy and Elron when it came down to it for me. If it was just an every now and then thing I would have gone with WebSpy, but Elron makes it really easy to do on-demand reporting for non-tech folks like HR. There's still log importing you need to do with WebSpy (if I remember correctly) which I didn't want to have to deal with. Elron's a packet sniffer and as such is a real-time analyzer. Costs a bit more and implementation is quite a bit more involved, but with the dynamic LDAP support it now has it is quite powerful. Since it's a packet analyzer, though, you do have to have it on the same segment as whichever ISA interface you're interested in. That meant I had to have 2 Elron boxes, one for each of our proxies, since the proxies are in two different offices a few hundred miles apart. Can also mean some fun with switches, getting ports mirrored and all. It is designed with distributed environments in mind so you can have X number of capture points that log to one central database for your reports. In that scenario you'd probably want some decent bandwidth between those locations if you have a lot of web traffic getting logged. For large organizations with the resources I'd go with Elron. Smaller places, I'd go with WebSpy. -Shawn ----- Shawn R. Quillman Robert Bosch Corporation RBNA/CSA1 38000 Hills Tech Drive Farmington Hills, MI 48331 (248) 553-1164 (P) (248) 848-6969 (F) shawn.quillman@xxxxxxxxxxxx -----Original Message----- From: Thomas W Shinder [mailto:tshinder@xxxxxxxxxxx] Sent: Thursday, April 15, 2004 1:07 PM To: [ISAserver.org Discussion List] Subject: [isalist] Re: ISA Logs Problem http://www.ISAserver.org Hi Shawn, How about WebSpy Giga? I've been using that to process large logs, and it does tax my P4-1.8 with 1 GB, but the results do finally appear. Thanks! Tom Thomas W Shinder www.isaserver.org/shinder ISA 2004 Beta - Get it now! http://www.microsoft.com/isaserver/beta/default.asp ISA Server and Beyond: http://tinyurl.com/1jq1 Configuring ISA Server: http://tinyurl.com/1llp -----Original Message----- From: Quillman Shawn (RBNA/CSA1) [mailto:Shawn.Quillman@xxxxxxxxxxxx] Sent: Thursday, April 15, 2004 12:02 PM To: [ISAserver.org Discussion List] Subject: [isalist] Re: ISA Logs Problem http://www.ISAserver.org Sweet, that's good to hear :) I like WebTrends, but my logs outgrew it. I had to throw some serious memory at the box I ran it on. My logs were getting to be > 1GB per day on each of my 2 proxies. I had to do some preparsing just to do a specific user report and couldn't do any general reporting for more than a day or two at a time. We ended up switching over to Elron's Internet Manager now that they have dynamic LDAP support and can tie into AD without having to setup import jobs for user information. We did that just before I transferred out of the IT dept, though, so I don't really know how it's working for them. Elron has the SurfControl database built into it for access restrictions and categorization (I think it's SurfControl's) and I guarantee you that that part is working... No more web mail for us! Which is, I gotta admin, as it should be :) -Shawn ----- Shawn R. Quillman Robert Bosch Corporation RBNA/CSA1 38000 Hills Tech Drive Farmington Hills, MI 48331 (248) 553-1164 (P) (248) 848-6969 (F) shawn.quillman@xxxxxxxxxxxx -----Original Message----- From: cismic [mailto:cismic@xxxxxxx] Sent: Thursday, April 15, 2004 12:38 PM To: [ISAserver.org Discussion List] Subject: [isalist] Re: ISA Logs Problem http://www.ISAserver.org It is a good script too! I used it on my linux box for testing. What do you think about WebTrends? Thank you, Joseph ----- Original Message ----- From: "Quillman Shawn (RBNA/CSA1)" <Shawn.Quillman@xxxxxxxxxxxx> To: "[ISAserver.org Discussion List]" <isalist@xxxxxxxxxxxxx> Sent: Thursday, April 15, 2004 9:10 AM Subject: [isalist] Re: ISA Logs Problem http://www.ISAserver.org The perl script is just a log converter, not an analyzer. It will take a W3C formatted log and convert it to ISA format. I wrote it for a problem similar to this. I had a slew of W3C formatted logs and an analyzer (WebTrends Firewall Suite) that only accepted ISA formatted logs. -Shawn ----- Shawn R. Quillman Robert Bosch Corporation RBNA/CSA1 38000 Hills Tech Drive Farmington Hills, MI 48331 (248) 553-1164 (P) (248) 848-6969 (F) shawn.quillman@xxxxxxxxxxxx -----Original Message----- From: cismic [mailto:cismic@xxxxxxx] Sent: Thursday, April 15, 2004 12:06 PM To: [ISAserver.org Discussion List] Subject: [isalist] Re: ISA Logs Problem http://www.ISAserver.org Hi Ahmed, There are a number of packages out there. I've actually written my own. Although I do my queries when I get the data into my database. Microsoft has the log parser which you can use to query the data. Also, out on http://isatools.org you can find a couple of other parsers. I think that there even is a pearl script. Thank you, Joseph ----- Original Message ----- From: "Nabil, Ahmed" <anmahmou@xxxxxxxxxx> To: "[ISAserver.org Discussion List]" <isalist@xxxxxxxxxxxxx> Sent: Thursday, April 15, 2004 4:04 AM Subject: [isalist] Re: ISA Logs Problem http://www.ISAserver.org Thanks for the Info. How can I change then my logs to reflect my time, the whole logs (Firewall, proxy.....etc) Also do you know any Software to read my logs instead of the ISA format, a software to filter and search for items. Thanks, Ahmed -----Original Message----- From: cismic [mailto:cismic@xxxxxxx] Sent: Thursday, April 15, 2004 11:09 AM To: [ISAserver.org Discussion List] Subject: [isalist] Re: ISA Logs Problem http://www.ISAserver.org Hi Nabil, Logs are kept in what was used to be called greenwich time. or the 24 hour clock etc. Pacific time is -8 from the green wich time. Take a look at this UTC OR GMT I think that a good site for those who would like to know is http://greenwichmeantime.com from this site you'll be able to determine what your Zulu time settings should be. For example : convlog -ie Logfile.log -t ncsa:-0800 is for the pacific time zone. The convlog also has the following syntax available: Usage: convlog [options] LogFile Options: -i<i|n|e> = input logfile type i - MS Internet Standard Log File n - NCSA Common Log File format e - W3C Extended Log File Format -t <ncsa[:GMTOffset] | none> default i -o <output directory> default = curren -x save non-www entries to a .dmp logf -d = convert IP addresses to DNS -l<0|1|2> = Date locale format for MS 0 - MM/DD/YY (default e.g. US) 1 - YY/MM/DD (e.g. Japan) 2 - DD.MM.YY (e.g. Germany) Examples: convlog -ii in*.log -d -t ncsa:+0800 convlog -in ncsa*.log -d convlog -ii jra*.log -t none ----- Original Message ----- From: "Nabil, Ahmed" <anmahmou@xxxxxxxxxx> To: "[ISAserver.org Discussion List]" <isalist@xxxxxxxxxxxxx> Sent: Thursday, April 15, 2004 12:18 AM Subject: [isalist] ISA Logs Problem http://www.ISAserver.org Good morning All, I am having two problems with my ISA Logs and I need your advice. 1. There is a time shift with almost 7 hours in the logs, its not showing the correct exact time of each web request. How can I fix this issue ? 2. Its very hard to check these logs in this format, is there any well known program to import this Logs to read it in an organized way ? Thanks for your help, Ahmed ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: cismic@xxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub') ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: anmahmou@xxxxxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub') ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: cismic@xxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub') ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: shawn.quillman@xxxxxxxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub') ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: cismic@xxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub') ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: shawn.quillman@xxxxxxxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub') ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: tshinder@xxxxxxxxxxxxxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub') ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: shawn.quillman@xxxxxxxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub') ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: cismic@xxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub') ------------------------------------------------------ List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ ------------------------------------------------------ Other Internet Software Marketing Sites: Leading Network Software Directory: http://www.serverfiles.com No.1 Exchange Server Resource Site: http://www.msexchange.org Windows Security Resource Site: http://www.windowsecurity.com/ Network Security Library: http://www.secinf.net/ Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com ------------------------------------------------------ You are currently subscribed to this ISAserver.org Discussion List as: shawn.quillman@xxxxxxxxxxxx To unsubscribe send a blank email to $subst('Email.Unsub')