Re: ISA Logs Problem

Yeah, I hope so too!  I woulda developed further, but I do a ton with my church 
and don't have the time...  That and a 2 year old and another on the way! :)

You know what would be cool:  an application filter that would query the mail 
blacklists (MAPS, et al) and reject messages from those lists.  Let someone 
else figure out who the spammers are and maintain the lists!  I'm more than 
happy to subscribe to their services.  That should't be too difficult to 
implement.  It's just a matter of parsing out a domain and then doing a dns 
query to their dns servers.

-Shawn

-----
Shawn R. Quillman
Robert Bosch Corporation RBNA/CSA1
38000 Hills Tech Drive
Farmington Hills, MI 48331
(248) 553-1164 (P) (248) 848-6969 (F)
shawn.quillman@xxxxxxxxxxxx


-----Original Message-----
From: cismic [mailto:cismic@xxxxxxx]
Sent: Thursday, April 15, 2004 2:16 PM
To: [ISAserver.org Discussion List]
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org

Hi Shawn,

I'm not getting paid either, only in experience! Will that find me a paying
gig??? Um, I hope so.

I've actually written my own portal that will be released soon. With the
portal I can quickly add
modules and have others download the modules.  Things, like whois,
displaying the packet filter by port against the IANA and trogjan port
listings that I've been putting togeter etc.  Then I started converting some
of the modules
so that they will run under portal server 2003.

I really like this list because I always find a tidbit of information that I
can add to what I've been trying to put together.
So, any and all ideas <grin> that you or others have are greatly
appreciated.

Thank you,

Joseph

----- Original Message ----- 
From: "Quillman Shawn (RBNA/CSA1)" <Shawn.Quillman@xxxxxxxxxxxx>
To: "[ISAserver.org Discussion List]" <isalist@xxxxxxxxxxxxx>
Sent: Thursday, April 15, 2004 11:06 AM
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org


Yeah, there are a ton of them.  Took me a couple of months to narrow down my
choices to Elron and WebSpy.  It's always better to have your own custom
app, but for those that don't have a development team or the resources....
I wrote an ASP app that sounds similar to what you've put together that
worked for a little while, but our needs outgrew it and I wasn't being paid
to bang out code at the time.  It also depends on who will be generating the
reports and what security you need to put into generating the reports.

-Shawn

-----
Shawn R. Quillman
Robert Bosch Corporation RBNA/CSA1
38000 Hills Tech Drive
Farmington Hills, MI 48331
(248) 553-1164 (P) (248) 848-6969 (F)
shawn.quillman@xxxxxxxxxxxx


-----Original Message-----
From: cismic [mailto:cismic@xxxxxxx]
Sent: Thursday, April 15, 2004 1:53 PM
To: [ISAserver.org Discussion List]
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org

There are too many choices. I think as any developer who starts a product
seeing his/her idea
put into a program can somtime limit the big picture.  I mean, to put all
the beans in one basket.
Sniffer, packet pusher, import/export, and then to analyze all that stuff.
Me, I like pulling all that stuff into
my sql box ( not every one will have that) and then do the churning there.
I seperate the raw daily import into
various daily summaries. Weekly the daily summaries are pulled into weekly
summaries etc.
An example report:(cust and paste into notepad and remove wordwrap) This an
hourly report of status codes
my version of cross tab for sql.
SummaryYear SummaryMonth SummaryDay  sc_Status   Hour00      Hour01
Hour02      Hour03      Hour04      Hour05      Hour06      Hour07
Hour08      Hour09      Hour10      Hour11      Hour12      Hour13
Hour14      Hour15      Hour16      Hour17      Hour18      Hour19
Hour20      Hour21      Hour22      Hour23
----------- ------------ ----------- ----------- ----------- -----------
----------- ----------- ----------- ----------- ----------- -----------
----------- ----------- ----------- ----------- ----------- -----------
----------- ----------- ----------- ----------- ----------- -----------
----------- ----------- ----------- ----------- 
2004        4            14          200         0           0
2           2           0           10          0           0
0           1           3           6           0           0
0           0           0           0           1           0
0           0           0           10
2004        4            14          302         0           0
0           0           0           0           0           0
0           0           0           1           0           0
0           0           0           0           0           0
0           0           0           0
2004        4            14          304         0           0
0           0           0           11          0           0
0           11          11          2           0           0
0           0           0           0           0           0
0           0           0           0
2004        4            14          404         0           7
12          12          3           4           0           0
0           0           1           0           0           0
0           0           0           0           0           0
0           3           3           8
2004        4            14          414         0           1
0           0           0           0           0           0
0           0           0           0           0           0
1           0           0           0           0           0
0           0           0           1
2004        4            14          500         0           0
0           0           0           1           0           0
0           0           2           0           0           0
0           2           0           0           1           3
1           0           3           2
2004        4            14          10054       0           0
0           0           0           0           0           0
0           0           0           2           0           0
0           0           0           0           0           0
0           0           0           0
2004        4            14          12202       0           1
0           0           0           5           6           2
5           7           2           3           3           7
0           3           4           0           4           3
3           0           0           0


and another example report
Month       c2000       c2001       c2002       c2003       c2004
----------- ----------- ----------- ----------- ----------- ----------- 
1           0           0           0           10455       85
2           0           0           0           18742       110
3           0           0           0           57478       3808
4           0           0           0           9498        4439
5           0           0           0           12983       0
6           0           0           0           5449        0
7           0           0           0           24344       0
8           0           0           0           131974      0
9           0           0           7833        27176       0
10          0           0           18759       94402       0
11          0           0           20958       150         0
12          0           0           61536       150         0

I can quickly show the month and year with the hits for each month.
I've been working with logs for awhile and I keep learning new things. I'm
hoping that once people
use the vbscripts that I've created for importing logs that I'll get some
feedback With that I'll start
uploading the SQL that does all the parsing and functions that I've been
working on to analyze the logs.
I also have some C# programs that I'm testing that are faster then the
vbscripts.

I really appreciate all the great information.

Joseph

----- Original Message ----- 
From: "Quillman Shawn (RBNA/CSA1)" <Shawn.Quillman@xxxxxxxxxxxx>
To: "[ISAserver.org Discussion List]" <isalist@xxxxxxxxxxxxx>
Sent: Thursday, April 15, 2004 10:25 AM
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org


WebSpy is, in my opinion, the best log analyzer out there.  It was quite a
bit quicker than WebTrends and had a nice interface.  It was between WebSpy
and Elron when it came down to it for me.  If it was just an every now and
then thing I would have gone with WebSpy, but Elron makes it really easy to
do on-demand reporting for non-tech folks like HR.  There's still log
importing you need to do with WebSpy (if I remember correctly) which I
didn't want to have to deal with.  Elron's a packet sniffer and as such is a
real-time analyzer.  Costs a bit more and implementation is quite a bit more
involved, but with the dynamic LDAP support it now has it is quite powerful.
Since it's a packet analyzer, though, you do have to have it on the same
segment as whichever ISA interface you're interested in.  That meant I had
to have 2 Elron boxes, one for each of our proxies, since the proxies are in
two different offices a few hundred miles apart.  Can also mean some fun
with switches, getting ports mirrored and all.  It is designed with
distributed environments in mind so you can have X number of capture points
that log to one central database for your reports.  In that scenario you'd
probably want some decent bandwidth between those locations if you have a
lot of web traffic getting logged.  For large organizations with the
resources I'd go with Elron.  Smaller places, I'd go with WebSpy.

-Shawn

-----
Shawn R. Quillman
Robert Bosch Corporation RBNA/CSA1
38000 Hills Tech Drive
Farmington Hills, MI 48331
(248) 553-1164 (P) (248) 848-6969 (F)
shawn.quillman@xxxxxxxxxxxx


-----Original Message-----
From: Thomas W Shinder [mailto:tshinder@xxxxxxxxxxx]
Sent: Thursday, April 15, 2004 1:07 PM
To: [ISAserver.org Discussion List]
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org

Hi Shawn,

How about WebSpy Giga? I've been using that to process large logs, and
it does tax my P4-1.8 with 1 GB, but the results do finally appear.

Thanks!
Tom

Thomas W Shinder
www.isaserver.org/shinder
ISA 2004 Beta - Get it now!
http://www.microsoft.com/isaserver/beta/default.asp
ISA Server and Beyond: http://tinyurl.com/1jq1
Configuring ISA Server: http://tinyurl.com/1llp




-----Original Message-----
From: Quillman Shawn (RBNA/CSA1) [mailto:Shawn.Quillman@xxxxxxxxxxxx]
Sent: Thursday, April 15, 2004 12:02 PM
To: [ISAserver.org Discussion List]
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org


Sweet, that's good to hear :)  I like WebTrends, but my logs outgrew it.
I had to throw some serious memory at the box I ran it on.  My logs were
getting to be > 1GB per day on each of my 2 proxies.  I had to do some
preparsing just to do a specific user report and couldn't do any general
reporting for more than a day or two at a time.  We ended up switching
over to Elron's Internet Manager now that they have dynamic LDAP support
and can tie into AD without having to setup import jobs for user
information.  We did that just before I transferred out of the IT dept,
though, so I don't really know how it's working for them.  Elron has the
SurfControl database built into it for access restrictions and
categorization (I think it's SurfControl's) and I guarantee you that
that part is working... No more web mail for us!  Which is, I gotta
admin, as it should be :)

-Shawn

-----
Shawn R. Quillman
Robert Bosch Corporation RBNA/CSA1
38000 Hills Tech Drive
Farmington Hills, MI 48331
(248) 553-1164 (P) (248) 848-6969 (F)
shawn.quillman@xxxxxxxxxxxx


-----Original Message-----
From: cismic [mailto:cismic@xxxxxxx]
Sent: Thursday, April 15, 2004 12:38 PM
To: [ISAserver.org Discussion List]
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org

It is a good script too!  I used it on my linux box for testing.
What do you think about WebTrends?
Thank you,

Joseph
----- Original Message ----- 
From: "Quillman Shawn (RBNA/CSA1)" <Shawn.Quillman@xxxxxxxxxxxx>
To: "[ISAserver.org Discussion List]" <isalist@xxxxxxxxxxxxx>
Sent: Thursday, April 15, 2004 9:10 AM
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org


The perl script is just a log converter, not an analyzer.  It will take
a
W3C formatted log and convert it to ISA format.  I wrote it for a
problem
similar to this.  I had a slew of W3C formatted logs and an analyzer
(WebTrends Firewall Suite) that only accepted ISA formatted logs.

-Shawn

-----
Shawn R. Quillman
Robert Bosch Corporation RBNA/CSA1
38000 Hills Tech Drive
Farmington Hills, MI 48331
(248) 553-1164 (P) (248) 848-6969 (F)
shawn.quillman@xxxxxxxxxxxx


-----Original Message-----
From: cismic [mailto:cismic@xxxxxxx]
Sent: Thursday, April 15, 2004 12:06 PM
To: [ISAserver.org Discussion List]
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org

Hi Ahmed,

There are a number of packages out there.  I've actually written my own.
Although I do my queries when I get the data
into my database.  Microsoft has the log parser which you can use to
query
the data. Also, out on
http://isatools.org you can find a couple of other parsers. I think that
there even is a pearl script.

Thank you,

Joseph

----- Original Message ----- 
From: "Nabil, Ahmed" <anmahmou@xxxxxxxxxx>
To: "[ISAserver.org Discussion List]" <isalist@xxxxxxxxxxxxx>
Sent: Thursday, April 15, 2004 4:04 AM
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org

Thanks for the Info. How can I change then my logs to reflect my time,
the
whole logs (Firewall, proxy.....etc)

Also do you know any Software to read my logs instead of the ISA format,
a
software to filter and search for items.

Thanks,

Ahmed

-----Original Message-----
From: cismic [mailto:cismic@xxxxxxx]
Sent: Thursday, April 15, 2004 11:09 AM
To: [ISAserver.org Discussion List]
Subject: [isalist] Re: ISA Logs Problem


http://www.ISAserver.org

Hi Nabil,
Logs are kept in what was used to be called greenwich time. or the 24
hour
clock etc.
Pacific time is -8 from the green wich time. Take a look at this

UTC OR GMT
I think that a good site for those who would like to know is
http://greenwichmeantime.com  from this site you'll be able to determine
what your Zulu time settings should be. For example :  convlog -ie
Logfile.log -t ncsa:-0800 is for the pacific time zone. The convlog also
has the following syntax available:

Usage: convlog [options] LogFile
Options:
-i<i|n|e> = input logfile type
    i - MS Internet Standard Log File
    n - NCSA Common Log File format
    e - W3C Extended Log File Format
    -t <ncsa[:GMTOffset] | none> default i
    -o <output directory> default = curren
    -x save non-www entries to a .dmp logf
    -d = convert IP addresses to DNS
    -l<0|1|2> = Date locale format for MS
                    0 - MM/DD/YY (default e.g. US)
                    1 - YY/MM/DD (e.g. Japan)
                    2 - DD.MM.YY (e.g. Germany)

Examples:
convlog -ii in*.log -d -t ncsa:+0800
convlog -in ncsa*.log -d
convlog -ii jra*.log -t none


----- Original Message ----- 
From: "Nabil, Ahmed" <anmahmou@xxxxxxxxxx>
To: "[ISAserver.org Discussion List]" <isalist@xxxxxxxxxxxxx>
Sent: Thursday, April 15, 2004 12:18 AM
Subject: [isalist] ISA Logs Problem


http://www.ISAserver.org

Good morning All,

I am having two problems with my ISA Logs and I need your advice.

1. There is a time shift with almost 7 hours in the logs, its not
showing
the correct exact time of each web request. How can I fix this issue ?

2. Its very hard to check these logs in this format, is there any well
known
program to import this Logs to read it in an organized way ?

Thanks for your help,

Ahmed

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
cismic@xxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
anmahmou@xxxxxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
cismic@xxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
shawn.quillman@xxxxxxxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
cismic@xxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
shawn.quillman@xxxxxxxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
tshinder@xxxxxxxxxxxxxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
shawn.quillman@xxxxxxxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
cismic@xxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
shawn.quillman@xxxxxxxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as:
cismic@xxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')

------------------------------------------------------
List Archives: http://www.webelists.com/cgi/lyris.pl?enter=isalist
ISA Server Newsletter: http://www.isaserver.org/pages/newsletter.asp
ISA Server FAQ: http://www.isaserver.org/pages/larticle.asp?type=FAQ
------------------------------------------------------
Other Internet Software Marketing Sites:
Leading Network Software Directory: http://www.serverfiles.com
No.1 Exchange Server Resource Site: http://www.msexchange.org
Windows Security Resource Site: http://www.windowsecurity.com/
Network Security Library: http://www.secinf.net/
Windows 2000/NT Fax Solutions: http://www.ntfaxfaq.com
------------------------------------------------------
You are currently subscribed to this ISAserver.org Discussion List as: 
shawn.quillman@xxxxxxxxxxxx
To unsubscribe send a blank email to $subst('Email.Unsub')


Other related posts: