[isapros] Re: applying changes takes ages - reloaded
- From: "Greg Mulholland" <gmulholland@xxxxxxxxxxxx>
- To: <isapros@xxxxxxxxxxxxx>
- Date: Fri, 7 Sep 2007 15:03:05 +1000
This is the screen dump.. interestingly the two screen dumps were taken when
enabling and then disabling the same rule so all things on that front are
equal.
Greg
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Greg Mulholland
Sent: Friday, 7 September 2007 2:56 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
I can show you the screenshot of "show all processes" if you like but even
without it the firewall service appears sometimes and then not others. In
those other times, there are no kernel mode processors that have high cpu
usage.
The only evidence I do have is that the firewall service is using high cpu
usage. All attempts either point to nothing or wspfsrv
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Jim Harrison
Sent: Friday, 7 September 2007 2:43 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
The point is, your screenshot didn't indicate that.
if you don't have a definite indicator, what leads you to believe it's the
firewall service?
In your screenshot, procmon is sucking up 24%.
Something else to bear in mind; there are plenty of kernel-mode processes
that taskman doesn't show.
_____
From: isapros-bounce@xxxxxxxxxxxxx on behalf of Greg Mulholland
Sent: Thu 9/6/2007 7:30 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
Yeah I did, its either the wspfsrv.exe firewall service or its nothing.. !!
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Thor (Hammer of God)
Sent: Friday, 7 September 2007 12:06 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
Try "show processes for all users."
t
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Greg Mulholland
Sent: Thursday, September 06, 2007 6:14 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
Sorry to drag up this old chestnut but it doesn't seem to have fixed
anything as such. It seems the process wspsvr.exe is the main cpu, but whats
even weirder is that sometimes there is no real evidence of a cpu hog in the
task manager but it still sits at 100% (to prove im not joking, attached).
So I can specifically blame the firewall service as such.
I have disable all rules relating to url and domain sets for testing. I have
added another non system drive for cache files. I have disabled all logging.
It takes almost exactly 10 minutes to apply changes. Network service has
perms for the cache drive. Filemon, procmon don't show anything of interest.
Bizzaro! Im sorta lost now short of a rebuild which I will have a chance to
do in two weeks but it bugs me. If anyone has any more theory's im all ears.
Greg
You don't have to remove necessarily. Just disable them so ISA doesn't have
to process them while your making rule changes. Then enable those rules at
the end.
Glad it worked. But now Steve won't speak to me.
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Greg Mulholland
Sent: Wednesday, September 05, 2007 6:09 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
2 days ago I followed my gut feel and removed all the rules that used those
imported url and domain sets and it seems like it might have. I am back at
it today so I will report any changes. At least I have something to go on.
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Gerald G. Young
Sent: Wednesday, 5 September 2007 11:02 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
Greg,
Did Amy's suggestion work?
You wouldn't happen to have a policy backup from before the last set of
banned URLs and domain sets were imported, would you? Might you be able to
restore to confirm the additions were the cause? Providing, of course, that
Amy's suggestion didn't work.
Jason,
Thanks for the clarification. Having only worked with Enterprise, I wasn't
even aware that Standard stored the config in the local Registry. Good to
know. You wouldn't happen to know the key, would you?
HKLM\Software\Microsoft\ISA Server?? Or something along those lines?
Greg,
A couple of other thoughts, although probably unrelated. How big is the
Registry on that box? How's Registry fragmentation looking?
Cordially yours,
Jerry G. Young II
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Application Engineer
Platform Engineering & Architecture
NTT America, an NTT Communications Company
=====================================
22451 Shaw Rd.
Sterling, VA 20166
Office: 571-434-1319
Fax: 703-333-6749
Email: g.young@xxxxxxxx
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
"The truth is that there is nothing noble in being superior to somebody
else.
The only real nobility is in being superior to your former self."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Greg Mulholland
Sent: Tuesday, September 04, 2007 6:35 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
Yeah I know what you're saying. Its std version, single server, no sql
involved at all. The problem was not always there but I can't yet pinpoint
what it was that changed it. As I said I think it could have been importing
a number of banned url and domain sets.
Greg
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Gerald G. Young
Sent: Wednesday, 5 September 2007 6:47 AM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
Greg,
I didn't mean the SQL Server or MSDE for logging. There is a configuration
server piece that should use SQL Server or MSDE to hold the policies. As I
understand it - and please, someone correct me if I am wrong - applying
changes submit the changes made to policies to the configuration server.
These changes are then committed to the database in which the policy is
stored. If you have an I/O problem on that database server, that could be
the issue, especially if you aren't seeing CPU utilization spike to 100% but
only a delay that feels like it's spiked.
Is the configuration server and ISA firewall on the same box? Do you have
only one server? Is this a new server that has had this problem since
installation or did this just recently start to occur? Are you using
standard or enterprise edition?
Sorry about all the extra questions; I'm just trying to dig to help find the
scope.
Cordially yours,
Jerry G. Young II
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Application Engineer
Platform Engineering & Architecture
NTT America, an NTT Communications Company
=====================================
22451 Shaw Rd.
Sterling, VA 20166
Office: 571-434-1319
Fax: 703-333-6749
Email: g.young@xxxxxxxx
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
"The truth is that there is nothing noble in being superior to somebody
else.
The only real nobility is in being superior to your former self."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Greg Mulholland
Sent: Tuesday, September 04, 2007 4:32 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
Thanks Gerald
I was logging to flat txt but have since turn it off. The funny thing was
the last time I looked the task manager didn't show any service with above
average cpu utilization which was bizarre, hence why I never bothered with
procmon. I'll try it again in case my eyes lies.
Greg
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Gerald G. Young
Sent: Tuesday, 4 September 2007 11:52 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
EDITED
Oops... I lied about ProcMon showing loaded DLLs, although, it will give you
a LOT more insight on what a given process is doing. The other piece, which
DOES show DLLs loaded for a process, is Process Explorer. That's found at
the following location:
http://www.microsoft.com/technet/sysinternals/utilities/processexplorer.mspx
-------
Previously on RE: [isapros] applying changes takes ages...
Task Manager isn't showing the name of the process that is using all CPU
resources?
If it is and you just have no insight into what that process is doing, take
a look at Process Monitor from Sysinternals/Microsoft. It provides a lot
more detail about what a process is doing, even going as far as showing
which DLLs are loaded for it. This tool can be found at the URL below.
http://www.microsoft.com/technet/sysinternals/utilities/processmonitor.mspx
A couple of other questions. Where is the database sitting? Is it SQL
Server of MSDE? If it's on a different server, what does its performance
counters look like? If SQL Server (whether local or remote), have you
checked SQL logs and locks? Anything at all in the Event Logs on the ISA
box?
Cordially yours,
Jerry G. Young II
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Application Engineer
Platform Engineering & Architecture
NTT America, an NTT Communications Company
=====================================
22451 Shaw Rd.
Sterling, VA 20166
Office: 571-434-1319
Fax: 703-333-6749
Email: g.young@xxxxxxxx
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
"The truth is that there is nothing noble in being superior to somebody
else.
The only real nobility is in being superior to your former self."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx] On
Behalf Of Greg Mulholland
Sent: Tuesday, September 04, 2007 6:28 AM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] applying changes takes ages
Hey gang got an issue here I'm trying to work through.
I have an isa 2006 std box running in firewall mode that takes an exorbitant
amount of time to apply any changes. Like I'm talking 5-10 minutes.
A little background:
It's a basic win2k3 sp2 install with around 20 rules (least privilege), a
few published services like smtp,web, owa, VPN clients (pptp). I read about
an issue on multi core processors but this is pure single core. I also read
about when a number of web servers are published with multiple link
translations it can take time, but we aren't using any link translation atm.
The server seems to go CPU bound in a big way when I apply any rule or
config changes, sits around 100%. Incidentally there are no simultaneous or
subsequent ISA alters nor are there any system or app log events fired.
I'm not sure what's pinning the CPU, the task manager doesn't give me any
real heads up on anything, the box at load is sitting around 300-500mb ram
free, we've recently added a new disk for logging but in this case I've even
turned off logging for the time being to try and get to the bottom of the
performance. My gut feeling was that there was a bad rule or something in
the ruleset but I've reviewed these and they seem to be OK. I have run an
ISABPA and didn't find anything more than the usual you are using strict
rpc, and a few other red herrings. I haven't as yet ran a sniff whilst the
changes are being applied but I would have assumed that would be somewhat
fruitless anyway.
Can anyone shed any more light or give me any pov's.
Thanks
Greg
=================================
This email message is intended for the use of the person to whom it has been
sent, and may contain information that is confidential or legally protected.
If you are not the intended recipient or have received this message in
error, you are not authorized to copy, distribute, or otherwise use this
message or its attachments. Please notify the sender immediately by return
e-mail and permanently delete this message and any attachments. NTT America
makes no warranty that this email is error or virus free. Thank you.
=================================
This email message is intended for the use of the person to whom it has been
sent, and may contain information that is confidential or legally protected.
If you are not the intended recipient or have received this message in
error, you are not authorized to copy, distribute, or otherwise use this
message or its attachments. Please notify the sender immediately by return
e-mail and permanently delete this message and any attachments. NTT America
makes no warranty that this email is error or virus free. Thank you.
=================================
This email message is intended for the use of the person to whom it has been
sent, and may contain information that is confidential or legally protected.
If you are not the intended recipient or have received this message in
error, you are not authorized to copy, distribute, or otherwise use this
message or its attachments. Please notify the sender immediately by return
e-mail and permanently delete this message and any attachments. NTT America
makes no warranty that this email is error or virus free. Thank you.
All mail to and from this domain is GFI-scanned.
Other related posts:
