[isapros] Re: applying changes takes ages - reloaded
- From: "Gerald G. Young" <g.young@xxxxxxxx>
- To: <isapros@xxxxxxxxxxxxx>
- Date: Fri, 7 Sep 2007 03:34:50 -0400
Are you using McAfee on the ISA Server, by any chance?
Do any of the hits from the query below help you?
http://www.google.com/search?hl=en&q=wspsrv.exe+high+cpu
Cordially yours,
Jerry G. Young II
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Application Engineer
Platform Engineering & Architecture
NTT America, an NTT Communications Company
=====================================
22451 Shaw Rd.
Sterling, VA 20166
Office: 571-434-1319
Fax: 703-333-6749
Email: g.young@xxxxxxxx
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
"The truth is that there is nothing noble in being superior to somebody
else.
The only real nobility is in being superior to your former self."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Greg Mulholland
Sent: Friday, September 07, 2007 1:03 AM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
This is the screen dump.. interestingly the two screen dumps were taken
when enabling and then disabling the same rule so all things on that
front are equal.
Greg
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Greg Mulholland
Sent: Friday, 7 September 2007 2:56 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
I can show you the screenshot of "show all processes" if you like but
even without it the firewall service appears sometimes and then not
others. In those other times, there are no kernel mode processors that
have high cpu usage.
The only evidence I do have is that the firewall service is using high
cpu usage. All attempts either point to nothing or wspfsrv
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Jim Harrison
Sent: Friday, 7 September 2007 2:43 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
The point is, your screenshot didn't indicate that.
if you don't have a definite indicator, what leads you to believe it's
the firewall service?
In your screenshot, procmon is sucking up 24%.
Something else to bear in mind; there are plenty of kernel-mode
processes that taskman doesn't show.
________________________________
From: isapros-bounce@xxxxxxxxxxxxx on behalf of Greg Mulholland
Sent: Thu 9/6/2007 7:30 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
Yeah I did, its either the wspfsrv.exe firewall service or its nothing..
!!
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Thor (Hammer of God)
Sent: Friday, 7 September 2007 12:06 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
Try "show processes for all users."
t
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Greg Mulholland
Sent: Thursday, September 06, 2007 6:14 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages - reloaded
Sorry to drag up this old chestnut but it doesn't seem to have fixed
anything as such. It seems the process wspsvr.exe is the main cpu, but
whats even weirder is that sometimes there is no real evidence of a cpu
hog in the task manager but it still sits at 100% (to prove im not
joking, attached). So I can specifically blame the firewall service as
such.
I have disable all rules relating to url and domain sets for testing. I
have added another non system drive for cache files. I have disabled all
logging. It takes almost exactly 10 minutes to apply changes. Network
service has perms for the cache drive. Filemon, procmon don't show
anything of interest. Bizzaro! Im sorta lost now short of a rebuild
which I will have a chance to do in two weeks but it bugs me. If anyone
has any more theory's im all ears.
Greg
You don't have to remove necessarily. Just disable them so ISA doesn't
have to process them while your making rule changes. Then enable those
rules at the end.
Glad it worked. But now Steve won't speak to me.
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Greg Mulholland
Sent: Wednesday, September 05, 2007 6:09 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
2 days ago I followed my gut feel and removed all the rules that used
those imported url and domain sets and it seems like it might have. I am
back at it today so I will report any changes. At least I have something
to go on.
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Gerald G. Young
Sent: Wednesday, 5 September 2007 11:02 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
Greg,
Did Amy's suggestion work?
You wouldn't happen to have a policy backup from before the last set of
banned URLs and domain sets were imported, would you? Might you be able
to restore to confirm the additions were the cause? Providing, of
course, that Amy's suggestion didn't work.
Jason,
Thanks for the clarification. Having only worked with Enterprise, I
wasn't even aware that Standard stored the config in the local Registry.
Good to know. You wouldn't happen to know the key, would you?
HKLM\Software\Microsoft\ISA Server?? Or something along those lines?
Greg,
A couple of other thoughts, although probably unrelated. How big is the
Registry on that box? How's Registry fragmentation looking?
Cordially yours,
Jerry G. Young II
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Application Engineer
Platform Engineering & Architecture
NTT America, an NTT Communications Company
=====================================
22451 Shaw Rd.
Sterling, VA 20166
Office: 571-434-1319
Fax: 703-333-6749
Email: g.young@xxxxxxxx
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
"The truth is that there is nothing noble in being superior to somebody
else.
The only real nobility is in being superior to your former self."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Greg Mulholland
Sent: Tuesday, September 04, 2007 6:35 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
Yeah I know what you're saying. Its std version, single server, no sql
involved at all. The problem was not always there but I can't yet
pinpoint what it was that changed it. As I said I think it could have
been importing a number of banned url and domain sets.
Greg
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Gerald G. Young
Sent: Wednesday, 5 September 2007 6:47 AM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
Greg,
I didn't mean the SQL Server or MSDE for logging. There is a
configuration server piece that should use SQL Server or MSDE to hold
the policies. As I understand it - and please, someone correct me if I
am wrong - applying changes submit the changes made to policies to the
configuration server. These changes are then committed to the database
in which the policy is stored. If you have an I/O problem on that
database server, that could be the issue, especially if you aren't
seeing CPU utilization spike to 100% but only a delay that feels like
it's spiked.
Is the configuration server and ISA firewall on the same box? Do you
have only one server? Is this a new server that has had this problem
since installation or did this just recently start to occur? Are you
using standard or enterprise edition?
Sorry about all the extra questions; I'm just trying to dig to help find
the scope.
Cordially yours,
Jerry G. Young II
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Application Engineer
Platform Engineering & Architecture
NTT America, an NTT Communications Company
=====================================
22451 Shaw Rd.
Sterling, VA 20166
Office: 571-434-1319
Fax: 703-333-6749
Email: g.young@xxxxxxxx
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
"The truth is that there is nothing noble in being superior to somebody
else.
The only real nobility is in being superior to your former self."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Greg Mulholland
Sent: Tuesday, September 04, 2007 4:32 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
Thanks Gerald
I was logging to flat txt but have since turn it off. The funny thing
was the last time I looked the task manager didn't show any service with
above average cpu utilization which was bizarre, hence why I never
bothered with procmon. I'll try it again in case my eyes lies.
Greg
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Gerald G. Young
Sent: Tuesday, 4 September 2007 11:52 PM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] Re: applying changes takes ages
EDITED
Oops... I lied about ProcMon showing loaded DLLs, although, it will give
you a LOT more insight on what a given process is doing. The other
piece, which DOES show DLLs loaded for a process, is Process Explorer.
That's found at the following location:
http://www.microsoft.com/technet/sysinternals/utilities/processexplorer.
mspx
-------
Previously on RE: [isapros] applying changes takes ages...
Task Manager isn't showing the name of the process that is using all CPU
resources?
If it is and you just have no insight into what that process is doing,
take a look at Process Monitor from Sysinternals/Microsoft. It provides
a lot more detail about what a process is doing, even going as far as
showing which DLLs are loaded for it. This tool can be found at the URL
below.
http://www.microsoft.com/technet/sysinternals/utilities/processmonitor.m
spx
A couple of other questions. Where is the database sitting? Is it SQL
Server of MSDE? If it's on a different server, what does its
performance counters look like? If SQL Server (whether local or
remote), have you checked SQL logs and locks? Anything at all in the
Event Logs on the ISA box?
Cordially yours,
Jerry G. Young II
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Application Engineer
Platform Engineering & Architecture
NTT America, an NTT Communications Company
=====================================
22451 Shaw Rd.
Sterling, VA 20166
Office: 571-434-1319
Fax: 703-333-6749
Email: g.young@xxxxxxxx
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
"The truth is that there is nothing noble in being superior to somebody
else.
The only real nobility is in being superior to your former self."
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
From: isapros-bounce@xxxxxxxxxxxxx [mailto:isapros-bounce@xxxxxxxxxxxxx]
On Behalf Of Greg Mulholland
Sent: Tuesday, September 04, 2007 6:28 AM
To: isapros@xxxxxxxxxxxxx
Subject: [isapros] applying changes takes ages
Hey gang got an issue here I'm trying to work through.
I have an isa 2006 std box running in firewall mode that takes an
exorbitant amount of time to apply any changes. Like I'm talking 5-10
minutes.
A little background:
It's a basic win2k3 sp2 install with around 20 rules (least privilege),
a few published services like smtp,web, owa, VPN clients (pptp). I read
about an issue on multi core processors but this is pure single core. I
also read about when a number of web servers are published with multiple
link translations it can take time, but we aren't using any link
translation atm. The server seems to go CPU bound in a big way when I
apply any rule or config changes, sits around 100%. Incidentally there
are no simultaneous or subsequent ISA alters nor are there any system or
app log events fired.
I'm not sure what's pinning the CPU, the task manager doesn't give me
any real heads up on anything, the box at load is sitting around
300-500mb ram free, we've recently added a new disk for logging but in
this case I've even turned off logging for the time being to try and get
to the bottom of the performance. My gut feeling was that there was a
bad rule or something in the ruleset but I've reviewed these and they
seem to be OK. I have run an ISABPA and didn't find anything more than
the usual you are using strict rpc, and a few other red herrings. I
haven't as yet ran a sniff whilst the changes are being applied but I
would have assumed that would be somewhat fruitless anyway.
Can anyone shed any more light or give me any pov's.
Thanks
Greg
=================================
This email message is intended for the use of the person to whom it has
been sent, and may contain information that is confidential or legally
protected. If you are not the intended recipient or have received this
message in error, you are not authorized to copy, distribute, or
otherwise use this message or its attachments. Please notify the sender
immediately by return e-mail and permanently delete this message and any
attachments. NTT America makes no warranty that this email is error or
virus free. Thank you.
=================================
This email message is intended for the use of the person to whom it has
been sent, and may contain information that is confidential or legally
protected. If you are not the intended recipient or have received this
message in error, you are not authorized to copy, distribute, or
otherwise use this message or its attachments. Please notify the sender
immediately by return e-mail and permanently delete this message and any
attachments. NTT America makes no warranty that this email is error or
virus free. Thank you.
=================================
This email message is intended for the use of the person to whom it has
been sent, and may contain information that is confidential or legally
protected. If you are not the intended recipient or have received this
message in error, you are not authorized to copy, distribute, or
otherwise use this message or its attachments. Please notify the sender
immediately by return e-mail and permanently delete this message and any
attachments. NTT America makes no warranty that this email is error or
virus free. Thank you.
All mail to and from this domain is GFI-scanned.
=================================
This email message is intended for the use of the person to whom it has been
sent, and may contain information that is confidential or legally protected. If
you are not the intended recipient or have received this message in error, you
are not authorized to copy, distribute, or otherwise use this message or its
attachments. Please notify the sender immediately by return e-mail and
permanently delete this message and any attachments. NTT America makes no
warranty that this email is error or virus free. Thank you.
Other related posts:
