[THIN] Re: How I Killed IE and Outlook on our Citrix Farm by triggering a Nasty WMI Bug
- From: Martin Stephenson <mwstephenson@xxxxxxxxx>
- To: thin@xxxxxxxxxxxxx
- Date: Thu, 18 Nov 2004 07:21:29 +1100 (EST)
Microsoft are definitely aware of this bug and there have been 5 revisions of
the hotfix. It
seems to be a little tricky to re-create which may be why most people don't
stumble across it.
I've posted it here as I think that Citrix servers are the one place where this
bug can really
wipe you out (even if the bug was present on Win XP, how many people run
Performance monitor
against all of their XP Workstations simultaneously?). I was particularily
caught out because
even though I thought I was only monitoring 2 servers I was actually impacting
most servers in the
Farm. What I had done was to export the Perf Log config, modify it for each
server and re-import
it into the Logging tool. Even though I didnt Start the logging for each
server, the fact that I
had loaded the config into the MMC tool was enough to trigger the bug.
The other reason this bug catches you out is that your totally unaware of the
connection between
you running Perf Logging on your PC and the sudden breaking of WSH, IE and
Outlook on several
servers. It took about 20 hrs for the nature and full scale of the problem to
be escalated to me.
One of the first things I did was to close the Perf Logging tool, only to
discover later that it
was still logging in the background (I had set the scheduling feature), and
even when MMC Logging
really is totally killed, the problem can continue to occur on the servers for
several hours.
The way I was able to pinpoint the problem in the end was to create a
diagnostic VB script, that
ran in an endless loop in a command prompt published from each server. That
way I could observe
in real time when the issue occured. As soon as I started up Perf Logging
again, sure enough my
VB script stopped and I knew I had found the culprit - Me!
The other reason I posted here is that I have been a member of this list, on
and off for several
years. If somebody else wants to cross post this to other lists, then by all
means go ahead.
Regards,
Martin.
mwstephenson@xxxxxxxxx
--- Amer Karim <amerk@xxxxxxxxxxxxxxxx> wrote:
> I'm just wondering if you shouldn't also be posting this on the bugtraq
> list - though if MS already has a hot fix for it, then someone may
> already have done so at some point. Generally, though, MS has people
> monitoring those lists and if it generates enough noise there, they may
> end up making the patch public.
>
> Regards,
> Amer Karim
> Nautilis Information Systems
> e-mail: amerk@xxxxxxxxxxxxxxxx
> www.nautilis-sys.com
>
>
> -----Original Message-----
> From: thin-bounce@xxxxxxxxxxxxx [mailto:thin-bounce@xxxxxxxxxxxxx] On
> Behalf Of Martin Stephenson
> Sent: November 17, 2004 12:46 AM
> To: Thin List
> Subject: [THIN] How I Killed IE and Outlook on our Citrix Farm by
> triggering a Nasty WMI Bug
>
> If your running Windows 2000 Terminal Services you may want to read
> this, especially if you have
> been vigilant and installed your Security patches, specifically MS04-011
> (KB835732) or you have
> service Pack 4 installed.
>
> What did I break?
> =================
> By accidentally triggering this serious WMI bug, I managed to break
> Internet Explorer (5.5 SP2),
> Outlook 2003, Windows Scripting Host 5.6 and Add/Remove programs. IE
> hangs on a white screen,
> Outlook will occasionally start in Safe mode or just not at all and the
> WSH failure caused our VB
> Script based login scripts to hang. All 4 applications hung
> simultaneously and the effect is
> instantaneous and can in some circumstances be quite prolonged, hours or
> days.
>
> How to Cause the Outage
> =======================
> >From your Windows XP SP1 PC, open MMC and add in the snap-in called
> Performance Logs and Alerts.
> Then add in several counters against a W2K Terminal Server. Set the
> sample data interval to 15
> secs.
>
> I used the following counters:
>
> Memory\Available Bytes
> Memory\Page Faults/sec
> Memory\Pages/sec
> Memory\Write Copies/sec
> PhysicalDisk(0 C:)\% Disk Time
> PhysicalDisk(0 C:)\Current Disk Queue Length
> Processor(_Total)\% Processor Time
> Processor(_Total)\Interrupts/sec
> Terminal Services\Active Sessions
>
> Now just by doing this could be enough to trigger the WMI bug. You dont
> even need to activate the
> logging by pressing the play button. This is because when you add in
> the counters it queries the
> counters from the registry on the W2K server.
>
> Note: Before you can even get XP's Performance Logs and Alerts to work
> against remote servers you
> need to reconfigure the Performane Logs and Alerts service on your
> Windows XP PC, so that it runs
> under an account that has rights to view the performance counters in the
> registry of your W2K
> server. An account which has local Admin rights on the server is
> usually adequate, you can of
> course set specific ACLs on the appropriate registry key of the server.
>
> Once you have entered in the credentials of an appropriate account,
> start the Performance Logs and
> Alerts service. It gives you a couple of messages, the first indicating
> that that account has
> been given the logon as service right and the second saying something
> like the service started
> then stopped because it was not needed at this time (I'm not quoting
> this). The reason it gives
> you this second message is that the service is only started by the
> Performance Logs and Alerts
> snap-in. It also helps if you have the Performance Logs and Alerts MMC
> console closed *before* you
> change the credentials on the service.
>
> Quick Fix
> =========
> The quick fix to this WMI bug is to restart the "Remote Registry
> Service" on the affected W2K
> server. However as soon as you load up Performance logging again you
> can trigger the bug. Be
> aware that if you set the Performance Logging to operate on a scheduled
> basis it will continue to
> run in the background even after you have closed the MMC console.
>
> Permanent Fix
> =============
> The permanent fix the the WMI bug is Hotfix - 834010. More info on this
> hot fix can be found at
> http://support.microsoft.com/default.aspx?scid=kb;en-us;833974
>
> and
>
> http://support.microsoft.com/default.aspx?scid=kb;en-us;834010
>
> Disclaimer
> ==========
> If you follow my directions and break your production environment I
> won't take responsibility :-).
> So I recommend you only try this out on a test environment and if you
> do break it get hold of the
> hotfix directly from Microsoft.
>
> While I do have the Hotfix and could email it to you, this would short
> circuit Microsoft and they
> would continue to think that this is an issue that only affects a very
> few customers.
>
> While this problem was easily repeatable on our Production and existing
> Test servers, I had mixed
> results on subsequent testing with freshly built test servers. The new
> Test servers were not
> based on previous images of existing Citrix servers but were built by
> hand and had SP3, MS04-011,
> IE 5.5 SP2, WSH 5.6, eTrust Anti Virus 7 and all of the latest Critical
> and Security Patches
> installed. I was able to repeat the problem on one test server but not
> another. Neither of
> my freshly built Test servers had an Citrix software installed but I did
> put both of them into TS
> Application mode, although I suspect the WMI bug would still be
> triggered in Terminal Services was
> not installed.
>
> Now it is possible that there is something unusual about our environment
> that doesnt exist in many
> others or it depends on the exact order that you installed IE 5.5, WSH
> 5.6 and MS04-011.
>
> So if it does affect your environment please voice your concern to
> Microsoft and if enough people
> are affected by this it should encourage them to release this as a
> Critical Patch.
>
> Regards,
> Martin Stephenson.
> Capital & Coast District Health (for 2 more days anyway!)
>
> Find local movie times and trailers on Yahoo! Movies.
> http://au.movies.yahoo.com
> ********************************************************
> This Weeks Sponsor Emergent Online ThinCity Conference
> Join us at ThinCity 2004: The 1st Annual Emergent OnLine Technology
> Conference
> http://www.ThinCity.com
> **********************************************************
> Useful Thin Client Computing Links are available at:
> http://thin.net/links.cfm
> ***********************************************************
> For Archives, to Unsubscribe, Subscribe or
> set Digest or Vacation mode use the below link:
> http://thin.net/citrixlist.cfm
>
>
>
> ********************************************************
> This Weeks Sponsor Emergent Online ThinCity Conference
> Join us at ThinCity 2004: The 1st Annual Emergent OnLine Technology Conference
> http://www.ThinCity.com
> **********************************************************
> Useful Thin Client Computing Links are available at:
> http://thin.net/links.cfm
> ***********************************************************
> For Archives, to Unsubscribe, Subscribe or
> set Digest or Vacation mode use the below link:
> http://thin.net/citrixlist.cfm
>
Find local movie times and trailers on Yahoo! Movies.
http://au.movies.yahoo.com
********************************************************
This Weeks Sponsor Emergent Online ThinCity Conference
Join us at ThinCity 2004: The 1st Annual Emergent OnLine Technology Conference
http://www.ThinCity.com
**********************************************************
Useful Thin Client Computing Links are available at:
http://thin.net/links.cfm
***********************************************************
For Archives, to Unsubscribe, Subscribe or
set Digest or Vacation mode use the below link:
http://thin.net/citrixlist.cfm
Other related posts:
