[THIN] How I Killed IE and Outlook on our Citrix Farm by triggering a Nasty WMI Bug

  • From: Martin Stephenson <mwstephenson@xxxxxxxxx>
  • To: Thin List <thin@xxxxxxxxxxxxx>
  • Date: Wed, 17 Nov 2004 19:30:03 +1100 (EST)

If your running Windows 2000 Terminal Services you may want to read this, 
especially if you have
been vigilant and installed your Security patches, specifically MS04-011 
(KB835732) or you have
service Pack 4 installed.

What did I break?
=================
By accidentally triggering this serious WMI bug, I managed to break Internet 
Explorer (5.5 SP2),
Outlook 2003, Windows Scripting Host 5.6 and Add/Remove programs.  IE hangs on 
a white screen,
Outlook will occasionally start in Safe mode or just not at all and the WSH 
failure caused our VB
Script based login scripts to hang.  All 4 applications hung simultaneously and 
the effect is
instantaneous and can in some circumstances be quite prolonged, hours or days.

How to Cause the Outage
=======================
From your Windows XP SP1 PC, open MMC and add in the snap-in called Performance 
Logs and Alerts. 
Then add in several counters against a W2K Terminal Server. Set the sample data 
interval to 15
secs.  

I used the following counters:

        Memory\Available Bytes
        Memory\Page Faults/sec
        Memory\Pages/sec
        Memory\Write Copies/sec
        PhysicalDisk(0 C:)\% Disk Time
        PhysicalDisk(0 C:)\Current Disk Queue Length
        Processor(_Total)\% Processor Time
        Processor(_Total)\Interrupts/sec
        Terminal Services\Active Sessions

Now just by doing this could be enough to trigger the WMI bug.  You dont even 
need to activate the
logging by pressing the play button.  This is because when you add in the 
counters it queries the
counters from the registry on the W2K server.

Note: Before you can even get XP's Performance Logs and Alerts to work against 
remote servers you
need to reconfigure the Performane Logs and Alerts service on your Windows XP 
PC, so that it runs
under an account that has rights to view the performance counters in the 
registry of your W2K
server.  An account which has local Admin rights on the server is usually 
adequate, you can of
course set specific ACLs on the appropriate registry key of the server.

Once you have entered in the credentials of an appropriate account, start the 
Performance Logs and
Alerts service.  It gives you a couple of messages, the first indicating that 
that account has
been given the logon as service right and the second saying something like the 
service started
then stopped because it was not needed at this time (I'm not quoting this).  
The reason it gives
you this second message is that the service is only started by the Performance 
Logs and Alerts
snap-in. It also helps if you have the Performance Logs and Alerts MMC console 
closed *before* you
change the credentials on the service.

Quick Fix
=========
The quick fix to this WMI bug is to restart the "Remote Registry Service" on 
the affected W2K
server.  However as soon as you load up Performance logging again you can 
trigger the bug.  Be
aware that if you set the Performance Logging to operate on a scheduled basis 
it will continue to
run in the background even after you have closed the MMC console.

Permanent Fix
=============
The permanent fix the the WMI bug is Hotfix - 834010.  More info on this hot 
fix can be found at
http://support.microsoft.com/default.aspx?scid=kb;en-us;833974

and

http://support.microsoft.com/default.aspx?scid=kb;en-us;834010

Disclaimer
==========
If you follow my directions and break your production environment I won't take 
responsibility :-).
 So I recommend you only try this out on a test environment and if you do break 
it get hold of the
hotfix directly from Microsoft.

While I do have the Hotfix and could email it to you, this would short circuit 
Microsoft and they
would continue to think that this is an issue that only affects a very few 
customers.

While this problem was easily repeatable on our Production and existing Test 
servers, I had mixed 
results on subsequent testing with freshly built test servers.  The new Test 
servers were not
based on previous images of existing Citrix servers but were built by hand and 
had SP3, MS04-011,
IE 5.5 SP2, WSH 5.6, eTrust Anti Virus 7 and all of the latest Critical and 
Security Patches
installed.  I was able to repeat the problem on one test server but not 
another.  Neither of 
my freshly built Test servers had an Citrix software installed but I did put 
both of them into TS 
Application mode, although I suspect the WMI bug would still be triggered in 
Terminal Services was
not installed.

Now it is possible that there is something unusual about our environment that 
doesnt exist in many
others or it depends on the exact order that you installed IE 5.5, WSH 5.6 and 
MS04-011.  

So if it does affect your environment please voice your concern to Microsoft 
and if enough people
are affected by this it should encourage them to release this as a Critical 
Patch.

Regards,
Martin Stephenson.
Capital & Coast District Health (for 2 more days anyway!)

Find local movie times and trailers on Yahoo! Movies.
http://au.movies.yahoo.com
********************************************************
This Weeks Sponsor Emergent Online ThinCity Conference
Join us at ThinCity 2004: The 1st Annual Emergent OnLine Technology Conference
http://www.ThinCity.com
********************************************************** 
Useful Thin Client Computing Links are available at:
http://thin.net/links.cfm
***********************************************************
For Archives, to Unsubscribe, Subscribe or 
set Digest or Vacation mode use the below link:
http://thin.net/citrixlist.cfm

Other related posts: