[THIN] Re: Printer Hell

  • From: "Roger Riggins" <roger.riggins@xxxxxxxxxx>
  • To: <thin@xxxxxxxxxxxxx>
  • Date: Fri, 2 Feb 2007 16:40:25 -0600

We use several of those drivers in production and we don't have any
printing issues <knock on wood>. We probably don't get more than two or
three jobs queued on any of our printers, though. We have approx 75
network printers and use native and UPD as much as possible. I didn't
know about addprinter until your post, but I'll throw some drivers at it
to see what happens. It'll be interesting to see at what point they
break. Thanks for the info.


-----Original Message-----
From: thin-bounce@xxxxxxxxxxxxx [mailto:thin-bounce@xxxxxxxxxxxxx] On
Behalf Of Joe Shonk
Sent: Friday, February 02, 2007 3:48 PM
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Printer Hell

I noticed that ALL of the bad drivers are HPs.  You'd think they'd
eventually get their crap together.  And this is a company that brags
that they have a full time engineer at Citrix.  It's time they fired
this person (and a few other developers) and get someone else in there
who can better QA drivers.


From: thin-bounce@xxxxxxxxxxxxx [mailto:thin-bounce@xxxxxxxxxxxxx] On
Behalf Of Rick Mack
Sent: Friday, February 02, 2007 1:53 PM
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Printer Hell

Hi Mike,

The main attraction of third party printing solutions is that they
sharrply restrict the number of printer drivers you've got on your
systems. Aside from that, they aren't any better or worse than what
Citrix already have for you.

Now that Citrix have got most of the bugs out of their UPD solution, the
one real weakness left is the support for session network printers and
the need to support the odd application that does it own thing with
regard to printing and just passes a raw spool job to the spooler

I can turn a "good" scenario into a bad one just by ticking a single box
in the CMC, the one that controls if the client printer is a network
printer and I know about it, create a network printer in the session
instead of printing via the ICA client (and UPD). And it doesn't matter
if I've got a trusted printer driver source or anything else because the
trusted source probably has bad drivers.

Supporting network printers means installing drivers, and with HP at the
top of the list, there are some really, really bad printer drivers out
there (see below). They work okay in a single user environment, but just
don't handle a multi-user environment terribly well and will hang or
crash the spooler subsystem and the Citrix print manager service.

Symptoms are handle leaks in the cpsvc, login hangs running cmstart, all
sorts of printer creation and assignment problems, a crashed spooler,
failed RPC service, toasted servers. Great stuff.

And do third party solution solve this problem? They sure do.

Toss out all the bad drivers, don't use network printers, only use a UPD
and printing is stable and reliable. Move the whole overhead for
printing on to a print server (eg Thinprint VC gateway) and life is

But that doesn't mean you have to go for third party printing solutions.
Believe it or not, but you can actually make your Citrix printing
reliable as well. Even when things have gone to hell.

One of the things I do to "tune" our sites is to apply an unmanaged
group policy that does something really simple, it stops RDP printer
autocreation when admins log on to servers. Because if you don't do
that, and you haven't disabled printer remapping in the RDP client
configuration, then RDP printer autocreation can suck the printer
drivers off your workstation, laptop or home computer and install them
on your servers. Every time you attach to a network print queue because
you want to print something while logged on as an admin, you suck
drivers across on to your servers.

Create a Citrix policy as well for admins only that prevents printer
autocreation. Because admins are the most likely source of rogue
drivers. Most of the default printer driver restrictions apply to
non-admins so if things go bad, guess who's responsible? Probably you or
me, the admins. Not Citrix.

Anyway, I hope I've made the point about who is most often guilty when
things go bad. But stay tuned 'cause printer hell isn't all our fault.

As an aside, the other thing you HAVE to do if you're supporting network
printers, is to give users write access to the
%systemroot%\system32\spool\drivers\w32x86\3 folder. If you don't,
introducung new (mostly HP) printer drivers to your farm is going to
cause all sorts of interesting problems with other drivers, even when
they're "good" drivers. It'd take too long to explain but you really
need to do this.

Of course if you set things up to just use the UPD (we're talking PS4
UPD here), you've just avoided a whole stack of problems, even if it
isn't the most efficient way to go.

Anyway, I'd better finish this up.

Since Citrix released AddPrinter, we've been cleaning up printing
problems on all our customer sites. All AddPrinter does is create a
bunch of printer queues using the driver being tested. It simulates a
worst case scenario where you have a few users creating a printer at the
same time. No rocket science but it's a real life saver becauae it'll
show you what's causing your printer hell.

Most of the time it's HP.

I had a lot of issues with HP printer drivers a few years ago, (remember
the Stop 0x1E and stop 0x50 bluescreens?)  and got to know the guys at
HP pretty well. They really cleaned up their act and I guess I just
assumed, stupidly, that if it was HP it was terminal server friendly. I
used to bag Xerox because they were using so many kernel mode drivers,
but it's not Xerox on my hit list these days because it's been a real
eye opener to see just how bad some of the HP drivers are.

If you try testing the HP LaserJet 8150 driver with AddPrinter, as an
example, it'll crash the spooler almost instantly. And that's not the
worst one because that's a clean death and only effects the spooler. If
you set the spooler to auto restart, you hardly notice. Of course there
are a lot of instances of HPBOID.exe running on the server but at least
it's not really broken. It's the HP drivers that cause memory leaks and
memory corruption that really hurt because they'll eventually toast your

Check out CTX111947 (Intermittent Client Printer Autocreation Failures)
for what I now consider to be a partial list of the bad drivers:

* HP Color LaserJet 4600

* HP LaserJet 4200

* HP LaserJet 4300

* HP Color LaserJet 4550

* HP Color LaserJet 5500

* HP Color LaserJet 8550

* HP LaserJet 4100

* HP LaserJet 5100

* HP LaserJet 8150

* HP LaserJet 9000

* HP Color LaserJet 2500

* HP LaserJet 2300

* HP Color LaserJet 5550

* HP Color LaserJet 9500

* HP Color LaserJet 9500 MFP

* HP LaserJet 2410/2420/2430

* HP LaserJet 4250

* HP LaserJet 4345 mfp

* HP LaserJet 4350

* HP LaserJet 9050

* HP LaserJet 9050/9040 MFP

* HP LaserJet 9055 MFP

* HP LaserJet 9065 MFP

* HP Color LaserJet 2800 Series

* HP Color LaserJet 3000

* HP Color LaserJet 3600

* HP Color LaserJet 4700

* HP Color LaserJet 4730 mfp

* HP ColorLaserJet 3800

* HP LaserJet 5200/5200L

* HP Color LaserJet 3700

* HP Color LaserJet 4650

* HP Color LaserJet 3550

There are more.

We've gone to the trouble of cleaning up all the bad drivers in a farm,
and something marvellous has happened. Printing just works reliably.

Testing all your printer drivers is easy. But a word of warning, don't
try using AddPrinter on a production server with users logged on. The
experience won't win you any friends.

All you need is some way of listing all the printer drivers on a server.
The following VB script is a pretty simple way to do it.

------ start listdrivers.vbs -------
' listdrivers.vbs - list all installed printer drivers
DIM DrvArray

strComputer = "."
Set objWMIService = GetObject("winmgmts:" _
    & "{impersonationLevel = impersonate}!\\" & strComputer &

Set colInstalledPrinters =  objWMIService.ExecQuery _
    ("Select * from Win32_PrinterDriver")

For each objPrinter in colInstalledPrinters
    DrvArray = split(objPrinter.Name,",")
    Wscript.Echo DrvArray(0)
----- end listdrivers.vbs -------

Run "vbscript //nologo listdrivers.vbs > drivers.txt" and you've got a
list of drivers.

A single command will now do your testing:

for /f "delims=:" %i in (drivers.lst) do @echo %i >> addprinters.log &
addprinter - name test -driver "%i" -conc 5 -iter 10 - port lpt1: -delay
500 | find /v "%%" >> addprinters.log 2>&1

Note: If you cut and paste this, the quotes ["] may not come across
properly and the script will generate an addprinter error. Replace the
quotes and it will work.

Things aren't completely straightforward because some of the drivers
will crash the spooler, so you'll have to shepherd the testing and
restart the spooler (or the server) and edit drivers.txt to restart the
testing past the bad driver. It'll take some time as well.

When you look at the log, thread wait errors mean the driver is probably
okay (but why risk it) and hard errors mean get rid of that driver. RPC
server errors mean the previous driver crashed the spooler and you'll
have to restart with that driver removed from drivers.txt. At the end of
the execise you'll have a list of bad, suspect and passed (I almost said
good, but that's be lying) drivers.

On average we've had to replace about 10-15% of the drivers being used
in a farm. You have to start with upgrading/modifying the printer driver
on the file server(s) and then propagate the updated driver to your
terminal servers. Once you've finished cleaning up the printer drivers,
your problems are nearly over.

You've still got the occasional orphaned print job (print job queued to
autocreated client printer on client no longer conected) that will cause
problems, but you've just gotten rid of the bulk of your printing



Ulrich Mack
Commander Australia

On 2/2/07, Mike MacDonald <mike5287@xxxxxxxxx> wrote:

        I know I am not the first one here to be in printer hell, but
recently things have taken a turn for the worse. Something changed that
caused drivers to get loaded on our W2K/PS4 servers from clients. This
even though driver installation was restricted to administrators, only
allowed from a single trusted source and our PS4 policies said that
client printers would only use native drivers if they were available.
Long story short is all of a sudden drivers started installing - we got
up to 135 drivers on each of the 20 servers in the farm. Along with the
high number of drivers we started having frequent spool service/Citrix
Print service hangs or crashes which has resulted in logon delays as
printers try to get connected at logon.

        Currently I am going through each server and eliminating
unneeded drivers. I was able to go from 135 down to 53 drivers per
server. Still too many, but there are that many different models of
network printers here. I also renamed the ntprint.inf to make sure no
additional drivers got installed without me putting them there. After I
get rid of the extra drivers I am going to go through the remaining
drivers on the print server and the Citrix servers to make sure they are
compatible. If they are not I will change the driver used for the device
to one that is compatible.

        One last bit of background is that we use a mix of client
printers and network printers that logon scripts connect users to at
logon and most users connect to a published desktop from thin-clients or
via the Web Interface.

        I guess I am looking for any suggestions on how to get out of
printer hell. I am ready to build new servers running W2K3/PS4 but don't
want to move the same problems forward to the new servers. I also feel I
must be doing something wrong related to the Citrix Print Service
because I can restart it and the spool service then instantly try to
stop it and it won't stop  - has to be killed. Is there something else I
should be doing or has anyone had similair issues, and what was done to
correct the problems. I am going to go through cleaning up drivers as
mentioned, but if I need to do something else I am all ears!

        Thanks in advance for any suggestions or assistance!

        -Mike MacDonald

<b>Lutheran Services in Iowa Confidentiality Notice:</b><br>
<red>The information contained in this communication may be confidential,
is intended only for the use of the recipient(s) named above, and
may be legally privileged. If the reader of this message is not the
intended recipient, you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of its
contents, is strictly prohibited. If you have received this
communication in error, please return it to the sender immediately
and delete the original message and any copy of it from your computer
system. If you have any questions concerning this message, please
contact the sender.</red>

Other related posts: