Another novel from Rick...Good lad... Is this the longest post so far this year? J From: thin-bounce@xxxxxxxxxxxxx [mailto:thin-bounce@xxxxxxxxxxxxx] On Behalf Of Rick Mack Sent: 02 February 2007 20:53 To: thin@xxxxxxxxxxxxx Subject: [THIN] Re: Printer Hell Hi Mike, The main attraction of third party printing solutions is that they sharrply restrict the number of printer drivers you've got on your systems. Aside from that, they aren't any better or worse than what Citrix already have for you. Now that Citrix have got most of the bugs out of their UPD solution, the one real weakness left is the support for session network printers and the need to support the odd application that does it own thing with regard to printing and just passes a raw spool job to the spooler subsystem. I can turn a "good" scenario into a bad one just by ticking a single box in the CMC, the one that controls if the client printer is a network printer and I know about it, create a network printer in the session instead of printing via the ICA client (and UPD). And it doesn't matter if I've got a trusted printer driver source or anything else because the trusted source probably has bad drivers. Supporting network printers means installing drivers, and with HP at the top of the list, there are some really, really bad printer drivers out there (see below). They work okay in a single user environment, but just don't handle a multi-user environment terribly well and will hang or crash the spooler subsystem and the Citrix print manager service. Symptoms are handle leaks in the cpsvc, login hangs running cmstart, all sorts of printer creation and assignment problems, a crashed spooler, failed RPC service, toasted servers. Great stuff. And do third party solution solve this problem? They sure do. Toss out all the bad drivers, don't use network printers, only use a UPD and printing is stable and reliable. Move the whole overhead for printing on to a print server (eg Thinprint VC gateway) and life is wonderful. But that doesn't mean you have to go for third party printing solutions. Believe it or not, but you can actually make your Citrix printing reliable as well. Even when things have gone to hell. One of the things I do to "tune" our sites is to apply an unmanaged group policy that does something really simple, it stops RDP printer autocreation when admins log on to servers. Because if you don't do that, and you haven't disabled printer remapping in the RDP client configuration, then RDP printer autocreation can suck the printer drivers off your workstation, laptop or home computer and install them on your servers. Every time you attach to a network print queue because you want to print something while logged on as an admin, you suck drivers across on to your servers. Create a Citrix policy as well for admins only that prevents printer autocreation. Because admins are the most likely source of rogue drivers. Most of the default printer driver restrictions apply to non-admins so if things go bad, guess who's responsible? Probably you or me, the admins. Not Citrix. Anyway, I hope I've made the point about who is most often guilty when things go bad. But stay tuned 'cause printer hell isn't all our fault. As an aside, the other thing you HAVE to do if you're supporting network printers, is to give users write access to the %systemroot%\system32\spool\drivers\w32x86\3 folder. If you don't, introducung new (mostly HP) printer drivers to your farm is going to cause all sorts of interesting problems with other drivers, even when they're "good" drivers. It'd take too long to explain but you really need to do this. Of course if you set things up to just use the UPD (we're talking PS4 UPD here), you've just avoided a whole stack of problems, even if it isn't the most efficient way to go. Anyway, I'd better finish this up. Since Citrix released AddPrinter, we've been cleaning up printing problems on all our customer sites. All AddPrinter does is create a bunch of printer queues using the driver being tested. It simulates a worst case scenario where you have a few users creating a printer at the same time. No rocket science but it's a real life saver becauae it'll show you what's causing your printer hell. Most of the time it's HP. I had a lot of issues with HP printer drivers a few years ago, (remember the Stop 0x1E and stop 0x50 bluescreens?) and got to know the guys at HP pretty well. They really cleaned up their act and I guess I just assumed, stupidly, that if it was HP it was terminal server friendly. I used to bag Xerox because they were using so many kernel mode drivers, but it's not Xerox on my hit list these days because it's been a real eye opener to see just how bad some of the HP drivers are. If you try testing the HP LaserJet 8150 driver with AddPrinter, as an example, it'll crash the spooler almost instantly. And that's not the worst one because that's a clean death and only effects the spooler. If you set the spooler to auto restart, you hardly notice. Of course there are a lot of instances of HPBOID.exe running on the server but at least it's not really broken. It's the HP drivers that cause memory leaks and memory corruption that really hurt because they'll eventually toast your server. Check out CTX111947 (Intermittent Client Printer Autocreation Failures) for what I now consider to be a partial list of the bad drivers: * HP Color LaserJet 4600 * HP LaserJet 4200 * HP LaserJet 4300 * HP Color LaserJet 4550 * HP Color LaserJet 5500 * HP Color LaserJet 8550 * HP LaserJet 4100 * HP LaserJet 5100 * HP LaserJet 8150 * HP LaserJet 9000 * HP Color LaserJet 2500 * HP LaserJet 2300 * HP Color LaserJet 5550 * HP Color LaserJet 9500 * HP Color LaserJet 9500 MFP * HP LaserJet 2410/2420/2430 * HP LaserJet 4250 * HP LaserJet 4345 mfp * HP LaserJet 4350 * HP LaserJet 9050 * HP LaserJet 9050/9040 MFP * HP LaserJet 9055 MFP * HP LaserJet 9065 MFP * HP Color LaserJet 2800 Series * HP Color LaserJet 3000 * HP Color LaserJet 3600 * HP Color LaserJet 4700 * HP Color LaserJet 4730 mfp * HP ColorLaserJet 3800 * HP LaserJet 5200/5200L * HP Color LaserJet 3700 * HP Color LaserJet 4650 * HP Color LaserJet 3550 There are more. We've gone to the trouble of cleaning up all the bad drivers in a farm, and something marvellous has happened. Printing just works reliably. Testing all your printer drivers is easy. But a word of warning, don't try using AddPrinter on a production server with users logged on. The experience won't win you any friends. All you need is some way of listing all the printer drivers on a server. The following VB script is a pretty simple way to do it. ------ start listdrivers.vbs ------- ' listdrivers.vbs - list all installed printer drivers DIM DrvArray strComputer = "." Set objWMIService = GetObject("winmgmts:" _ & "{impersonationLevel = impersonate}!\\" & strComputer & "\root\cimv2") Set colInstalledPrinters = objWMIService.ExecQuery _ ("Select * from Win32_PrinterDriver") For each objPrinter in colInstalledPrinters DrvArray = split(objPrinter.Name,",") Wscript.Echo DrvArray(0) Next ----- end listdrivers.vbs ------- Run "vbscript //nologo listdrivers.vbs > drivers.txt" and you've got a list of drivers. A single command will now do your testing: for /f "delims=:" %i in (drivers.lst) do @echo %i >> addprinters.log & addprinter - name test -driver "%i" -conc 5 -iter 10 - port lpt1: -delay 500 | find /v "%%" >> addprinters.log 2>&1 Note: If you cut and paste this, the quotes ["] may not come across properly and the script will generate an addprinter error. Replace the quotes and it will work. Things aren't completely straightforward because some of the drivers will crash the spooler, so you'll have to shepherd the testing and restart the spooler (or the server) and edit drivers.txt to restart the testing past the bad driver. It'll take some time as well. When you look at the log, thread wait errors mean the driver is probably okay (but why risk it) and hard errors mean get rid of that driver. RPC server errors mean the previous driver crashed the spooler and you'll have to restart with that driver removed from drivers.txt. At the end of the execise you'll have a list of bad, suspect and passed (I almost said good, but that's be lying) drivers. On average we've had to replace about 10-15% of the drivers being used in a farm. You have to start with upgrading/modifying the printer driver on the file server(s) and then propagate the updated driver to your terminal servers. Once you've finished cleaning up the printer drivers, your problems are nearly over. You've still got the occasional orphaned print job (print job queued to autocreated client printer on client no longer conected) that will cause problems, but you've just gotten rid of the bulk of your printing problems. regards, Rick -- Ulrich Mack Commander Australia On 2/2/07, Mike MacDonald <mike5287@xxxxxxxxx> wrote: I know I am not the first one here to be in printer hell, but recently things have taken a turn for the worse. Something changed that caused drivers to get loaded on our W2K/PS4 servers from clients. This even though driver installation was restricted to administrators, only allowed from a single trusted source and our PS4 policies said that client printers would only use native drivers if they were available. Long story short is all of a sudden drivers started installing - we got up to 135 drivers on each of the 20 servers in the farm. Along with the high number of drivers we started having frequent spool service/Citrix Print service hangs or crashes which has resulted in logon delays as printers try to get connected at logon. Currently I am going through each server and eliminating unneeded drivers. I was able to go from 135 down to 53 drivers per server. Still too many, but there are that many different models of network printers here. I also renamed the ntprint.inf to make sure no additional drivers got installed without me putting them there. After I get rid of the extra drivers I am going to go through the remaining drivers on the print server and the Citrix servers to make sure they are compatible. If they are not I will change the driver used for the device to one that is compatible. One last bit of background is that we use a mix of client printers and network printers that logon scripts connect users to at logon and most users connect to a published desktop from thin-clients or via the Web Interface. I guess I am looking for any suggestions on how to get out of printer hell. I am ready to build new servers running W2K3/PS4 but don't want to move the same problems forward to the new servers. I also feel I must be doing something wrong related to the Citrix Print Service because I can restart it and the spool service then instantly try to stop it and it won't stop - has to be killed. Is there something else I should be doing or has anyone had similair issues, and what was done to correct the problems. I am going to go through cleaning up drivers as mentioned, but if I need to do something else I am all ears! Thanks in advance for any suggestions or assistance! -Mike MacDonald