Eh, that was only a bit longer than usual. J Good post though. I'll remember to reference this one for future "Roaming Profile" questions. Jamie Nelson | Operations Consultant | BI&T Infrastructure-Intel | Devon Energy Corporation | Work: 405.552.8054 | Mobile: 405.200.8088 | http://www.dvn.com <http://www.dvn.com/> From: gptalk-bounce@xxxxxxxxxxxxx [mailto:gptalk-bounce@xxxxxxxxxxxxx] On Behalf Of Cruz, Jerome L Sent: Thursday, October 16, 2008 9:29 PM To: gptalk@xxxxxxxxxxxxx Subject: [gptalk] Re: Do I need a custom adm, and/or where is the setting in the regsitry Hi Booker, My answer is going to be a bit (okay, maybe a LOT) longer than the usual. Hopefully others will see the things you can run into with roaming profiles and not just assume the technology and it's support system are a breeze to implement. Folks, "Roaming Profiles" are one of those 'sounds easy' to do technologies that starts to eat all your time unless you are ready for it. Yes, you can update it by manually changing the ADM template (see MS KB article # 290324). Funny that the article "applies to Vista" but doesn't describe how to update the matching ADMX template (hmmm.. I'll have to try that some time...or see if it's still an issue at all). Anyway, you do need to understand some of the behaviors you might run into. As noted in the "More Info" section of the article. UNDERSTATEMENT ALERT! "If you increase the maximum profile size to greater that 30 MB, it may take users longer to log on while the profile is loaded from the network." That 'can' be true for a user that 'really' roams from box to box on a daily basis. In practice, we found that our End Users tended to use the "same device" from day-to-day. So in our case, the most pressing issues began to be issues logging off at the end of their shift. Okay, we did have some issues with users logging onto devices in local Conference rooms...and experiencing long delays in getting to their Desktop while their 'large profile' downloaded. And try traveling to an office in a different city, logging onto your laptop in a Conf Room (using your company's LAN). There are all the meeting attendees looking at you and waiting for your presentation...for 20 minutes (or longer!)...talk about ouch! The solution was to train users to switch their profile to Local Mode 'before' leaving. We even gave them a desktop utility to easily switch...of course, then they'd conveniently 'forget' to switch back to roaming mode when they returned. Yeah.. right.... We expected that most users would have a profile way less than 20 MB and set out initial limits that way (just so you know, we based this on a scan of quite a few user profiles and 'most' were well within the limits before we updated their systems from Windows 2000 to Windows XP...things grow huh?). Additionally, we excluded a bunch of folders including the user's My Documents folder. We slowly discovered all the applications that cause larger and larger profiles and had to exclude more and more of them. Example We had to talk to Google about their Google Earth application (love that app by the way). They were loading their cache file (and yes, it was just a single file) in the C:\Docs&Settings\Users_Profile\Application Data\Google\Google Earth\... folder and they didn't need to be placing it there. Their default cache file size was 100 MB and a user can get there by just running their demo tour a couple of times. All the user had to do was launch GE and the cache file was updated. SO there's a 100 MB file to sync at logoff. More recent versions of GE more correctly placed the cache file in the C:\Docs&Settings\Users_Profile\Temporary Files\Application Data\Google... folder tree. In the meantime, we were out of space (260 char limit at the time) in the GPO to Exclude folders, so instead we wrote a custom ADM template to reset the Google Earth's cache to the minimum value of 16 MB... and as you can see, there went our 20 MB limit right out the door. Don't even get me started on why Roaming Profiles 'sometimes' go into a stupid state and only allow a user to access a temporary copy of a profile. And WATCH OUT for your Help Center. Sometimes they are too helpful (AND untrained). Consider this one. [EU - End User HC - Help Center] EU: I need help...all my desktop shortcuts and documents are missing. HC: Okay, do you have a Roaming Profile? [Side note, we built a web tool for our Help Center analysts to use to look this up and also trained them in the Profile support scripts we developed. Watch as they ignore all this. Also note that all End Users were given two full weeks to start using a Data Backup tool for their system devices. Multiple e-mails reminding them of this were sent out of this requirement. Sadly, not all used it as you're about to find out.] I see you have a laptop. [Oh, so the HC Tech finally start using their tools.] EU: I think so... is that what caused this problem? HC: Probably... You know that since you have a laptop, you really don't need a roaming profile don't you? [Oh yeah, thanks for that HC Tech. You haven't even looked at anything, have already diagnosed the problem, and have just 'dissed' the IT department.] Can I have your permission to do a remote takeover? EU: Yes, please, I need my stuff for a meeting in twenty minutes. HC: [Logs on...and wanders around the user's system.] Hmm, yes you do have a roaming profile. I think the system threw you into a temporary profile, but I can fix this for you... I did this kind of support at my previous job. All I have to do is copy your files over into this profile [into a "temporary" profile...what?... it's 'temporary...!!!! What about using our support scripts?] and then I'll just switch a few registry entries around so the system thinks that this is your real profile. Then you just need to reboot.. [WHAT... HC Techs Are NOT allowed to access and change End User registry entries. NOT PROCESS! NOT PROCESS! The HC tech "moves, not copies, the user's files, makes registry changes, and then cleans up the "old" profile by deleting the old one. Then goes to the profile server and 'deletes' the copy up there so the EU can 'start cleanly'. I hope every IT person reading this is screaming "No, No, No" by now.] Okay, you can reboot now. EU: Okay, well, it seems to be doing something. Okay I have my desktop back... wait, all my shortcuts and documents are missing again. My meeting is in 5 minutes. [So the user logged off and the system 'deleted' the temporary profile with all the files that the HC tech had moved there. Since the HC Tech had "moved" the files, there was no backup of them left on the device. Since we never roamed the Desktop in the first place, those files weren't up on the profile server in the first place, so not part of a server side data restoration from tape would help. Now it gets better...or worse L.] HC: Well, we can get the files back from the XXXXX tool [XXXXX = The tool the user's were supplied with to backup their device data.] EU: Is that that the tool those e-mails were talking about? Well, I haven't had time to load and run it. Don't you know how busy some of us are? HC: You haven't backed up your data? Ummm, well, I'm not sure we'll be able to get it back for you then. I think I'm going to have to escalate this to the IT Tech group. EU: Now you wait just a minute...WHERE's MY STUFF? Don't you know that I keep all my work data on my Desktop? That's two years worth of work and I NEED IT BACK RIGHT NOW! I have a meeting with some of the Company's directors... [Have we had enough now? Let's just say that you've GOT to have the HC staff trained to follow process and in whatever they do... "First Do No Harm"... And sadly, no, I didn't make this up. BY the time we were called to assist, a bunch of folks had tried all sorts of things..we immediately attempted to utilize some undelete utilities, but the damage was done and we were not able to recover anything significant for the End User...sigh. You can expect many IT managers heard an earful on this one. And just in case you don't know... ~ 95% of these kinds of issues are resolved by a simple reboot-which by the way was fully documented in the HC support scripts we provided.] Moving on, how about profile server storage and access issues? We had 40,000+ roaming users. In a population like that, quite a few had VERY large (100+ MB) profiles whose logoffs were taking 30-40 minutes at the end of the day. Most users profiles were taking 2-7 minutes to perform a logoff and profile sync until we were able to optimize the traffic and hardware resources. The IT department was NOT very popular for awhile. Imagine each server having to handling the logoff profile sync for 4,000 - 6,000 users within a ten minute timeframe...and we had multiple "dedicated" roaming profile servers...and of course they were all on the same company network LAN. That'd be a Time/Date stamp lookup of approximately 5,000 users times of an average 3,500 files = an access of 17.5 million files all within about 10 minutes, for each profile server. And then copying up the files that needed to be updated... Wow! Disk Queing Perf counters were easily in the 5 - 10 range (though at one point we actually recorded some in the mid-thirties.. WAY BIG Ouch... Most Server Admins will go into PANIC mode if they see Disk Queues higher than '2' (yes..only two)! We eventually tuned it back down to below that in general. For each server setup, we used servers with 2 GB RAM in 2 node clusters, Fibre-channel Controllers each with hardware buffers of 256 MB all set to 100% Write mode (obviously caching Reads is worthless with each user having their own files), RAID 5 on the hard disk drives (20 drives with 460 GB available on each-the amount of disk space was almost never an issue), and Gigabit network connectivity. Needless to say, these were some significantly powerful boxes. And don't forget the possible impact of taking one cluster down for maintenance. That can easily cause any accessing user account to switch to a temporary profile on their local device and the Help Centers calls start pouring in. Very few folks have to support implementations of such scale, but there it is. Hints * To support any deployments, go 'slowly' and be ready to help your "Help Center" analysts so that they can in turn help the End Users. * And by the way, laptop users need their data backed up even more than a desktop user. Laptop users will say that they already roam because they have a laptop. That isn't the point. Most any hardware maintenance group will tell you that laptops have higher maintenance costs. * As you ramp up profiles on a server, watch your Perf Counters VERY closely. We found out that once you start to hit the limits of the hardware, the bad perf numbers increase exponentially, not linearly. * Monitor your Performance metrics. * Deploy Microsoft's UPHClean to all Win2K and WinXP devices (it's built into Vista...yea!). * If possible for 'any' Roaming Profile deployment, spread the load to multiple servers. In case of a power outage in the middle of the day, just watch every single user try to log back on at the same time off a single server. If that load isn't spread across multiple servers...ouch! * Deploy Anti-virus to the desktops and seriously consider turning it 'off' for the profile server share folders (hey, it's the same data that was just on the PC and just got scanned there). That helps with server performance tuning. [Besides, we were starting weekly scans on Friday night and they were still running in the middle of the day on Monday...whew.) * Watch out especially for Java application folders...usually the applications that use them are coded by Java developers who are less familiar with Windows profiles and boy do they tend to 'load up' the profile. * User Profile data is NOT like other server storage data. It's typically a few big files and then literally thousands of 1KB - 2 KB files. (Remember about that 17.5 million file number? See above...). Most Server Admins have 'no experience' tuning servers to support this kind of data and will use the same process thinking to support a design for it. Your deployment team needs throw out all preconception and make sure everyone starts from scratch. * Test out your server data restoration processes and repeat testing them on a regular basis. * Monitor your Performance metrics. * Did I mention "Monitor your Performance metrics"? Pilot testing cannot be used as 'the' expectation. We had 400 pilot users on a single server whose logoff time increased by about 30 seconds. When we ramped up for production and got to about 1,000 users on the server...wham! * Go get and read Darren and Derek's (Melber) GPO book as well as Jeremy Moskowitz'es books on GPO's/Managed Desktops and read all you can about the various Roaming Profile scenarios. GPOs? We'll there's actually not too many. * Use the 'Add the Administrators security group to roaming user profiles' setting * We set the "Timeout for dialog boxes" setting to 1 second (the minimum you can set it too. BTW: If you set it to '0', the messages stay visible until the user explicitly clicks it off...so '1's the ticket to set). This minimizes calls to the Help Center for stuff the User would click OK on anyway. And if the user "has" a problem, the data is logged in the App Event log for the Help Center to find anyway. * We turned on Verbose UserEnv logging for all clients for debugging purposes, used tools to gather them from time to time, and wrote some parsers to extract certain types of data. The folks at SysProsoft have a 'free' and handy utility to look at individual UserEnv log files called: Policy Log Reporter http://www.sysprosoft.com/index.php?ref=activedir&f=policyreporter.shtml * Control the "Exclude directories in roaming profile setting" to exclude necessary folders. Here were some (not all) of ours: Desktop;My Documents;Recent;Application Data\Adobe;Application Data\AutoDesk;Application Data\Macromedia;Application Data\Microsoft\MSDAIPP;Application Data\Microsoft\Clip Organizer;Application Data\Roxio;Documents * If you limit the size of profiles, then consider updating the text of the popup message with the "Limit Profile Size" setting and also redirect the users to local resources (like your Help Center). * Consider controlling the "Prohibit User from manually redirecting Profile Folders" setting. So, can you successfully deploy Roaming Profiles to either small, medium, or large numbers of End Users? Sure, but be prepared to (1) go slowly, (2) spend some significant time supporting it (in terms of both hardware and personal time)...and more the larger numbers you go for, and (3) have some fun, it's a learning experience. It's a great feeling when you see the light come on for an End User who has almost everything back after logging onto a repaired system (after a system crash and reload). It 'can' and 'is' worth it. Jerry From: gptalk-bounce@xxxxxxxxxxxxx [mailto:gptalk-bounce@xxxxxxxxxxxxx] On Behalf Of Booker.Washington@xxxxxxxxxxxxxx Sent: Thursday, October 16, 2008 3:07 PM To: gptalk@xxxxxxxxxxxxx Subject: [gptalk] Do I need a custom adm, and/or where is the setting in the regsitry For my labs, I have set the limit profile size to its maximum of 30000kb. I want to raise it, but it gives me the error message that 30000kb is the max. Can I override the max with a custom adm, and/or where is that limit found in the registry. If I make the change t osomething higher than 30000kb will it even be recognized by policy? Booker T. Washington III Systems Support Specialist Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s), are confidential, and may be privileged. If you are not the intended recipient, you are hereby notified that any review, retransmission, conversion to hard copy, copying, circulation or other use of all or any portion of this message and any attachments is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return e-mail, and delete this message and any attachments from your system.