[gptalk] Re: Do I need a custom adm, and/or where is the setting in the regsitry
- From: "Cruz, Jerome L" <jerome.l.cruz@xxxxxxxxxx>
- To: "gptalk@xxxxxxxxxxxxx" <gptalk@xxxxxxxxxxxxx>
- Date: Thu, 16 Oct 2008 19:29:21 -0700
Hi Booker,
My answer is going to be a bit (okay, maybe a LOT) longer than the usual.
Hopefully others will see the things you can run into with roaming profiles and
not just assume the technology and it's support system are a breeze to
implement. Folks, "Roaming Profiles" are one of those 'sounds easy' to do
technologies that starts to eat all your time unless you are ready for it.
Yes, you can update it by manually changing the ADM template (see MS KB article
# 290324). Funny that the article "applies to Vista" but doesn't describe how
to update the matching ADMX template (hmmm.. I'll have to try that some
time...or see if it's still an issue at all). Anyway, you do need to understand
some of the behaviors you might run into. As noted in the "More Info" section
of the article.
UNDERSTATEMENT ALERT!
"If you increase the maximum profile size to greater that 30 MB, it may take
users longer to log on while the profile is loaded from the network."
That 'can' be true for a user that 'really' roams from box to box on a daily
basis. In practice, we found that our End Users tended to use the "same device"
from day-to-day. So in our case, the most pressing issues began to be issues
logging off at the end of their shift.
Okay, we did have some issues with users logging onto devices in local
Conference rooms...and experiencing long delays in getting to their Desktop
while their 'large profile' downloaded. And try traveling to an office in a
different city, logging onto your laptop in a Conf Room (using your company's
LAN). There are all the meeting attendees looking at you and waiting for your
presentation...for 20 minutes (or longer!)...talk about ouch! The solution was
to train users to switch their profile to Local Mode 'before' leaving. We even
gave them a desktop utility to easily switch...of course, then they'd
conveniently 'forget' to switch back to roaming mode when they returned. Yeah..
right....
We expected that most users would have a profile way less than 20 MB and set
out initial limits that way (just so you know, we based this on a scan of quite
a few user profiles and 'most' were well within the limits before we updated
their systems from Windows 2000 to Windows XP...things grow huh?).
Additionally, we excluded a bunch of folders including the user's My Documents
folder. We slowly discovered all the applications that cause larger and larger
profiles and had to exclude more and more of them.
Example We had to talk to Google about their Google Earth application (love
that app by the way). They were loading their cache file (and yes, it was just
a single file) in the C:\Docs&Settings\Users_Profile\Application
Data\Google\Google Earth\... folder and they didn't need to be placing it
there. Their default cache file size was 100 MB and a user can get there by
just running their demo tour a couple of times. All the user had to do was
launch GE and the cache file was updated. SO there's a 100 MB file to sync at
logoff. More recent versions of GE more correctly placed the cache file in the
C:\Docs&Settings\Users_Profile\Temporary Files\Application Data\Google...
folder tree. In the meantime, we were out of space (260 char limit at the time)
in the GPO to Exclude folders, so instead we wrote a custom ADM template to
reset the Google Earth's cache to the minimum value of 16 MB... and as you can
see, there went our 20 MB limit right out the door.
Don't even get me started on why Roaming Profiles 'sometimes' go into a stupid
state and only allow a user to access a temporary copy of a profile. And WATCH
OUT for your Help Center. Sometimes they are too helpful (AND untrained).
Consider this one. [EU - End User HC - Help Center]
EU: I need help...all my desktop shortcuts and documents are missing.
HC: Okay, do you have a Roaming Profile? [Side note, we built a web tool for
our Help Center analysts to use to look this up and also trained them in the
Profile support scripts we developed. Watch as they ignore all this. Also note
that all End Users were given two full weeks to start using a Data Backup tool
for their system devices. Multiple e-mails reminding them of this were sent out
of this requirement. Sadly, not all used it as you're about to find out.] I see
you have a laptop. [Oh, so the HC Tech finally start using their tools.]
EU: I think so... is that what caused this problem?
HC: Probably... You know that since you have a laptop, you really don't need a
roaming profile don't you? [Oh yeah, thanks for that HC Tech. You haven't even
looked at anything, have already diagnosed the problem, and have just 'dissed'
the IT department.] Can I have your permission to do a remote takeover?
EU: Yes, please, I need my stuff for a meeting in twenty minutes.
HC: [Logs on...and wanders around the user's system.] Hmm, yes you do have a
roaming profile. I think the system threw you into a temporary profile, but I
can fix this for you... I did this kind of support at my previous job. All I
have to do is copy your files over into this profile [into a "temporary"
profile...what?... it's 'temporary...!!!! What about using our support
scripts?] and then I'll just switch a few registry entries around so the system
thinks that this is your real profile. Then you just need to reboot.. [WHAT...
HC Techs Are NOT allowed to access and change End User registry entries. NOT
PROCESS! NOT PROCESS! The HC tech "moves, not copies, the user's files, makes
registry changes, and then cleans up the "old" profile by deleting the old one.
Then goes to the profile server and 'deletes' the copy up there so the EU can
'start cleanly'. I hope every IT person reading this is screaming "No, No, No"
by now.] Okay, you can reboot now.
EU: Okay, well, it seems to be doing something. Okay I have my desktop back...
wait, all my shortcuts and documents are missing again. My meeting is in 5
minutes. [So the user logged off and the system 'deleted' the temporary profile
with all the files that the HC tech had moved there. Since the HC Tech had
"moved" the files, there was no backup of them left on the device. Since we
never roamed the Desktop in the first place, those files weren't up on the
profile server in the first place, so not part of a server side data
restoration from tape would help. Now it gets better...or worse :(.]
HC: Well, we can get the files back from the XXXXX tool [XXXXX = The tool the
user's were supplied with to backup their device data.]
EU: Is that that the tool those e-mails were talking about? Well, I haven't had
time to load and run it. Don't you know how busy some of us are?
HC: You haven't backed up your data? Ummm, well, I'm not sure we'll be able to
get it back for you then. I think I'm going to have to escalate this to the IT
Tech group.
EU: Now you wait just a minute...WHERE's MY STUFF? Don't you know that I keep
all my work data on my Desktop? That's two years worth of work and I NEED IT
BACK RIGHT NOW! I have a meeting with some of the Company's directors... [Have
we had enough now? Let's just say that you've GOT to have the HC staff trained
to follow process and in whatever they do... "First Do No Harm"... And sadly,
no, I didn't make this up. BY the time we were called to assist, a bunch of
folks had tried all sorts of things..we immediately attempted to utilize some
undelete utilities, but the damage was done and we were not able to recover
anything significant for the End User...sigh. You can expect many IT managers
heard an earful on this one. And just in case you don't know... ~ 95% of these
kinds of issues are resolved by a simple reboot-which by the way was fully
documented in the HC support scripts we provided.]
Moving on, how about profile server storage and access issues? We had 40,000+
roaming users. In a population like that, quite a few had VERY large (100+ MB)
profiles whose logoffs were taking 30-40 minutes at the end of the day. Most
users profiles were taking 2-7 minutes to perform a logoff and profile sync
until we were able to optimize the traffic and hardware resources. The IT
department was NOT very popular for awhile. Imagine each server having to
handling the logoff profile sync for 4,000 - 6,000 users within a ten minute
timeframe...and we had multiple "dedicated" roaming profile servers...and of
course they were all on the same company network LAN. That'd be a Time/Date
stamp lookup of approximately 5,000 users times of an average 3,500 files = an
access of 17.5 million files all within about 10 minutes, for each profile
server. And then copying up the files that needed to be updated... Wow! Disk
Queing Perf counters were easily in the 5 - 10 range (though at one point we
actually recorded some in the mid-thirties.. WAY BIG Ouch... Most Server Admins
will go into PANIC mode if they see Disk Queues higher than '2' (yes..only
two)! We eventually tuned it back down to below that in general.
For each server setup, we used servers with 2 GB RAM in 2 node clusters,
Fibre-channel Controllers each with hardware buffers of 256 MB all set to 100%
Write mode (obviously caching Reads is worthless with each user having their
own files), RAID 5 on the hard disk drives (20 drives with 460 GB available on
each-the amount of disk space was almost never an issue), and Gigabit network
connectivity. Needless to say, these were some significantly powerful boxes.
And don't forget the possible impact of taking one cluster down for
maintenance. That can easily cause any accessing user account to switch to a
temporary profile on their local device and the Help Centers calls start
pouring in.
Very few folks have to support implementations of such scale, but there it is.
Hints
* To support any deployments, go 'slowly' and be ready to help your
"Help Center" analysts so that they can in turn help the End Users.
* And by the way, laptop users need their data backed up even more than
a desktop user. Laptop users will say that they already roam because they have
a laptop. That isn't the point. Most any hardware maintenance group will tell
you that laptops have higher maintenance costs.
* As you ramp up profiles on a server, watch your Perf Counters VERY
closely. We found out that once you start to hit the limits of the hardware,
the bad perf numbers increase exponentially, not linearly.
* Monitor your Performance metrics.
* Deploy Microsoft's UPHClean to all Win2K and WinXP devices (it's
built into Vista...yea!).
* If possible for 'any' Roaming Profile deployment, spread the load to
multiple servers. In case of a power outage in the middle of the day, just
watch every single user try to log back on at the same time off a single
server. If that load isn't spread across multiple servers...ouch!
* Deploy Anti-virus to the desktops and seriously consider turning it
'off' for the profile server share folders (hey, it's the same data that was
just on the PC and just got scanned there). That helps with server performance
tuning. [Besides, we were starting weekly scans on Friday night and they were
still running in the middle of the day on Monday...whew.)
* Watch out especially for Java application folders...usually the
applications that use them are coded by Java developers who are less familiar
with Windows profiles and boy do they tend to 'load up' the profile.
* User Profile data is NOT like other server storage data. It's
typically a few big files and then literally thousands of 1KB - 2 KB files.
(Remember about that 17.5 million file number? See above...). Most Server
Admins have 'no experience' tuning servers to support this kind of data and
will use the same process thinking to support a design for it. Your deployment
team needs throw out all preconception and make sure everyone starts from
scratch.
* Test out your server data restoration processes and repeat testing
them on a regular basis.
* Monitor your Performance metrics.
* Did I mention "Monitor your Performance metrics"? Pilot testing
cannot be used as 'the' expectation. We had 400 pilot users on a single server
whose logoff time increased by about 30 seconds. When we ramped up for
production and got to about 1,000 users on the server...wham!
* Go get and read Darren and Derek's (Melber) GPO book as well as
Jeremy Moskowitz'es books on GPO's/Managed Desktops and read all you can about
the various Roaming Profile scenarios.
GPOs? We'll there's actually not too many.
* Use the 'Add the Administrators security group to roaming user
profiles' setting
* We set the "Timeout for dialog boxes" setting to 1 second (the
minimum you can set it too. BTW: If you set it to '0', the messages stay
visible until the user explicitly clicks it off...so '1's the ticket to set).
This minimizes calls to the Help Center for stuff the User would click OK on
anyway. And if the user "has" a problem, the data is logged in the App Event
log for the Help Center to find anyway.
* We turned on Verbose UserEnv logging for all clients for debugging
purposes, used tools to gather them from time to time, and wrote some parsers
to extract certain types of data. The folks at SysProsoft have a 'free' and
handy utility to look at individual UserEnv log files called: Policy Log
Reporter
http://www.sysprosoft.com/index.php?ref=activedir&f=policyreporter.shtml
* Control the "Exclude directories in roaming profile setting" to
exclude necessary folders. Here were some (not all) of ours:
Desktop;My Documents;Recent;Application Data\Adobe;Application
Data\AutoDesk;Application Data\Macromedia;Application
Data\Microsoft\MSDAIPP;Application Data\Microsoft\Clip Organizer;Application
Data\Roxio;Documents
* If you limit the size of profiles, then consider updating the text of
the popup message with the "Limit Profile Size" setting and also redirect the
users to local resources (like your Help Center).
* Consider controlling the "Prohibit User from manually redirecting
Profile Folders" setting.
So, can you successfully deploy Roaming Profiles to either small, medium, or
large numbers of End Users? Sure, but be prepared to (1) go slowly, (2) spend
some significant time supporting it (in terms of both hardware and personal
time)...and more the larger numbers you go for, and (3) have some fun, it's a
learning experience. It's a great feeling when you see the light come on for an
End User who has almost everything back after logging onto a repaired system
(after a system crash and reload). It 'can' and 'is' worth it.
Jerry
From: gptalk-bounce@xxxxxxxxxxxxx [mailto:gptalk-bounce@xxxxxxxxxxxxx] On
Behalf Of Booker.Washington@xxxxxxxxxxxxxx
Sent: Thursday, October 16, 2008 3:07 PM
To: gptalk@xxxxxxxxxxxxx
Subject: [gptalk] Do I need a custom adm, and/or where is the setting in the
regsitry
For my labs, I have set the limit profile size to its maximum of 30000kb. I
want to raise it, but it gives me the error message that 30000kb is the max.
Can I override the max with a custom adm, and/or where is that limit found in
the registry. If I make the change t osomething higher than 30000kb will it
even be recognized by policy?
Booker T. Washington III
Systems Support Specialist
Other related posts:
