[THIN] Re: Hyperthreading and effect on network i/o
- From: "Ron Oglesby" <roglesby@xxxxxxxxxxxx>
- To: <thin@xxxxxxxxxxxxx>
- Date: Fri, 26 Sep 2003 07:48:07 -0500
Quoted Rick : Funny thing is that after my post about hyperthreading, I
started thinking about what advantage hyperthreading could give a
file/print server, and couldn't think of a single reason to support it.
I have to agree. What hyperthreading buys you on a file server is pretty
much nothing. Citrix/app servers is a different story. Might have to
start looking at that a little more as we move the FnP servers to newer
models.
Ron
Ron Oglesby
Senior Technical Architect
RapidApp
Office 312.372.7188
Mobile 815.325.7618
email roglesby@xxxxxxxxxxxx
-----Original Message-----
From: Mack, Rick [mailto:RMack@xxxxxxxxxxxxxx]
Sent: Thursday, September 25, 2003 8:04 PM
To: 'thin@xxxxxxxxxxxxx'
Subject: [THIN] Re: Hyperthreading and effect on network i/o
Hi Erik,
We upped mampxct/maxworkitems on the file/print cluster members
(~0x1000, 0x4000), set virus checking for inbound files only, made sure
we were running all the latest drivers from IBM, and lastly turned off
hyperthreading on the cluster members (IBM 345's with a back end EMC
SAN).
The biggest change was initially induced by turning off virus checking
altogether. The hangs became much less frequent. Maxpxct and
maxworkitmes are standard tuning so that was done already, though the
values we had wre big enough , with current commands rarely exceeding
250. Sorting out all the drivers improved things some more, including
some odd file copy anomalies (sometimes file reads could be
comparatively slow). We could now turn virus checking back on (inbound
only) without a negative impact.
However the hangs continued, even if not very often.
Had a good look at other things causing excessive network i/o. We were
using folder redirection and had observed slow keyboard response in
office doing "save" operations, due to large recent file lists. Pruning
the recent file list and setting a policy to keep it short helped. But
still didn't sort the hangs completely.
Funny thing is that after my post about hyperthreading, I started
thinking about what advantage hyperthreading could give a file/print
server, and couldn't think of a single reason to support it.
Turned off hyperthreading on the cluster members (2 x file plus a
"dedicated" print) and the hangs have stopped completely. Used to get a
hang every 1-2 days, even with everything else we'd done, but haven't
had one for over a week now.
Also had something that gives us a lot of hope that the problem IS
solved. Load balancing went awry on the farm 3 days ago, and one server
ended up with 87 users before we sorted things out. It continued to run
all day with a load that had reduced to 60 users by mid afternoon.
So we had a 335 (2.5 GB RAM) with 87 users using office product, SAPGUI
6.2 and a mix of about 70 other applications. It slowed down but kept
going with no problems and it was a system with hyperthreading turned
off.
Previously, as you mentioned with your setup, 30 users had been about
the limit before things got interesting. Admittedly we are controlling
CPU utilization, but I've gone from starting to have grave misgivings
about the scalability of Server 2003 to being pretty positive again.
As a final note, the differences in current commands on TS systems with
hyperthreading turned on or off has reduced markedly since we finished
tuning the back end cluster. I haven't got a good reason why at the
moment, but having seen the number of users we can run on a
non-hyperthreaded system, and getting rid of our hangs by turning off
hyperthreading on the back end file/print cluster, I'm more than
prepared to stand up and recommend that hyperthreading should only be
used where a significant benefit can be demonstrated, under load.
I'm aware of the potential benefits of hyperthreading, eg reduced
context switching etc, but in the real world, I like a quiet life as
well, and this is one new technology that's going to be heavily muzzled
in my system designs for quite a while.
Regards,
Rick
Ulrich Mack
rmack@xxxxxxxxxxxxxx
Volante Systems
18 Heussler Terrace, Milton 4064
Queensland Australia.
tel +61 7 3246 7704
-----Original Message-----
From: Erik Blom [mailto:erik.blom@xxxxxxxxxx]
Sent: Thursday, 25 September 2003 7:57 PM
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Hyperthreading and effect on network i/o
Rick,
In your post ("Hyperthreading and effect on network i/o") you mention 8
IBM X335 systems and a file/print cluster. I have a client with exactly
the same setup (15 X335 servers and a file/print cluster) who is
experiencing exactly the same problem: temporary hangs when more than 30
users are logged on. We contacted MS, Citrix -- all to no avail. Now I
agree with you that it probably is network related: at the time where
the hangs were a serious problem, everyone was working with Outlook and
an AS/400 as a mail server. Probably due to the excessive i/o which was
caused by the frequent .pst file access, the system hung frequently.
Now this company is in a later stadium of their IT project, which
involves migrating the mail from the AS/400 to a native Exchange server
solution. Result: the hangs seem to be disappearing as more and more
users are migrated to Exchange.
Now I don't know if this is an IBM problem, a MS problem or a Citrix
problem. I'm starting to think I should re-address IBM 'cause no one in
this newsgroup seems to be having the same problems, except now you and
a few months earlier someone else (I may be wrong but I think he had
IBM's also).
In the meantime, could you be so kind to tell what tuning you did on the
back-end file/print server? If I cannot solve the problem completely,
maybe I would be able to at least alleviate it.
tx!
Erik Blom
PS 1 difference: this client runs W2K server and not W2003 server
Mack, Rick wrote:
> Hi People,
>
> I've been having some ongoing fun on one of our sites where 2003
> servers are hanging, luckily infrequently, when a bunch of users are
> logging on.
>
> We've found quite a few things that needed sorting out including
> tuning a back-end file/print cluster that have improved things
markedly.
>
> Nevertheless, we've been monitoring and playing with just about
> everything to try and get a handle on the hangs. This has included
> debug dumps to Microsoft, Citrix etc.
>
> All that aside, one very interesting thing did crop up. We have a
> total of 8 new IBM X-series model 335 systems, (and a bunch of older
> systems as well) running Windows server 2003. These new systems
> support hyperthreading and in an attempt to deal with the hangs, one
> of the things we did was to disable hyperthreading on a couple of the
> 335s.
>
> Since we had a suspicion that the hangs could be network i/o related,
> one of the things (amongst many!) that we've been monitoring is the
> local redirector (redirector > current commands). The 2 systems with
> hyperthreading disabled were consistently running with a current
> commands queue length that was 50%-60% of that is seen on the systems
> with hyprthreading enabled.
>
> Stated simply, the redirector's pending i/o request queue length
> (current commands) was much shorter on non-hyperthreaded systems
> compared to hyperthreaded systems with the same application and user
> load. The non-hyperthreaded systems were handling network i/o requests
> more efficiently
>
> Now this kind of makes sense in a scenario where a single-threaded NIC
> driver latches on to a logical CPU rather than a physical CPU.
> Certainly, a logical CPU is going to have less resources available
> than a physical CPU in terms of interrupt handling capability.
>
> This observation, if it's real, has some interesting ramifications if
> the effect of hyperthreading benefits cpu intensive processes at the
> expense of i/o intensive processes. Do I really want my file/print
> servers to go slower than they could?
>
> I guess the bottom line for me is that I'm going to be very cautious
> about recommending or using hyperthreading on our TS or file/print
> systems for a while until the effect of hyperthreading on i/o has been
> clarified. Terminal server systems are way too dependent on network
> i/o responsiveness for their good health for me to risk anything
> screwing up the works.
>
> regards,
>
> Rick
>
> Ulrich Mack
> rmack@xxxxxxxxxxxxxx
> Volante Systems
> 18 Heussler Terrace
> Milton 4064 Queensland, Australia
> tel +61 7 32467704
>
>
> ----------------------------------------------------------------------
> ----------------------------------------------
> The information contained in this e-mail is confidential and may be
> subject
> to legal professional privilege. It is intended solely for the
addressee.
> If you receive this e-mail by mistake please promptly inform us by
reply
> e-mail and then delete the e-mail and destroy any printed copy. You
must
> not disclose or use in any way the information in the e-mail. There is
no
> warranty that this email or any attachment or message is error or
> virus free. It may be a private
> communication, and if so, does not represent the views of Volante
> group Limited.
********************************************************
This Week's Sponsor - RTO Software / TScale
What's keeping you from getting more from your terminal servers? Did you
know, in most cases, CPU Utilization IS NOT the single biggest
constraint to scaling up?! Get this free white paper to understand the
real constraints & how to overcome them. SAVE MONEY by scaling-up rather
than buying more servers. http://www.rtosoft.com/Enter.asp?ID=147
**********************************************************
Useful Thin Client Computing Links are available at:
http://thethin.net/links.cfm New! Online Thin Computing Magazine Site
http://www.OnDemandAccess.com
For Archives, to Unsubscribe, Subscribe or
set Digest or Vacation mode use the below link:
http://thethin.net/citrixlist.cfm
------------------------------------------------------------------------
--------------------------------------------
The information contained in this e-mail is confidential and may be
subject
to legal professional privilege. It is intended solely for the
addressee.
If you receive this e-mail by mistake please promptly inform us by reply
e-mail and then delete the e-mail and destroy any printed copy. You must
not disclose or use in any way the information in the e-mail. There is
no
warranty that this email or any attachment or message is error or virus
free. It may be a private
communication, and if so, does not represent the views of Volante group
Limited.
This message has been checked by SurfControl
Other related posts:
