[THIN] Re: Multiple servers load set to 10000 at same time

  • From: Angela Smith <angela_smith9@xxxxxxxxxxx>
  • To: <thin@xxxxxxxxxxxxx>
  • Date: Thu, 18 Oct 2012 21:32:32 +1100


Hi Jeremy

1) I ran Perfmon from another machine and added all memory and cpu counters.  I 
couldn't see any resource issues at all

2) Health Monitoring is not enabled on our farm.

Regards
Ang

From: jeremy@xxxxxxxxxxxxxxxxxxxx
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Multiple servers load set to 10000 at same time
Date: Thu, 18 Oct 2012 17:56:49 +0800

Hi Ang, Two things: 1)      Yes, you can run Perfmon from a remote machine.2)   
   Do you have the Health Monitoring & Recovery (HMR) tests in place? The 
default recovery action for the XML Service test is set to Remove Server from 
Load Balancing. This can cause confusion, so I change it to Alert Only, 
allowing administrators to monitor it instead. If a server is removed from Load 
Balancing due to a Health Monitoring Test failure, it will remain out of Load 
Balancing until one of the following actions occurs: The enablelb command is 
executed from a command prompt or the server is rebooted. But I’d say there is 
some underlying problem if you can’t run any of your tools from the servers 
themselves. Cheers,Jeremy. From: thin-bounce@xxxxxxxxxxxxx 
[mailto:thin-bounce@xxxxxxxxxxxxx] On Behalf Of Angela Smith
Sent: Thursday, 18 October 2012 5:16 PM
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Multiple servers load set to 10000 at same time Hi

Yes, when the server reboots, all is working OK.  Server isn't set to 10000 and 
there are no down sessions.  However, issue re-occurs after a few hours.  Issue 
is happening on random servers so I don't think its a WMI Issue (very unlikely 
that WMI is broken on 6 different servers).  Nothing has changed on our servers 
either.

We have a custom Load Evaluator which we have been using for several years 
without issue.  It basically checks CPU, Memory and User Load.  Load is 
definitely not an issue as the servers only have 15 users on them when the 
issue occurs whereas they normally have 40.  I cannot easily check resources as 
all GUI tools fail to load.  Task Manager, Perfmon, Sysinternals Process 
Explorer etc don't launch. Whats the best way to see what is happening with CPU 
and Memory?  Perhaps I can run the tool remotely?  As a test should I change 
all servers to the Adv evaluator?

Ive never had any issues with memory.  Each server has 16GB RAM.  In relation 
to CPU, Ive had the occasional process go crazy but its never used all cores.  
Servers have 2 x Quad core CPU's.  Print Spooler and Antivirus all appears OK 
in that the service is started but I'm working blind as I cannot see the 
processes due to the tools not launching 

Perfmon wont load so I cannot rell is perfmon counters are corrupt

We do have SCOM.  How do I reload the MOF?. Would the MOF generate a ICA down 
session?  Every server that goes to 10000 has a down session so its definitely 
causing the issue or is related..


Thanks
AngCC: thin@xxxxxxxxxxxxx
From: magnus@xxxxxxxx
Subject: [THIN] Re: Multiple servers load set to 10000 at same time
Date: Wed, 17 Oct 2012 22:17:55 -0400
To: thin@xxxxxxxxxxxxxxx notice wmi issues as well, this was do to MOM/SCOM. We 
had to reload the mof's to fix it. 

Sent from my iPhone
On Oct 17, 2012, at 20:15, Alan Tropper <Alan.Tropper@xxxxxxxxxxxxx> wrote:Hi, 
I have had WMI issues also in the past and had to re-build WMI components as 
below: http://musumeci.blogspot.com.au/2009/01/ms-rebuild-wmi.html Cheers
Al From: thin-bounce@xxxxxxxxxxxxx [mailto:thin-bounce@xxxxxxxxxxxxx] On Behalf 
Of julien, Sybille
Sent: Wednesday, 17 October 2012 6:56 PM
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Multiple servers load set to 10000 at same time did you 
check if your perfmon counter is corrupted ?  
http://support.citrix.com/article/CTX129350  <~WRD000.jpg> Julien  De : Jeremy 
Saunders <jeremy@xxxxxxxxxxxxxxxxxxxx>
À : thin@xxxxxxxxxxxxx 
Envoyé le : Mercredi 17 octobre 2012 10h38
Objet : [THIN] Re: Multiple servers load set to 10000 at same time The server 
thinks it’s at full load. What’s the load evaluator assigned to it? If Task 
Manager is not loading, it sounds like perhaps a CPU/memory utilisation issue, 
which is effecting the load evaluator, which is placing the server at full 
load. So what’s the CPU and memory utilisation on the server? Could you have a 
rouge service, such as antivirus causing problems? What about the print spooler 
and Citrix print manager services? Cheers,Jeremy. From: 
thin-bounce@xxxxxxxxxxxxx [mailto:thin-bounce@xxxxxxxxxxxxx] On Behalf Of 
Hamilton, Ronnie
Sent: Wednesday, 17 October 2012 4:00 PM
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Multiple servers load set to 10000 at same time Hi, You say 
that even a reboot doesn’t help ? Is that correct ? I had an RDP session in a 
down state that was doing the same thing I had to reset it via the Microsoft 
Terminal Services Console and this resolved that issue.  It did take a few 
times and I think I deleted my admin user profile as well as they don’t get 
deleted like all our user ones. Regards,Ronnie From: thin-bounce@xxxxxxxxxxxxx 
[mailto:thin-bounce@xxxxxxxxxxxxx] On Behalf Of Angela Smith
Sent: 17 October 2012 05:10
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Multiple servers load set to 10000 at same time Hi

I jumped the gun..  PSE450R05W2K3026 is in Rollup 6 which I already have 
installed..  Back to the drawing board!

AngFrom: angela_smith9@xxxxxxxxxxx
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Multiple servers load set to 10000 at same time
Date: Wed, 17 Oct 2012 15:07:23 +1100Hi

This patch looks promising..  Only issue is I cannot find it.  

http://support.citrix.com/article/CTX123956

This has been superseded by R07 patch which I dont want to install right now.  
Im currently running R06.  

Does anyone have hotfix PSE450R05W2K3026 that I can download for x86 systems?

Thanks
AngCC: thin@xxxxxxxxxxxxx
From: magnus@xxxxxxxx
Subject: [THIN] Re: Multiple servers load set to 10000 at same time
Date: Tue, 16 Oct 2012 22:13:30 -0400
To: thin@xxxxxxxxxxxxxxxxxx the nonpagepool memory usage in taskmgr. I have 
seen this usage spike which was due to an ms bug and XTE.exe. Good news there 
is a fix for it. 936655 I beleive the number is 

Sent from my iPhone
On Oct 16, 2012, at 18:10, Angela Smith <angela_smith9@xxxxxxxxxxx> wrote:Hi

I think my issue is different.  All user sessions on these servers lock up and 
they end up calling the helpdesk to reset them..  I'm getting down sessions on 
the same servers each day (even after a reboot)..  I applied CTX112103 but it 
didn't make any difference..  

Any other ideas apart from rebuilding?

Thanks
AngSubject: [THIN] Re: Multiple servers load set to 10000 at same time
Date: Mon, 15 Oct 2012 12:48:11 +0100
From: ronnie.hamilton@xxxxxxx
To: thin@xxxxxxxxxxxxxxx Angela, This does ring a bell and I’m sure I have one 
server with this patch and made no difference to me, but the csrss.exe process 
wasn’t showing a high CPU usage for me. It more just creating error in console 
Session Down state every few mins. It only stopped users connecting on the very 
odd ocasion  Regards,Ronnie From: thin-bounce@xxxxxxxxxxxxx 
[mailto:thin-bounce@xxxxxxxxxxxxx] On Behalf Of Angela Smith
Sent: 15 October 2012 12:36
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Re: Multiple servers load set to 10000 at same time Hi Ronnie

Unfortunately rebuilding wont be an option for me..  Did you try:

http://support.microsoft.com/kb/934330

My entire farm will reboot tonight.  If issue re-occurs then I might try this 
patch..

Thanks
AngSubject: [THIN] Re: Multiple servers load set to 10000 at same time
Date: Mon, 15 Oct 2012 12:25:28 +0100
From: ronnie.hamilton@xxxxxxx
To: thin@xxxxxxxxxxxxxxx Angela, I get this from time to time, server doesn’t 
always report full load but there’s always  a session down state, and a reboot 
usually clears. I had a look at the following thread but never actually got to 
the bottom of it. I am currently rebuilding all my server to upgrade to Office 
2010 and hoping that the rebuild does the trick. 
http://forums.citrix.com/message.jspa?messageID=1386285#1386285 It’s seems to 
be going well as all the new builds have not reports the issue, but one of the 
pre Office 2010 builds did yesterday.  Regards,Ronnie From: 
thin-bounce@xxxxxxxxxxxxx [mailto:thin-bounce@xxxxxxxxxxxxx] On Behalf Of 
Angela Smith
Sent: 15 October 2012 11:54
To: thin@xxxxxxxxxxxxx
Subject: [THIN] Multiple servers load set to 10000 at same time Hi 

I have an issue on my XenApp 4.5 farm where 4 servers are reporting Server load 
10000.  I tried to RDP to the servers and launch Task Manager. Unfortunately 
this doesn't load on any of the 4 servers.  I tried to use Process Explorer 
from Sysinternals tools but that also didn't load. What I did notice though was 
all 4 servers had a session in a down state which I couldn't reset.  It simply 
came back each time.  Error.jpg attached showing issue. I don't believe Im 
having a resource issue but I cannot be 100% sure.

I tried to change my Load Evaluator from Advanced to Default as a test but this 
made no difference.  Also tried to clear some sessions but load didn't change, 
still set to 10000.

Can anyone suggest how I can troubleshoot this 10000 Load issue without 
rebooting server?  FYI I do reboot my servers nightly but this issue has 
happened a few times now.  Also is there a way/tool to check system resources 
when Task Manager doesn't work? 


Thanks 
Ang  

 
Visit our website : http://www.ltai.ie/ 
__________________________________________
Lufthansa Technik Airmotive Ireland Limited. Registered in Ireland. Reg. No. 
45999. Registered Office: Naas Road, Rathcoole, Co.Dublin.
Lufthansa Technik Airmotive Ireland Leasing Limited. Registered in Ireland. 
Reg. No. 140891. Registered Office: Naas Road, Rathcoole, Co.Dublin.
__________________________________________The information in this email and in 
any attachments is confidential and may be privileged. If you are not the 
intended recipient, please destroy this message, delete any copies held on your 
systems and notify the sender by return email. You should not read, retain, 
copy, disseminate, distribute, disclose or use this email or its contents in 
any way. Any such action is strictly prohibited. Thank you.   This message may 
contain privileged and confidential information and is intended for the 
exclusive use of the addressee(s). You must not disclose this communication to 
anyone without the prior consent of the Department for Child Protection (DCP). 
If you have received this email in error, please notify us by return mail, 
delete it from your system and destroy all copies. DCP has exercised care to 
avoid errors in the information contained in this email but does not warrant 
that it is error or omission free.                                            

Other related posts: