[ExchangeList] Re: Messages stuck in "local delivery" queue

  • From: "Nimori, Nathan" <nimorin@xxxxxxxxxx>
  • To: <exchangelist@xxxxxxxxxxxxx>
  • Date: Fri, 21 Nov 2008 09:45:29 -0800

http://www.msexchange.org
-------------------------------------------------------
We are definitely over utilizing our disks.  We are on a SAN with two
exchange drives.  One is for the log files and the other has our
infostores.  The one with our infostores is a 9 drive fiber channel
array at 10k rpm's.  At the time we built the server we were good but
since we did not implement any storage limits, well, we are in trouble
now.  We have lots of drive space but not enough spindles.  So we are
adding another 14 fiber channel drives (15k rpm) in raid 10 which will
hopefully help us out until we can design and integrate more exchange
servers.  I did run the analyzer and we seem to be in good shape except
for rpc and io of course.  What we just want to find out now is what
changed.  Since we didn't have a baseline before our latency showed up
we don't know what or who specifically is generating all the IO.  We
just wanted to see who our most active users are and possibly ask them
to limit their activity until we have our new drives and new servers in
place. Our administrators would like our users to continue using
exchange as a sort of storage area so in our new design we are planning
for more robust SAN's to handle all the IO, data archiving to move older
data to cheaper storage, and more exchange servers instead of just one.

Thanks for all of the info James.  I really appreciate all of your
comments and suggestions.  

-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of James Chong
Sent: Thursday, November 20, 2008 6:36 PM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue

http://www.msexchange.org
-------------------------------------------------------Yes. RPC
Operations is key. It doesn't take much from going from good performance
to start experiencing latency. Sometimes it's gradual; other times not
so much. One environment; it only took 20 users running desktop search
apps to start affecting user experience. I know many others experience
the same scenario only to find out that they don't have enough IOs. For
example; users switching the online mode can dramatically change your IO
requirements by altering your read write ratio. Sometimes it is just an
isolated event; for example I recall where an admin would run a
scheduled outlook rule periodically on a large postmaster mailbox and
create log stalls crashing the system. However IO was not sized
correctly to begin with. If you've already established that baseline IO
has increased; you will have no option but to throw additional disks. 


1. Run Exchange Best Practice analyzer to do a performance check to see
if it detects high rpc activity in general. EXBPA will determine the RPC
Ops\sec per user. If will consider high RPC activity if it's greater
than .25 per user. 

RPC User Activity From Performance Counters
http://technet.microsoft.com/en-us/library/aa995886.aspx

2. How are your counters for below? They should typically be under 20ms.

Disk\sec read
Disk\sec write


3. How are you disks\lun layout? How many users are you supporting? What
is the avg mailbox size? You will need to calculate how many IOS your
LUNS have then calculate your IOPs per user. This will give you a
general idea of your load. Since you mention RAID 10 on your DB use
conservative 80 IOS at 20ms. Then multiply that by number of disks; for
example 80 X 10 disks = 800 IOS. 

Next get your IOPs per user; use perform and get disk\disk transfers per
second\MSExchangeIS\Active User Count. Get these counters during peak
time. Also best to get values for twenty mins and get avg for both
counters to get best results. 

Your SAN guys can help you with this. 

A few basic concepts in disk sizing
http://msexchangeteam.com/archive/2004/10/11/240868.aspx

3. I'm assuming these luns are not shared with any other apps? Are all
your DBs shared on single LUN do you have other files? If so; you can
will incur random IO causing file level fragmentation. Did you enable
any additional services; journaling; indexing etc? 



James Chong
Sr. Systems Engineer
Simplexity, LLC.
11130 Sunrise Valley Drive, Suite 300
Reston, VA 20191
O (703) 657-4612
C (703) 863-1483


-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of Nimori, Nathan
Sent: Thursday, November 20, 2008 2:53 PM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue

http://www.msexchange.org
-------------------------------------------------------
Wow, here I thought I was asking a simple question!  Yes that does help,
thank you and also thank you to John.

I can understand the blackberry and outlook plug-ins causing additional
IO.  In EXMON, would that activity show up in the "OPERATIONS" column?
In other words, would it be safe to say that the top users sorted by
OPERATIONS are contributing to our IO problem even though they are not
necessarily the top users in the BYTES IN or BYTES OUT columns?  What
I'm seeing are the same 5 to 10 users at the top of the list when sorted
by Operations.  

Using perfmon and our SAN utilities (and with help from EMC support) we
found we are basically over utilizing our SAN drives and the cache is
force flushing way too often because the drives cannot keep up.  Now we
are in the process of adding more drives to our SAN (raid 10) but in the
meantime we want to find out what changed to cause so much IO.  The
outlook lag did not happen gradually.  It pretty much happened in one or
two days.  So we are thinking someone or a group of people may have just
installed something that is causing this problem or one of our
applications is not playing nice anymore.  

Thanks.


-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of James Chong
Sent: Thursday, November 20, 2008 6:50 AM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue

http://www.msexchange.org
-------------------------------------------------------As far as your
question on what incurs more IO large file vs. smaller files; in essence
it's not the size of files or whether you actually break it up; it's the
IO pattern. I passed this question along to John Fullbright MVP who's
much more knowledgeable in storage than I am. Here's his explanation:


"That's a loaded question.  
 
Each piece of data has metadata associated with it.  If I have one large
piece vs. several small pieces, arguably the large piece has less
metadata.  On this level, copying many small files the sum of which
equals the size of one large file would still incur the overhead of the
additional metadata.  In reality the metadata only adds up to a few
percentage points.  
 
When I look at the total data transferred, the difference isn't much.
What is drastically different between the twoi scenarios is the IO
pattern.  If I copy one large file then the IO pattern is a mostly
sequential one.  If I copy a set of files the sum of which is the same
as that one large file, then as the number of members of the set
increases the IO pattern becomes less sequential and more random.  This
assumes that I have multiple concurrent streams.  If I copy the files
one after another, then the IO pattern is still mostly sequential.  
 
Disks have far better throughput when writing sequential data than they
do when writing random data.  This is mainly because of the difference
between the random seek time and the track to track seek time.  While it
takes about 5ms to position the head for a random seek, it only takes
about half a ms for a track to track.  Modern disks pack anywhere
between 128 and 1000 sectors per track depending on where exactly the
track is located on the platter.  A purely sequential write workload
presents the sectors to a disk in LBN order which can easily be staged
and written with a minimum of head movement.  As the workload becomes
less sequential and more random, this becomes more and more difficult to
do.  Add to that the problem of overwrites, in most systems if the data
is overwritten it is physically overwritten, and increased randomness of
a workload results in significantly longer IO times.  A disk may be able
to sustain 1000 8K IOPS at a 20ms response time for a purely sequential
workload, but the number for random workloads looks more like 160.  If
you look at the amount of data written in the two scenarios, it's not
that much different.  If you look at the time it takes to write the data
for the two scenarios, there's a large difference."


Most high activity users in exmon culprits are:

1. User downloading ost file
2. User has blackberry
3. User has desktop search indexing software or other Outlook third
party plug ins could be Outlook spam software etc
4. User has large mailbox\folder with many items 5000+


Hope this helps. 

James Chong
Sr. Systems Engineer
Simplexity, LLC.
11130 Sunrise Valley Drive, Suite 300
Reston, VA 20191
O (703) 657-4612
C (703) 863-1483

-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of Nimori, Nathan
Sent: Wednesday, November 19, 2008 3:30 PM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue

http://www.msexchange.org
-------------------------------------------------------Thanks James.  

When using exmon, what fields would show IO usage?  There is obviously
bytes in and bytes out but what about packets or operations?  Our disk
is being slammed by IO and I sometimes wonder is it really the amount of
data being transferred or could it be lots of searches, indexing, etc
where people aren't necessarily transferring large attachments but are
just doing a lot of manipulation of e-mail.  For example, which would
make more IO - copying one large file from one disk to another or
thousands of small files equal to the large file.  

Users on the top of the list in bytes in and bytes out are always
changing.  I noticed that users in the top of the list of packets and
operations are usually the same bunch.

Thanks again.

___________________________
Nathan Nimori
Systems Engineer
East Side Union High School District

-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of James Chong
Sent: Wednesday, May 28, 2008 11:34 AM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue

http://www.msexchange.org
-------------------------------------------------------Could be several
reasons

1. DSACCESS related issues. Turn up diagnostic logging.

2. Sites and services are misconfigured and Exchange is using a remote
GC for lookups

3. You are incurring additional IO for some reason. See if there is some
process running; some large mail sent etc. Run perform and check your
physical disk avg disk sec reads and avg disk sec write. If you're over
50ms sustained you need to investigate. Also can look into using exmon
to see if a particular user is incurring the additional IO

4. Turn up diagnostic logging msexchangetransport

5. Run Exchange best practice analyzer


James Chong
Sr. Systems Engineer
Simplexity, LLC.
11130 Sunrise Valley Drive, Suite 300
Reston, VA 20191
O (703) 657-4612
C (703) 863-1483


-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of Enoque Modesto
dos Santos
Sent: Wednesday, May 28, 2008 1:46 PM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Messages stuck in "local delivery" queue

http://www.msexchange.org
-------------------------------------------------------Hi, ExchangeList,

I'm facing a problem related to a very large amount of messages stuck in
my Exchange serve 2003 "local delivery" queue;  this is caunsig the
Server performance to slow down and delaying the delivery of my user
messages.

Has someone faced such a thing ?

Thanks in advance,

Emo.

-------------------------------------------------------
List Archives: http://www.freelists.org/archives/exchangelist/
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/
MSExchange Blogs: http://blogs.msexchange.org/
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx

-------------------------------------------------------
List Archives: http://www.freelists.org/archives/exchangelist/  
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp 
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/ 
MSExchange Blogs: http://blogs.msexchange.org/ 
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com 
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx 

-------------------------------------------------------
List Archives: http://www.freelists.org/archives/exchangelist/  
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp 
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/ 
MSExchange Blogs: http://blogs.msexchange.org/ 
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com 
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx 

-------------------------------------------------------
List Archives: http://www.freelists.org/archives/exchangelist/  
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp 
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/ 
MSExchange Blogs: http://blogs.msexchange.org/ 
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com 
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx 

-------------------------------------------------------
List Archives: http://www.freelists.org/archives/exchangelist/  
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp 
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/ 
MSExchange Blogs: http://blogs.msexchange.org/ 
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com 
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx 

-------------------------------------------------------
List Archives: http://www.freelists.org/archives/exchangelist/  
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp 
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/ 
MSExchange Blogs: http://blogs.msexchange.org/ 
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com 
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx 

-------------------------------------------------------
List Archives: http://www.freelists.org/archives/exchangelist/
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp
MSExchange Articles and Tutorials: http://www.msexchange.org/articles_tutorials/
MSExchange Blogs: http://blogs.msexchange.org/
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx

Other related posts: