[ExchangeList] Re: Messages stuck in "local delivery" queue
- From: James Chong <jchong@xxxxxxxxxxxxxx>
- To: "exchangelist@xxxxxxxxxxxxx" <exchangelist@xxxxxxxxxxxxx>
- Date: Thu, 20 Nov 2008 21:35:40 -0500
http://www.msexchange.org
-------------------------------------------------------Yes. RPC Operations is
key. It doesn't take much from going from good performance to start
experiencing latency. Sometimes it's gradual; other times not so much. One
environment; it only took 20 users running desktop search apps to start
affecting user experience. I know many others experience the same scenario only
to find out that they don't have enough IOs. For example; users switching the
online mode can dramatically change your IO requirements by altering your read
write ratio. Sometimes it is just an isolated event; for example I recall where
an admin would run a scheduled outlook rule periodically on a large postmaster
mailbox and create log stalls crashing the system. However IO was not sized
correctly to begin with. If you've already established that baseline IO has
increased; you will have no option but to throw additional disks.
1. Run Exchange Best Practice analyzer to do a performance check to see if it
detects high rpc activity in general. EXBPA will determine the RPC Ops\sec per
user. If will consider high RPC activity if it's greater than .25 per user.
RPC User Activity From Performance Counters
http://technet.microsoft.com/en-us/library/aa995886.aspx
2. How are your counters for below? They should typically be under 20ms.
Disk\sec read
Disk\sec write
3. How are you disks\lun layout? How many users are you supporting? What is the
avg mailbox size? You will need to calculate how many IOS your LUNS have then
calculate your IOPs per user. This will give you a general idea of your load.
Since you mention RAID 10 on your DB use conservative 80 IOS at 20ms. Then
multiply that by number of disks; for example 80 X 10 disks = 800 IOS.
Next get your IOPs per user; use perform and get disk\disk transfers per
second\MSExchangeIS\Active User Count. Get these counters during peak time.
Also best to get values for twenty mins and get avg for both counters to get
best results.
Your SAN guys can help you with this.
A few basic concepts in disk sizing
http://msexchangeteam.com/archive/2004/10/11/240868.aspx
3. I'm assuming these luns are not shared with any other apps? Are all your DBs
shared on single LUN do you have other files? If so; you can will incur random
IO causing file level fragmentation. Did you enable any additional services;
journaling; indexing etc?
James Chong
Sr. Systems Engineer
Simplexity, LLC.
11130 Sunrise Valley Drive, Suite 300
Reston, VA 20191
O (703) 657-4612
C (703) 863-1483
-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of Nimori, Nathan
Sent: Thursday, November 20, 2008 2:53 PM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue
http://www.msexchange.org
-------------------------------------------------------
Wow, here I thought I was asking a simple question! Yes that does help,
thank you and also thank you to John.
I can understand the blackberry and outlook plug-ins causing additional
IO. In EXMON, would that activity show up in the "OPERATIONS" column?
In other words, would it be safe to say that the top users sorted by
OPERATIONS are contributing to our IO problem even though they are not
necessarily the top users in the BYTES IN or BYTES OUT columns? What
I'm seeing are the same 5 to 10 users at the top of the list when sorted
by Operations.
Using perfmon and our SAN utilities (and with help from EMC support) we
found we are basically over utilizing our SAN drives and the cache is
force flushing way too often because the drives cannot keep up. Now we
are in the process of adding more drives to our SAN (raid 10) but in the
meantime we want to find out what changed to cause so much IO. The
outlook lag did not happen gradually. It pretty much happened in one or
two days. So we are thinking someone or a group of people may have just
installed something that is causing this problem or one of our
applications is not playing nice anymore.
Thanks.
-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of James Chong
Sent: Thursday, November 20, 2008 6:50 AM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue
http://www.msexchange.org
-------------------------------------------------------As far as your
question on what incurs more IO large file vs. smaller files; in essence
it's not the size of files or whether you actually break it up; it's the
IO pattern. I passed this question along to John Fullbright MVP who's
much more knowledgeable in storage than I am. Here's his explanation:
"That's a loaded question.
Each piece of data has metadata associated with it. If I have one large
piece vs. several small pieces, arguably the large piece has less
metadata. On this level, copying many small files the sum of which
equals the size of one large file would still incur the overhead of the
additional metadata. In reality the metadata only adds up to a few
percentage points.
When I look at the total data transferred, the difference isn't much.
What is drastically different between the twoi scenarios is the IO
pattern. If I copy one large file then the IO pattern is a mostly
sequential one. If I copy a set of files the sum of which is the same
as that one large file, then as the number of members of the set
increases the IO pattern becomes less sequential and more random. This
assumes that I have multiple concurrent streams. If I copy the files
one after another, then the IO pattern is still mostly sequential.
Disks have far better throughput when writing sequential data than they
do when writing random data. This is mainly because of the difference
between the random seek time and the track to track seek time. While it
takes about 5ms to position the head for a random seek, it only takes
about half a ms for a track to track. Modern disks pack anywhere
between 128 and 1000 sectors per track depending on where exactly the
track is located on the platter. A purely sequential write workload
presents the sectors to a disk in LBN order which can easily be staged
and written with a minimum of head movement. As the workload becomes
less sequential and more random, this becomes more and more difficult to
do. Add to that the problem of overwrites, in most systems if the data
is overwritten it is physically overwritten, and increased randomness of
a workload results in significantly longer IO times. A disk may be able
to sustain 1000 8K IOPS at a 20ms response time for a purely sequential
workload, but the number for random workloads looks more like 160. If
you look at the amount of data written in the two scenarios, it's not
that much different. If you look at the time it takes to write the data
for the two scenarios, there's a large difference."
Most high activity users in exmon culprits are:
1. User downloading ost file
2. User has blackberry
3. User has desktop search indexing software or other Outlook third
party plug ins could be Outlook spam software etc
4. User has large mailbox\folder with many items 5000+
Hope this helps.
James Chong
Sr. Systems Engineer
Simplexity, LLC.
11130 Sunrise Valley Drive, Suite 300
Reston, VA 20191
O (703) 657-4612
C (703) 863-1483
-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of Nimori, Nathan
Sent: Wednesday, November 19, 2008 3:30 PM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue
http://www.msexchange.org
-------------------------------------------------------Thanks James.
When using exmon, what fields would show IO usage? There is obviously
bytes in and bytes out but what about packets or operations? Our disk
is being slammed by IO and I sometimes wonder is it really the amount of
data being transferred or could it be lots of searches, indexing, etc
where people aren't necessarily transferring large attachments but are
just doing a lot of manipulation of e-mail. For example, which would
make more IO - copying one large file from one disk to another or
thousands of small files equal to the large file.
Users on the top of the list in bytes in and bytes out are always
changing. I noticed that users in the top of the list of packets and
operations are usually the same bunch.
Thanks again.
___________________________
Nathan Nimori
Systems Engineer
East Side Union High School District
-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of James Chong
Sent: Wednesday, May 28, 2008 11:34 AM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue
http://www.msexchange.org
-------------------------------------------------------Could be several
reasons
1. DSACCESS related issues. Turn up diagnostic logging.
2. Sites and services are misconfigured and Exchange is using a remote
GC for lookups
3. You are incurring additional IO for some reason. See if there is some
process running; some large mail sent etc. Run perform and check your
physical disk avg disk sec reads and avg disk sec write. If you're over
50ms sustained you need to investigate. Also can look into using exmon
to see if a particular user is incurring the additional IO
4. Turn up diagnostic logging msexchangetransport
5. Run Exchange best practice analyzer
James Chong
Sr. Systems Engineer
Simplexity, LLC.
11130 Sunrise Valley Drive, Suite 300
Reston, VA 20191
O (703) 657-4612
C (703) 863-1483
-----Original Message-----
From: exchangelist-bounce@xxxxxxxxxxxxx
[mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of Enoque Modesto
dos Santos
Sent: Wednesday, May 28, 2008 1:46 PM
To: exchangelist@xxxxxxxxxxxxx
Subject: [ExchangeList] Messages stuck in "local delivery" queue
http://www.msexchange.org
-------------------------------------------------------Hi, ExchangeList,
I'm facing a problem related to a very large amount of messages stuck in
my Exchange serve 2003 "local delivery" queue; this is caunsig the
Server performance to slow down and delaying the delivery of my user
messages.
Has someone faced such a thing ?
Thanks in advance,
Emo.
-------------------------------------------------------
List Archives: //www.freelists.org/archives/exchangelist/
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/
MSExchange Blogs: http://blogs.msexchange.org/
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx
-------------------------------------------------------
List Archives: //www.freelists.org/archives/exchangelist/
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/
MSExchange Blogs: http://blogs.msexchange.org/
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx
-------------------------------------------------------
List Archives: //www.freelists.org/archives/exchangelist/
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/
MSExchange Blogs: http://blogs.msexchange.org/
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx
-------------------------------------------------------
List Archives: //www.freelists.org/archives/exchangelist/
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/
MSExchange Blogs: http://blogs.msexchange.org/
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx
-------------------------------------------------------
List Archives: //www.freelists.org/archives/exchangelist/
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp
MSExchange Articles and Tutorials:
http://www.msexchange.org/articles_tutorials/
MSExchange Blogs: http://blogs.msexchange.org/
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx
-------------------------------------------------------
List Archives: //www.freelists.org/archives/exchangelist/
MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp
MSExchange Articles and Tutorials: http://www.msexchange.org/articles_tutorials/
MSExchange Blogs: http://blogs.msexchange.org/
-------------------------------------------------------
Visit TechGenix.com for more information about our other sites:
http://www.techgenix.com
-------------------------------------------------------
To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp
Report abuse to listadmin@xxxxxxxxxxxxxx
Other related posts:
