http://www.msexchange.org ------------------------------------------------------- Wow, here I thought I was asking a simple question! Yes that does help, thank you and also thank you to John. I can understand the blackberry and outlook plug-ins causing additional IO. In EXMON, would that activity show up in the "OPERATIONS" column? In other words, would it be safe to say that the top users sorted by OPERATIONS are contributing to our IO problem even though they are not necessarily the top users in the BYTES IN or BYTES OUT columns? What I'm seeing are the same 5 to 10 users at the top of the list when sorted by Operations. Using perfmon and our SAN utilities (and with help from EMC support) we found we are basically over utilizing our SAN drives and the cache is force flushing way too often because the drives cannot keep up. Now we are in the process of adding more drives to our SAN (raid 10) but in the meantime we want to find out what changed to cause so much IO. The outlook lag did not happen gradually. It pretty much happened in one or two days. So we are thinking someone or a group of people may have just installed something that is causing this problem or one of our applications is not playing nice anymore. Thanks. -----Original Message----- From: exchangelist-bounce@xxxxxxxxxxxxx [mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of James Chong Sent: Thursday, November 20, 2008 6:50 AM To: exchangelist@xxxxxxxxxxxxx Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue http://www.msexchange.org -------------------------------------------------------As far as your question on what incurs more IO large file vs. smaller files; in essence it's not the size of files or whether you actually break it up; it's the IO pattern. I passed this question along to John Fullbright MVP who's much more knowledgeable in storage than I am. Here's his explanation: "That's a loaded question. Each piece of data has metadata associated with it. If I have one large piece vs. several small pieces, arguably the large piece has less metadata. On this level, copying many small files the sum of which equals the size of one large file would still incur the overhead of the additional metadata. In reality the metadata only adds up to a few percentage points. When I look at the total data transferred, the difference isn't much. What is drastically different between the twoi scenarios is the IO pattern. If I copy one large file then the IO pattern is a mostly sequential one. If I copy a set of files the sum of which is the same as that one large file, then as the number of members of the set increases the IO pattern becomes less sequential and more random. This assumes that I have multiple concurrent streams. If I copy the files one after another, then the IO pattern is still mostly sequential. Disks have far better throughput when writing sequential data than they do when writing random data. This is mainly because of the difference between the random seek time and the track to track seek time. While it takes about 5ms to position the head for a random seek, it only takes about half a ms for a track to track. Modern disks pack anywhere between 128 and 1000 sectors per track depending on where exactly the track is located on the platter. A purely sequential write workload presents the sectors to a disk in LBN order which can easily be staged and written with a minimum of head movement. As the workload becomes less sequential and more random, this becomes more and more difficult to do. Add to that the problem of overwrites, in most systems if the data is overwritten it is physically overwritten, and increased randomness of a workload results in significantly longer IO times. A disk may be able to sustain 1000 8K IOPS at a 20ms response time for a purely sequential workload, but the number for random workloads looks more like 160. If you look at the amount of data written in the two scenarios, it's not that much different. If you look at the time it takes to write the data for the two scenarios, there's a large difference." Most high activity users in exmon culprits are: 1. User downloading ost file 2. User has blackberry 3. User has desktop search indexing software or other Outlook third party plug ins could be Outlook spam software etc 4. User has large mailbox\folder with many items 5000+ Hope this helps. James Chong Sr. Systems Engineer Simplexity, LLC. 11130 Sunrise Valley Drive, Suite 300 Reston, VA 20191 O (703) 657-4612 C (703) 863-1483 -----Original Message----- From: exchangelist-bounce@xxxxxxxxxxxxx [mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of Nimori, Nathan Sent: Wednesday, November 19, 2008 3:30 PM To: exchangelist@xxxxxxxxxxxxx Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue http://www.msexchange.org -------------------------------------------------------Thanks James. When using exmon, what fields would show IO usage? There is obviously bytes in and bytes out but what about packets or operations? Our disk is being slammed by IO and I sometimes wonder is it really the amount of data being transferred or could it be lots of searches, indexing, etc where people aren't necessarily transferring large attachments but are just doing a lot of manipulation of e-mail. For example, which would make more IO - copying one large file from one disk to another or thousands of small files equal to the large file. Users on the top of the list in bytes in and bytes out are always changing. I noticed that users in the top of the list of packets and operations are usually the same bunch. Thanks again. ___________________________ Nathan Nimori Systems Engineer East Side Union High School District -----Original Message----- From: exchangelist-bounce@xxxxxxxxxxxxx [mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of James Chong Sent: Wednesday, May 28, 2008 11:34 AM To: exchangelist@xxxxxxxxxxxxx Subject: [ExchangeList] Re: Messages stuck in "local delivery" queue http://www.msexchange.org -------------------------------------------------------Could be several reasons 1. DSACCESS related issues. Turn up diagnostic logging. 2. Sites and services are misconfigured and Exchange is using a remote GC for lookups 3. You are incurring additional IO for some reason. See if there is some process running; some large mail sent etc. Run perform and check your physical disk avg disk sec reads and avg disk sec write. If you're over 50ms sustained you need to investigate. Also can look into using exmon to see if a particular user is incurring the additional IO 4. Turn up diagnostic logging msexchangetransport 5. Run Exchange best practice analyzer James Chong Sr. Systems Engineer Simplexity, LLC. 11130 Sunrise Valley Drive, Suite 300 Reston, VA 20191 O (703) 657-4612 C (703) 863-1483 -----Original Message----- From: exchangelist-bounce@xxxxxxxxxxxxx [mailto:exchangelist-bounce@xxxxxxxxxxxxx] On Behalf Of Enoque Modesto dos Santos Sent: Wednesday, May 28, 2008 1:46 PM To: exchangelist@xxxxxxxxxxxxx Subject: [ExchangeList] Messages stuck in "local delivery" queue http://www.msexchange.org -------------------------------------------------------Hi, ExchangeList, I'm facing a problem related to a very large amount of messages stuck in my Exchange serve 2003 "local delivery" queue; this is caunsig the Server performance to slow down and delaying the delivery of my user messages. Has someone faced such a thing ? Thanks in advance, Emo. ------------------------------------------------------- List Archives: //www.freelists.org/archives/exchangelist/ MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp MSExchange Articles and Tutorials: http://www.msexchange.org/articles_tutorials/ MSExchange Blogs: http://blogs.msexchange.org/ ------------------------------------------------------- Visit TechGenix.com for more information about our other sites: http://www.techgenix.com ------------------------------------------------------- To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp Report abuse to listadmin@xxxxxxxxxxxxxx ------------------------------------------------------- List Archives: //www.freelists.org/archives/exchangelist/ MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp MSExchange Articles and Tutorials: http://www.msexchange.org/articles_tutorials/ MSExchange Blogs: http://blogs.msexchange.org/ ------------------------------------------------------- Visit TechGenix.com for more information about our other sites: http://www.techgenix.com ------------------------------------------------------- To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp Report abuse to listadmin@xxxxxxxxxxxxxx ------------------------------------------------------- List Archives: //www.freelists.org/archives/exchangelist/ MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp MSExchange Articles and Tutorials: http://www.msexchange.org/articles_tutorials/ MSExchange Blogs: http://blogs.msexchange.org/ ------------------------------------------------------- Visit TechGenix.com for more information about our other sites: http://www.techgenix.com ------------------------------------------------------- To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp Report abuse to listadmin@xxxxxxxxxxxxxx ------------------------------------------------------- List Archives: //www.freelists.org/archives/exchangelist/ MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp MSExchange Articles and Tutorials: http://www.msexchange.org/articles_tutorials/ MSExchange Blogs: http://blogs.msexchange.org/ ------------------------------------------------------- Visit TechGenix.com for more information about our other sites: http://www.techgenix.com ------------------------------------------------------- To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp Report abuse to listadmin@xxxxxxxxxxxxxx ------------------------------------------------------- List Archives: //www.freelists.org/archives/exchangelist/ MSExchange Newsletter: http://www.msexchange.org/pages/newsletter.asp MSExchange Articles and Tutorials: http://www.msexchange.org/articles_tutorials/ MSExchange Blogs: http://blogs.msexchange.org/ ------------------------------------------------------- Visit TechGenix.com for more information about our other sites: http://www.techgenix.com ------------------------------------------------------- To unsubscribe visit http://www.msexchange.org/pages/exchangelist.asp Report abuse to listadmin@xxxxxxxxxxxxxx