We have opened several SRs with Oracle for the database problems. They have never been able to identify anything as the source of the problem always indicating the problem was OS or disk storage. The core files and guds captured by our DBA and UNIX Admin team helped Oracle hardware/OS support to determine that we are running into an OS bug. The bug number is 6876962. This is an excerpt from the service request... This looks like CR# 6876962 - degraded write performance with threads held up by space_map_load_wait(). This bug is fixed in patch 147440-05, -06 or -07, which is current and contains the fix. I personally believe we have been running into this bug for more than 1 year - and have been pushing Oracle for well over 1 year for a resolution. The trace and core files we provided finally allowed them to identify the possible source of our problems. ________________________________ From: Jared Still [mailto:jkstill@xxxxxxxxx] Sent: Tuesday, December 13, 2011 9:40 PM To: Johnson, William L (TEIS) Cc: ORACLE-L Subject: Re: Solaris 10 dtrace command... Any idea why it is taking so long to resolve paths? And why it takes 7+ minutes to determine that a file is missing? I would be concerned about my storage. Or there could be some bug causing issues. Have you taken this up with the storage admins, or Sun^H^H^HOracle support? Jared Still Certifiable Oracle DBA and Part Time Perl Evangelist Oracle Blog: http://jkstill.blogspot.com Home Page: http://jaredstill.com On Mon, Dec 12, 2011 at 5:55 PM, Johnson, William L (TEIS) <WLJohnson@xxxxxx<mailto:WLJohnson@xxxxxx>> wrote: I was able to catch the hang in production again. Really ugly this time - one command took 8-9 minutes to execute and a simple lsnrctl status took close to 15 minutes to execute in a production environment. Here was my command... truss -d -D -f -o bill2.out ls -lrt This is a small section of the resulting output...A bad start right out of the gates... Base time stamp: 1323721323.0651 [ Mon Dec 12 15:22:03 EST 2011 ] 1186: 102.3725 102.3725 resolvepath("/usr/lib/ld.so.1", "/lib/ld.so.1", 1023) = 12 Here was the second incident...(I replaced some of our file system names with variables to protect the innocent... :) ) The wait times are outrageous... truss -d -D -f -o bill3.out lsnrctl status <listener_name> Base time stamp: 1323722672.7062 [ Mon Dec 12 15:44:32 EST 2011 ] 9898: 0.0000 0.0000 execve("{ORACLE_HOME}/bin/lsnrctl", 0xFFFFFFFF7FFFF368, 0xFFFFFFFF7FFFF388) argc = 3 9898: 80.3434 80.3434 resolvepath("/usr/lib/sparcv9/ld.so.1", "/lib/sparcv9/ld.so.1", 1023) = 20 9898: 116.6389 36.2955 resolvepath("{ORACLE_HOME}/bin/lsnrctl", "{ORACLE_HOME}/bin/lsnrctl", 1023) = 46 9898: 116.6403 0.0014 stat("{ORACLE_HOME}/bin/lsnrctl", 0xFFFFFFFF7FFFEF60) = 0 9898: 569.7705 453.1302 open("/var/ld/64/ld.config", O_RDONLY) Err#2 ENOENT -- //www.freelists.org/webpage/oracle-l