RE: Excessive IOPS to shared ORACLE_HOME

  • From: "Vasu Balla" <appsdba@xxxxxxxxx>
  • To: <riyaj.shamsudeen@xxxxxxxxx>, <cshapi@xxxxxxxxx>
  • Date: Fri, 22 Aug 2008 07:47:00 +0200

Oracle on netapp rings some bells for me. Oracle recommends using noac or
actime=o options when mounting nfs for Datafiles, Voting Disk and OCR. Noac
means "no attribute cache" means none of the file attributes are cached in
the filesystem cache, which  is very much needed for RAC. If you put your
shared oracle home also in that mountpoint which is mounted noac, every
access to a file in the oracle home requires a physical IO at the netapp. So
I recommend moving all software directories ( db oracle home, asm oracle
home and crs oracle home etc ) to a nfs mount which is not mounted with noac
or actime=o. This will reduce your excessive IO per sec issue


Vasu Balla
Apps DBA - The Pythian Group
www.pythian.com
-----Original Message-----
From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx]
On Behalf Of Riyaj Shamsudeen
Sent: Friday, August 22, 2008 5:18 AM
To: cshapi@xxxxxxxxx
Cc: oracle-l@xxxxxxxxxxxxx
Subject: Re: Excessive IOPS to shared ORACLE_HOME

Chen
  I have seen similar issue in Solaris platform. Can you see any difference
as to how Oracle_home file system is mounted between these servers ? It's a
common fallacy that Oracle home binaries are never accessed once  the
database is up and running. In reality, Oracle binaries are accessed
constantly and if the file system holding oracle_home is mounted with mount
options bypassing UNIX buffer cache, then IOPS to underlying file system
increases sharply. So, you might want to check all your DB nodes and make
sure Oracle home file system is using unix buffer cache.
  There are other reasons too. Enormous amount of audit files, logging in
listener log, CRS spitting out log entries etc. Check that out too.

Cheers
Riyaj Shamsudeen
The Pythian Group - http://www.pythian.com Personal blog:
http://orainternals.wordpress.com
 
Chen Shapira wrote:
> Hi Oracle Fans,
>
> I need some help and advice on tracking down source of excessive IO.
>
> Our 10.2.0.[2,3] RAC enviroments are configured with shared 
> ORACLE_HOME, residing on Netapp storage. The same volume also contains 
> the voting disk and OCR for the RAC.
> This volume, that we assumed will have almost no load, actually has 
> about 6000 IOPS (IO operations per second), sometimes up to 13000. For 
> comparison, our data volume has about 1500 IOPS.
>
> Interesting findings:
> 1) From Netapp reports and from some network sniffing, it appears that 
> over 90% of the IO is "access" and "get attribute" operations - 
> neither reads, nor writes.
> 2) This does not happen to the voting disks on RAC systems that have 
> no shared home. (although there are other configuration differences as 
> well).
> 3) This also does not happen in stand alone systems that have 
> ORACLE_HOME on the Netapp.
>
> I can't use IOSTAT to debug because it does not report NFS. Network 
> monitoring aggregates over all NFS traffic, so I can't seperate normal 
> data traffic from strange ORACLE_HOME traffic (it is all on same 
> storage interface). Linux does not report network activity per process 
> (not without special patches). Wait event tables will not show traffic 
> to Oracle_home or to voting disk.
>
> Any suggestions where to look?
> Anyone can guess what we could have configured wrong to cause such an 
> issue? ( I know that we are all against random guessing, but it feels 
> like all investigation channels are insufficient).
>
> Thanks,
> Chen Shapira
> --
> //www.freelists.org/webpage/oracle-l
>
>
>
>   

--
//www.freelists.org/webpage/oracle-l


--
//www.freelists.org/webpage/oracle-l


Other related posts: