Re: Solaris 10 and Oracle 10g - swap space problem

  • From: David Miller <David.J.Miller@xxxxxxx>
  • To: riyaj.shamsudeen@xxxxxxxxx
  • Date: Mon, 03 Aug 2009 09:21:02 -0500

Hi Riyaj,

I think the issue here is with understanding what the data from "swap -s" means.

Here's from one of my internal systems:

# swap -l
swapfile             dev  swaplo blocks   free
/dev/dsk/c0t1d0s1   32,9      16 8201840 8201840
/dev/md/dsk/d209    85,209     16 67108848 67108848
/dev/md/dsk/d210    85,210     16 67108848 67108848
/dev/md/dsk/d211    85,211     16 67108848 67108848
/dev/md/dsk/d212    85,212     16 67108848 67108848
/dev/md/dsk/d213    85,213     16 67108848 67108848

# sys
XXX   08/03/09   10:02:16   SunOS 5.10 Generic_139555-04   sun4u
2520/0 MHz   8 CPUs   32.00 GB memory   SUNW,SPARC-Enterprise

# swap -s
total: 71280k bytes allocated + 12368k reserved = 83648k used, 200001416k 
available

So the 200001416k is equal to the 32 * 5 + 4 GB in actual swap files plus
32 GB in memory minus the kernel space (and whatever is in /tmp; I didn't
check).

When I start oracle here:

Total System Global Area 4932501504 bytes
Fixed Size                  2047312 bytes
Variable Size             620757680 bytes
Database Buffers         4294967296 bytes
Redo Buffers               14729216 bytes
Database mounted.
Database opened.

# pmap -xs 9020 | grep -i ism
0000000380000000 4816896 4816896 - 4816896 4M rwxsR [ ism shmid=0x1 ] 00000004A6000000 16 16 - 16 8K rwxsR [ ism shmid=0x1 ]

So it's using ISM.

bur834-98# swap -s
total: 5140592k bytes allocated + 33968k reserved = 5174560k used, 194842040k available

So now the available space is about 6G less, which is pretty much the total
of the ~5G SGA plus swap space for the oracle processes.

If I set "sga_max_size = 7159641169" and restart:

Total System Global Area 7163871232 bytes
Fixed Size                  2051616 bytes
Variable Size            2852131296 bytes
Database Buffers         4294967296 bytes
Redo Buffers               14721024 bytes
Database mounted.
Database opened.

# ps -ef | grep lgwr
  oracle 11586     1   0 10:10:47 ?           0:00 ora_lgwr_XXX

bur834-98# pmap -xs 11586 | grep -i dism
0000000380000000 16384 16384 - - 4M rwxs- [ dism shmid=0x3 ] 0000000381000000 2191360 12288 - - - rwxs- [ dism shmid=0x3 ]
...

So it's now using DISM instead.

# swap -s
total: 5144680k bytes allocated + 2212928k reserved = 7357608k used, 187819608k available

Now the total is reduced by an additional 7G to save space for the DISM backing.
Note the other memory use hasn't changed, so the size in memory remains the
same, about 5G.  There's just extra swap space allocated.

So things are working as expected; it's just the interpretation of the "swap -s"
that's causing problems.

Regards,

Dave Miller

Riyaj Shamsudeen wrote, On 08/01/09 12:01:
Tanel
I must disagree with you. It looks like swap is allocated even for ISM pages in Solaris. Please feel free to correct me if there is any misunderstanding from my part.

Here is a small test case:

RS: Intially, 7.4G of swap used
swap -s
total: 7207256k bytes allocated + 233128k reserved = 7440384k used, 40446656k available

RS: ISM in use and SGA is locked as expected.
Address Kbytes RSS Anon Locked Mode Mapped File
..
0000000380000000 6422544 6422544 - 6422544 rwxsR [ ism shmid=0x300001c ]

RS: No sga_max_size set in spfile either.

strings  spfilePROD.ora |grep -i sga
*.sga_target=6272M

RS: Shutting down...
sqlplus "/ as sysdba"

Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>

RS: Just 500M of swap used after instance shutdown..
wsqfinc1e> swap -s
total: 451224k bytes allocated + 63928k reserved = 515152k used, 47400568k available

RS: Restarting instance..
ORACLE instance started.

Total System Global Area 6576668672 bytes
Fixed Size                  2050464 bytes
Variable Size            2583696992 bytes
Database Buffers         3976200192 bytes
Redo Buffers               14721024 bytes
Database mounted.
Database opened.

RS: Swap is back to 7.32GB
swap -s
total: 7135384k bytes allocated + 191968k reserved = 7327352k used, 40566696k available

RS: Still ISM..
Address Kbytes RSS Anon Locked Mode Mapped File
...
0000000380000000 6422544 6422544 - 6422544 rwxsR [ ism shmid=0x3000048 ]

Even though, ISM is in use and sga_max_size is not in use, swap is allocated.


Cheers

Riyaj Shamsudeen
Principal DBA,
Ora!nternals -  http://www.orainternals.com
Specialists in Performance, Recovery and EBS11i
Blog: http://orainternals.wordpress.com


On Sat, Aug 1, 2009 at 12:51 AM, Tanel Poder <tanel@xxxxxxxxxx <mailto:tanel@xxxxxxxxxx>> wrote:

    If you use ISM (when SGA_MAX_SIZE = SGA_TARGET or when SGA_MAX_SIZE
    is unset in parameter file) then all SGA pages are locked in
    physical memory (by OS VM design) - thus no swap space needs to be
    reserved as the pages in memory aren't pageable.

    If you are using DISM (when SGA_MAX_SIZE > SGA_TARGET) then SGA is
    using pageable large pages in Solaris (which are individually locked
    in physical memory by Oracle's mlock requests). But as the pages are
    essentially pageable, swap space needs to be reserved when SGA is
    created.

    If you want to avoid your SGA causing the swap space reservation,
    unset your SGA_MAX_SIZE parameter.

    I have mentioned this on this blog entry (and in the comments
    section there's a link to a Solaris kernel internals presentation):

    
http://blog.tanelpoder.com/2007/08/28/operating-systems-are-lazy-allocating-memory/

    --
    Tanel Poder
    http://blog.tanelpoder.com



    On Wed, Jul 29, 2009 at 7:31 AM, Crisler, Jon <Jon.Crisler@xxxxxxx
    <mailto:Jon.Crisler@xxxxxxx>> wrote:

        On Solaris 10, when I start my databases, I see my swap space
usage increase in a one for one ratio with memory utilization. System has 64gb of memory. If I start up a 4gb database, I see
        4gb of real memory used and 4gb of swap space used up as well
        (as shown by top, swap –l, swap –s etc.).     This is purely a
        Oracle database server, and /etc/system and projects / resource
        controls seem to be set up, so I do not understand why swap is
        being touched.  If I stop all oracle processes, swap drops down
        to about 100 Mb used.

        I have taken great pains to make sure that NO swap is being
        used, yet any little allocation of Oracle gets tossed into
        swap.  I don’t know where to look after this- I would have
        suspected /etc/system or resource control / projects to be at
fault, or /etc/security (ulimits) but everything looks ok. The system has 64g real, and about 54 gb swap- if I get close
        to 54gb memory allocated oracle will return out of memory
        errors..  Needless to say the system crawls.

        What am I overlooking ?  TOP and swap –s / swap –l agree so its
        not a TOP anomaly

        Example from TOP – I have 20g free, so why so much used in swap ?

load averages: 0.64, 0.57, 0.46 00:26:34

        1073 processes:1071 sleeping, 2 on cpu

CPU states: 98.3% idle, 1.2% user, 0.5% kernel, 0.0% iowait, 0.0% swap

        Memory: 64G real, 20G free, 40G swap in use, 17G swap free

           PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU
        COMMAND

          1997 oracle     1  39    0 4376K 2592K cpu/50   0:02  0.04% top

          3454 oracle     1  52    0   16G   16G cpu/32   0:00  0.03% oracle

          3456 oracle     1  24    0   16G   16G sleep    0:00  0.03%
        oracle

         18589 oracle     1  59    0   16G   16G sleep    0:13  0.02% oracle

Swap –s
        total: 39170184k bytes allocated + 2262264k reserved = 41432448k
        used, 18017608k available




-- Tanel Poder
    http://blog.tanelpoder.com


--
//www.freelists.org/webpage/oracle-l


Other related posts: