RE: OT - Blog entry on hugepages

From: Yong Huang <yong321@xxxxxxxxx>
To: oracle-l@xxxxxxxxxxxxx
Date: Fri, 6 Apr 2012 11:13:25 -0700 (PDT)
> HugePages_Total: 25000
> HugePages_Free:   6092
> HugePages_Rsvd:   5669
> 
> What I don't get is why some memory is Rsvd but not used sometimes: 
> probably some pool has not grown much and used the memory.

I agree. The reserved pages will be gradually used in the lifetime of the 
Oracle instance. In your case, you'll have 6092-5669=423 pages (846 MB if 
2M page size) that will be wasted, i.e. never be used. I would decrease 
that down to at most 10 pages. On Linux, you can do it without shutting 
down anything by "echo 24587 > /proc/sys/vm/nr_hugepages" (24587 is 25000 
minus 413).  Of course, echo'ing a number bigger than already configured 
(25000 here) may not work unless the server was just rebooted.

Here's an easy way to understand the HugePages entries in /proc/meminfo.

UUUUUFFFF <-- Total split into really used (U) and free (F)
UUUUURRR. <-- Total split into really used (U), reserved (R) and really free (.)

If one letter or dot is one HugePage, the above says

HugePages_Total: 9
HugePages_Free:  4
HugePages_Rsvd:  3

and you'll have 4-3=1 page completely wasted.

> I was wondering if there's any other way to truly see it [HugePages being 
> used].

`strace -f -e trace=process sqlplus / as sysdba' and startup Oracle. Look 
for SHM_HUGETLB in 3rd arg to shmget(). You can also save it to a file to 
search for it (-o filename). Unfortunately, you can only see the HUGETLB 
tag during instance startup, not after.

I just set up HugePages on my Red Hat 6 running Oracle. /proc/<pid>/smaps 
has the page size info:

# cat /proc/<any pid of Oracle instance>/smaps
...
60000000-61000000 rwxs 00000000 00:0c 1114116                            
/SYSV00000000 (deleted)
Size:              16384 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:     2048 kB
MMUPageSize:        2048 kB

[repeated for the other 2 shared memory segments]

You see the KernelPageSize and MMUPageSize with 2M page size. All other map 
entries show 2K page size. So now you have one more way to verify. But as 
of today, only Oracle Linux is certified and I don't have it to test. If 
you really want, I think you can apply the smaps Linux patch (see 
http://lkml.indiana.edu/hypermail/linux/kernel/0810.2/0226.html) to have 
that info in smaps. In the future, we may have a -s option for pmap on 
Linux to show page size just like that on Solaris (see 
//www.freelists.org/post/procps/RFE-Show-the-page-size-used-for-an-allocation-in-the-pmap).

Yong Huang
--
//www.freelists.org/webpage/oracle-l
RE: OT - Blog entry on hugepages

Other related posts:
