I agree about over allocating. I don't remember where but I read somewhere
about not over allocation. Maybe they just mean by too much. So I got the
recommended value from hugepage_settings.sh and added little but apparently
not enough. Your comments make sense and I have a scheduled reboot next
week so I'll increase it.
Thanks for everyone's help
On Thu, Feb 14, 2019 at 8:51 AM Jose Rodriguez <jrodriguez2@xxxxxxxxxxx>
wrote:
I am curious to know why over allocating HP is a bad idea.
I mean, if you know you will end up increasing the size of an SGA or the
number of SGAs in the server, why not allocate HP ahead of time.
Save some RAM for OS, some for PGA and the rest should go to SGA, that
'rest' should be in HP sooner or later, so do it sooner.
Does it make sense? Or am I missing something?
[image: Pythian] <http://www.pythian.com/>
*Jose Rodriguez* | Oracle Project Engineer | [image: LinkedIn]
<https://www.linkedin.com/company/pythian>
*t* +1 613 565 8696 <+1+613+565+8696> *ext.* 1393
*m* +34 607 55 49 91 <+34+607+55+49+91>
jrodriguez2@xxxxxxxxxxx
*www.pythian.com* <https://www.pythian.com/>
[image: Pythian] <https://www.pythian.com/email-footer-click>
On Thu, 14 Feb 2019 at 17:19, Jeff Chirco <backseatdba@xxxxxxxxx> wrote:
Hi Andy, yes I did reboot. Oh I originally set it to 81000 and then--
rebooted. Yesterday I tried to increase it to 9100 without a reboot.
Since it requires a reboot then that kind of is a bummer if I ever need
to add another database to a server. I'd have to bring everything down to
increase hugepages. I could over allocate hugepages but I read that is a
bad idea.
Jeff
On Thu, Feb 14, 2019 at 8:05 AM Andy Wattenhofer <watt0012@xxxxxxx>
wrote:
Did you reboot after the last change to hugepages?
[root]# grep nr_hugepages /etc/sysctl.conf
vm.nr_hugepages=*91000*[...]
[root]# cat /proc/meminfoHugePages_Free: 8
HugePages_Total: *81793*
HugePages_Rsvd: 7
HugePages_Surp: 0
Yes, you must reboot for 91000 to take effect. The huge page RAM is
reserved at boot time.
Andy
On Thu, Feb 14, 2019 at 9:15 AM Jeff Chirco <backseatdba@xxxxxxxxx>
wrote:
Thanks for the replies. Forgot to mention this is Oracle Linux 7
running 12.2.0.1. These are output from my Development server but
production is identicial except hugepage size is lower cause it has less
database.
Development and Production both have 256gb of RAM.
I also noticed when I first set huegpages using the below command it
increased hugepages but not to the amount I tried to set it to. Not until a
server reboot did it get set the full amount. If I need to increase it
again I would prefer to not have to reboot the server.
# sysctl -w vm.nr_hugepages=value
Regarding hugepages_settings.sh I did follow Tim's notes but I ended up
finding an updated script under Doc ID 401749.1
[root]# grep memlock /etc/security/limits.conf
# - memlock - max locked-in-memory address space (KB)
oracle hard memlock 237265300
# oracle-rdbms-server-11gR2-preinstall setting for memlock hard limit
is maximum of 128GB on x86_64 or 3GB on x86 OR 90 % of RAM
oracle soft memlock 237265300
[root]# grep nr_hugepages /etc/sysctl.conf
vm.nr_hugepages=91000
[root]# ulimit -l
64
[root]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1029648
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1029648
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Alert log from one of my databases
**********************************************************************
2019-02-12T21:26:58.786205-08:00
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
2019-02-12T21:26:58.790788-08:00
Per process system memlock (soft) limit = 226G
2019-02-12T21:26:58.793885-08:00
Expected per process system memlock (soft) limit to lock
SHARED GLOBAL AREA (SGA) into memory: 8196M
2019-02-12T21:26:58.800179-08:00
Available system pagesizes:
4K, 2048K
2019-02-12T21:26:58.806848-08:00
Supported system pagesize(s):
2019-02-12T21:26:58.809846-08:00
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
2019-02-12T21:26:58.813115-08:00
2048K 65468 4098 4098 NONE
2019-02-12T21:26:58.816156-08:00
Reason for not supporting certain system pagesizes:
2019-02-12T21:26:58.819060-08:00
4K - Large pagesizes only
2019-02-12T21:26:58.822058-08:00
**********************************************************************
[root]# ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 589824 oracle 600 10485760 78
0x00000000 622593 oracle 600 3372220416 78
0x00000000 655362 oracle 600 8388608 78
0x76c55318 688131 oracle 600 2097152 78
0x00000000 11862020 oracle 600 10485760 120
0x00000000 11894789 oracle 600 3204448256 120
0x00000000 11927558 oracle 600 8388608 120
0x34ef509c 11960327 oracle 600 2097152 120
0x00000000 11993096 oracle 600 10485760 95
0x00000000 12025865 oracle 600 2298478592 95
0x00000000 12058634 oracle 600 8388608 95
0x507a78a4 12091403 oracle 600 2097152 95
0x00000000 12124172 oracle 600 10485760 207
0x00000000 12156941 oracle 600 6425673728 207
0x00000000 12189710 oracle 600 8388608 207
0x866b8aa0 12222479 oracle 600 2097152 207
0x00000000 12255248 oracle 600 12582912 81
0x00000000 12288017 oracle 600 8573157376 81
0x00000000 12320786 oracle 600 6291456 81
0x0c51f55c 12353555 oracle 600 2097152 81
0x00000000 12386324 oracle 600 12582912 79
0x00000000 12419093 oracle 600 8573157376 79
0x00000000 12451862 oracle 600 6291456 79
0x7afafd64 12484631 oracle 600 2097152 79
0x00000000 12779544 oracle 600 12582912 90
0x00000000 12812313 oracle 600 8573157376 90
0x00000000 12845082 oracle 600 6291456 90
0xdde48098 12877851 oracle 600 2097152 90
0x00000000 12910620 oracle 600 12582912 86
0x00000000 12943389 oracle 600 8573157376 86
0x00000000 12976158 oracle 600 6291456 86
0xbae4e74c 13008927 oracle 600 2097152 86
0x00000000 13303840 oracle 600 12582912 85
0x00000000 13336609 oracle 600 8573157376 85
0x00000000 13369378 oracle 600 6291456 85
0x4c7ac944 13402147 oracle 600 2097152 85
0x00000000 13434916 oracle 600 12582912 86
0x00000000 13467685 oracle 600 8573157376 86
0x00000000 13500454 oracle 600 6291456 86
0xf06d661c 13533223 oracle 600 2097152 86
0x00000000 13828136 oracle 600 12582912 87
0x00000000 13860905 oracle 600 8573157376 87
0x00000000 13893674 oracle 600 6291456 87
0xdb5acf68 13926443 oracle 600 2097152 87
0x00000000 13959212 oracle 600 12582912 87
0x00000000 13991981 oracle 600 8573157376 87
0x00000000 14024750 oracle 600 6291456 87
0xcdd9f634 14057519 oracle 600 2097152 87
0x00000000 14352432 oracle 600 10485760 102
0x00000000 14385201 oracle 600 6425673728 102
0x00000000 14417970 oracle 600 8388608 102
0x4bd6517c 14450739 oracle 600 2097152 102
0x00000000 14483508 oracle 600 10485760 116
0x00000000 14516277 oracle 600 6425673728 116
0x00000000 14549046 oracle 600 8388608 116
0x33ec8074 14581815 oracle 600 2097152 116
0x00000000 14876728 oracle 600 12582912 87
0x00000000 14909497 oracle 600 8573157376 87
0x00000000 14942266 oracle 600 6291456 87
0xb40541dc 14975035 oracle 600 2097152 87
0x00000000 1179975740 oracle 600 12582912 90
0x00000000 1180008509 oracle 600 8455716864 90
0x00000000 1180041278 oracle 600 34359738368 90
0x00000000 1180074047 oracle 600 123731968 90
0x00000000 15401024 oracle 600 12582912 91
0x00000000 15433793 oracle 600 8573157376 91
0x00000000 15466562 oracle 600 6291456 91
0x36329238 15499331 oracle 600 2097152 91
0x00000000 15532100 oracle 600 12582912 118
0x00000000 15564869 oracle 600 8573157376 118
0x00000000 15597638 oracle 600 6291456 118
0xa0206b88 15630407 oracle 600 2097152 118
0x00000000 16449608 oracle 600 12582912 86
0x00000000 16482377 oracle 600 8573157376 86
0x00000000 16515146 oracle 600 6291456 86
0xc402bb18 16547915 oracle 600 2097152 86
0x00000000 16318540 oracle 600 12582912 87
0x00000000 16351309 oracle 600 8573157376 87
0x00000000 16384078 oracle 600 6291456 87
0x261e431c 16416847 oracle 600 2097152 87
0x00000000 16187472 oracle 600 12582912 168
0x00000000 16220241 oracle 600 20937965568 168
0x00000000 16253010 oracle 600 56623104 168
0x35b8fa94 16285779 oracle 600 2097152 168
0x00000000 547127380 oracle 600 12582912 85
0x00000000 547160149 oracle 600 251658240 85
0x00000000 547192918 oracle 600 8321499136 85
0x00000000 547225687 oracle 600 4595712 85
0x9cd83398 547258456 oracle 600 40960 85
0x00000000 792199257 oracle 600 10485760 106
0x00000000 792232026 oracle 600 1644167168 106
0x00000000 792264795 oracle 600 4781506560 106
0x00000000 792297564 oracle 600 8388608 106
0xc78a1c8c 792330333 oracle 600 32768 106
0x6762e034 1180106846 oracle 600 28672 90
[root]# cat /proc/meminfo
MemTotal: 263623524 kB
MemFree: 2169856 kB
MemAvailable: 25248596 kB
Buffers: 28 kB
Cached: 68234496 kB
SwapCached: 4392 kB
Active: 56288772 kB
Inactive: 32305392 kB
Active(anon): 38139416 kB
Inactive(anon): 28588332 kB
Active(file): 18149356 kB
Inactive(file): 3717060 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16777212 kB
SwapFree: 16732780 kB
Dirty: 1020 kB
Writeback: 0 kB
AnonPages: 20491988 kB
Mapped: 10605612 kB
Shmem: 46232620 kB
Slab: 2937544 kB
SReclaimable: 2197872 kB
SUnreclaim: 739672 kB
KernelStack: 68544 kB
PageTables: 1451148 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 64832940 kB
Committed_AS: 93046052 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 772444 kB
VmallocChunk: 34358900732 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 81793
HugePages_Free: 8
HugePages_Rsvd: 7
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 514768 kB
DirectMap2M: 41201664 kB
DirectMap1G: 228589568 kB
On Thu, Feb 14, 2019 at 6:50 AM Jeff Chirco <backseatdba@xxxxxxxxx>
wrote:
Ooops sorry yes Oracle Linux 7.4
On Wed, Feb 13, 2019 at 4:31 PM Paul Drake <bdbafh@xxxxxxxxx> wrote:
OS info might be relevant.
Assuming Linux:
ipcs -m
cat /proc/meminfo
On Wed, Feb 13, 2019, 7:12 PM Jeff Chirco <backseatdba@xxxxxxxxx
wrote:
Ok I just recently enabled hugepages on Production and Development.
Everything is going good but I noticed that on my development
environment
which consists of a bunch of thin clones, when I remove and recreate the
same clone the database fails to create, I am getting:
ORA-27137: unable to allocate large pages to create a shared memory
segment
I have the parameter *use_large_pages=ONLY*
So it makes sense I would get this error when there is not enough
pages. However I just removed this same database and am recreating it. I
would think that would free it up but apparently not. Does pages not
get
released immediately when a database is removed? Is there something I
should run to free it? Or just not have use_large_pages=ONLY and leave
it
at TRUE on Dev? I had to set it to TRUE to get this database up.
Thanks,
Jeff