Extending LuaJIT's memory limit to 4Gbytes.
- From: "Robert G. Jakabosky" <bobby@xxxxxxxxxxxxxxx>
- To: luajit@xxxxxxxxxxxxx
- Date: Mon, 20 Aug 2012 00:36:05 -0700
As an experiment I created a mmap wrapper [1] library to get around the 1Gbyte
memory limit when running LuaJIT on Linux (this might work on other Non-
windows systems).
The wrapper library takes over management of as much of the lower 4Gbytes of
address space in the process as it can. To do this it blocks the system's
malloc from using sbrk/brk to expand the programs data segment and uses a
custom page allocator to manage which pages of the lower 4Gbytes are free.
When LuaJIT calls mmap to request memory with the MAP_32BIT flag, the wrapper
will use the page allocator to find the first segment of unused address space
that is large enough and then call the system's mmap with that address as a
hint. Sometimes LuaJIT provides it's own address hint to mmap (when
allocating space for machine code in the JIT), the wrapper will check with the
page allocator to make sure that space is available.
The page allocator uses a simple linked list of free segments of address space
and uses a first-fit allocation algorithm, since free segments are kept
sorted, it will always allocate the lowest segment that is large enough. The
worst-case overhead of the allocator would be about 16Mbytes where there are
about 512,000 separate free segments (each being only 4Kbytes in size). All
my tests so far have used no more then 300 bytes to track the free segments.
Some caveats:
* The page allocator is currently not thread-safe, so this wrapper can't be
used by multi-thread code right now.
* There maybe some uses of mmap() that will break.
For testing you can use LD_PRELOAD to enable the wrapper:
LD_PRELOAD="/path/to/mmap_lowmem/libmmap_lowmem.so" luajit-2 <some script>
This way you can try it out with some scripts without forcing all usages of
luajit to use the wrapper. Later you can link luajit to the wrapper if you
always want to use it.
1. https://github.com/Neopallium/mmap_lowmem
--
Robert G. Jakabosky
Other related posts:
