Extending LuaJIT's memory limit to 4Gbytes.

  • From: "Robert G. Jakabosky" <bobby@xxxxxxxxxxxxxxx>
  • To: luajit@xxxxxxxxxxxxx
  • Date: Mon, 20 Aug 2012 00:36:05 -0700

As an experiment I created a mmap wrapper [1] library to get around the 1Gbyte 
memory limit when running LuaJIT on Linux (this might work on other Non-
windows systems).

The wrapper library takes over management of as much of the lower 4Gbytes of 
address space in the process as it can.  To do this it blocks the system's 
malloc from using sbrk/brk to expand the programs data segment and uses a 
custom page allocator to manage which pages of the lower 4Gbytes are free.

When LuaJIT calls mmap to request memory with the MAP_32BIT flag, the wrapper 
will use the page allocator to find the first segment of unused address space 
that is large enough and then call the system's mmap with that address as a 
hint.  Sometimes LuaJIT provides it's own address hint to mmap (when 
allocating space for machine code in the JIT), the wrapper will check with the 
page allocator to make sure that space is available.

The page allocator uses a simple linked list of free segments of address space 
and uses a first-fit allocation algorithm, since free segments are kept 
sorted, it will always allocate the lowest segment that is large enough.  The 
worst-case overhead of the allocator would be about 16Mbytes where there are 
about 512,000 separate free segments (each being only 4Kbytes in size).  All 
my tests so far have used no more then 300 bytes to track the free segments.

Some caveats:
* The page allocator is currently not thread-safe, so this wrapper can't be 
used by multi-thread code right now.
* There maybe some uses of mmap() that will break.

For testing you can use LD_PRELOAD to enable the wrapper:
LD_PRELOAD="/path/to/mmap_lowmem/libmmap_lowmem.so" luajit-2 <some script>

This way you can try it out with some scripts without forcing all usages of 
luajit to use the wrapper.  Later you can link luajit to the wrapper if you 
always want to use it.

1. https://github.com/Neopallium/mmap_lowmem
Robert G. Jakabosky

Other related posts: