Re: Extending LuaJIT's memory limit to 4Gbytes.

  • From: "Robert G. Jakabosky" <bobby@xxxxxxxxxxxxxxx>
  • To: luajit@xxxxxxxxxxxxx
  • Date: Tue, 21 Aug 2012 02:46:07 -0700

On Tuesday 21, Dmitri Shubin wrote:
> Hi Robert!
> 
> On 20.08.2012 11:36, Robert G. Jakabosky wrote:
> > As an experiment I created a mmap wrapper [1] library to get around the
> > 1Gbyte memory limit when running LuaJIT on Linux (this might work on
> > other Non- windows systems).
> > 
> > The wrapper library takes over management of as much of the lower 4Gbytes
> > of address space in the process as it can.  To do this it blocks the
> > system's malloc from using sbrk/brk to expand the programs data segment
> > and uses a custom page allocator to manage which pages of the lower
> > 4Gbytes are free.
> 
> Just FYI.
> 
> To support LuaJIT on Solaris 64-bit I had to do the similar thing -- use
> custom page allocator for low 2GiB of address space.


Is there some restriction keeping you from using up to 4Gbytes of address 
space?


> (mmap() on Solaris doesn't have MAP_32BIT flag and doesn't honor address
> hint)
> I "shift" executable as high as possible (by setting base address to
> 2GiB - 256MiB),

Can you build PIC executables [1] on Solaris?

I just figured out how to move LuaJIT & the program brk from the start of the 
address space to a high address (at some random place above the low 4G)

make all LDFLAGS=" -pie " CFLAGS=" -fPIC "

I haven't done any benchmarking to see if this causes any slow downs.  I was 
surprised that it didn't cause any problems with the JIT and that the JIT 
started using high pages for the machine code.  So far it seems to work just 
fine.

With this + my mmap wrapper LuaJIT can use a full 4Gbytes (minus the first few 
pages).

> "reserve" this space using MAP_NORESERVE and used vmem
> resource allocator from umem library [1][2] to allocate pages from it.

Are you "reserving" all of the low 2Gb with mmap?  How do you return pages 
back to the OS?  Does vmem provide a way to allocate at a fixed address?  If 
not how did you handle the mmap call at src/lj_mcode.c:101?

Originally I had reserved as much of the low 4Gbytes as possible using mmap, 
then used dlmalloc [2][3] with a custom sbrk function to manage allocations 
from that range.  The custom sbrk function had to munmap then remap pages to 
release them back to the OS.  It was very hacky and couldn't handle address 
hints.  You can see the old version at [3].

The current page allocator only tracks which pages are free, then the wrapper 
does a normal mmap/munmap/mremap system call with the free address as a hint.

> But I modified LuaJIT code instead of LD_PRELOAD'ing.

Preloading just makes it easier to test LuaJIT with & without the mmap 
wrapper.  It also works when LuaJIT is linked directly with the library.

I am thinking about exporting a set of functions that LuaJIT can call directly 
to allocate/free low pages.  Then only wrap mmap to detect any calls that get 
low pages from the OS (i.e. when using address hints or the MAP_32BIT flag).


1. http://en.wikipedia.org/wiki/Position-independent_code#Position-
independent_executables
2. http://g.oswego.edu/dl/html/malloc.html
3. 
https://github.com/Neopallium/mmap_lowmem/tree/6c9055bb722e0e2fc4bd7201f7c1a885a64b2471
 

-- 
Robert G. Jakabosky

Other related posts: