Re: Questions on Memory Address Space under Linux x64

  • From: Mike Pall <mike-1207@xxxxxxxxxx>
  • To: luajit@xxxxxxxxxxxxx
  • Date: Wed, 4 Jul 2012 15:07:29 +0200

Armin Pies wrote:
> In several posts I have read of a Luajit memory limit of 1 GB in
> the lowest 2GB of the process address space.

The actual limit is 2GB. The 1GB is due to a hardcoded Linux
kernel limit for the MAP_32BIT feature. Adventurous minds may
consider patching the kernel.

> We have a 64bit server application which highly uses memory mapping of many 
> big files (often greater than 4GB).
> 
> On the other hand we use Luajit to call Lua scripts from that application.
> 
> To avoid "out of memory" by Luajit we use light userdata.

Why lightuserdata? A userdata object may carry a pointer to the
mapping. That's a very small object, even if the mapping is big.

> 1. But how can we ensure (or is - better - can it be done by
> Luajit itself?) that the needed address space is not already
> occupied by another component/object  in our application?

The default mmap() code in the Linux/x64 kernel never allocates
from the low 4GB, so you should be safe.

> 2. Is it possible that luajit does not acquire the 1 GB address space at once?

IMHO neither necessary, nor very embedding-friendly.

> 3. Do all push functions use that 1 GB address space for storing
> the Luajit data or do any of them use a kind of indirection,
> e.g. storing strings by a pointer in the 1GB, which reference
> the "real" string data in an extra memory region controlled by
> Luajit?

Yes. No.

In any case, you should take a look at the FFI: it has 64 bit
pointers and allows you to use the full x64 address space. And you
can call mmap() from your Lua code (no C code needed).

--Mike

Other related posts: