Re: Questions on Memory Address Space under Linux x64

  • From: Armin Pies <armin.pies@xxxxxxxxxxx>
  • To: "luajit@xxxxxxxxxxxxx" <luajit@xxxxxxxxxxxxx>
  • Date: Thu, 5 Jul 2012 08:47:51 +0200

You say that our memory mapping is no problem. But what's about other memory 
allocation via new, STL allocators, boost etc.? How can we ensure that the 1GB 
of the 32bit address space is available for Luajit?

Am 04.07.2012 um 15:10 schrieb "Mike Pall" <mike-1207@xxxxxxxxxx>:

> Armin Pies wrote:
>> In several posts I have read of a Luajit memory limit of 1 GB in
>> the lowest 2GB of the process address space.
> 
> The actual limit is 2GB. The 1GB is due to a hardcoded Linux
> kernel limit for the MAP_32BIT feature. Adventurous minds may
> consider patching the kernel.
> 
>> We have a 64bit server application which highly uses memory mapping of many 
>> big files (often greater than 4GB).
>> 
>> On the other hand we use Luajit to call Lua scripts from that application.
>> 
>> To avoid "out of memory" by Luajit we use light userdata.
> 
> Why lightuserdata? A userdata object may carry a pointer to the
> mapping. That's a very small object, even if the mapping is big.
> 
>> 1. But how can we ensure (or is - better - can it be done by
>> Luajit itself?) that the needed address space is not already
>> occupied by another component/object  in our application?
> 
> The default mmap() code in the Linux/x64 kernel never allocates
> from the low 4GB, so you should be safe.
> 
>> 2. Is it possible that luajit does not acquire the 1 GB address space at 
>> once?
> 
> IMHO neither necessary, nor very embedding-friendly.
> 
>> 3. Do all push functions use that 1 GB address space for storing
>> the Luajit data or do any of them use a kind of indirection,
>> e.g. storing strings by a pointer in the 1GB, which reference
>> the "real" string data in an extra memory region controlled by
>> Luajit?
> 
> Yes. No.
> 
> In any case, you should take a look at the FFI: it has 64 bit
> pointers and allows you to use the full x64 address space. And you
> can call mmap() from your Lua code (no C code needed).
> 
> --Mike
> 

Other related posts: