Re: Workarounds for LuaJIT's 4GB limit
- From: Nicholas Hutchinson <nick.hutchinson@xxxxxxxxxxxxxxxx>
- To: luajit@xxxxxxxxxxxxx
- Date: Thu, 16 Oct 2014 08:21:08 +0100
Our problem is that we can't even create a new interpreter state --
luaL_newstate() fails.
On 16 Oct 2014, at 00:30, Alex <initrd.gz@xxxxxxxxx> wrote:
> The 4GB limit only applies to Lua objects and cdata allocated with ffi.new or
> from ffi.typeof constructors. If you use malloc directly (via declaring it
> with ffi.cdef then using it with ffi.C.malloc), you can use memory outside of
> the limit.
>
> On Wed, Oct 15, 2014 at 6:28 PM, Nicholas Hutchinson
> <nick.hutchinson@xxxxxxxxxxxxxxxx> wrote:
> Hi all,
>
> A (64-bit Linux) product I'm working on is running into issues relating to:
> - LuaJIT's requirement that the interpreter state reside in the first 4GB of
> your process' address space
> - its default allocator (see lj_alloc.c) getting memory from the system using
> mmap() with the MAP_32BIT flag, which further restricts LuaJIT to using using
> the first 2GB of your process' address space
>
> I gather there are cases where we're trying to initialise a Lua interpreter
> state quite late in the piece, and that this fails as we've exhausted that
> lower 2GB portion of our address space. I'd love to hear other people's
> experiences and how they're working around this limitation. The suggestions
> I've read seem to boil down to:
>
> - switching to a different memory allocator such as tcmalloc or jemalloc --
> tcmalloc can be configured to use mmap() rather than sbrk() to request memory
> from the system, and jemalloc seems to always use mmap(). As I understand it,
> mmap() generally (always?) returns memory from >= 4GB, so the the lower
> portion would then be free for LuaJIT.
> - customising LuaJIT's own memory allocator -- at process startup, allocate
> some suitably large chunk of memory from the < 4GB region and dole it out to
> LuaJIT in chunks.
>
> Am I missing something simpler?
>
> Finally, is there any chance restriction is going to be relaxed in a future
> version of LuaJIT?
>
> Thanks,
> Nick
>
>
>
> --
> Sincerely,
> Alex Parrill
Other related posts:
