Re: ffi.new intermittent failures
- From: Alexander Gladysh <agladysh@xxxxxxxxx>
- To: LuaJIT Mailing List <luajit@xxxxxxxxxxxxx>
- Date: Thu, 14 Nov 2013 13:19:59 +0400
On Thu, Nov 14, 2013 at 12:43 PM, Justin Cormack <
justin@xxxxxxxxxxxxxxxxxxxxx> wrote:
>
> On 14 Nov 2013 08:40, "Alexander Gladysh" <agladysh@xxxxxxxxx> wrote:
>>
>> On Thu, Nov 14, 2013 at 12:35 PM, Justin Cormack
>> <justin@xxxxxxxxxxxxxxxxxxxxx> wrote:
>> >
>> > On 14 Nov 2013 08:30, "Alexander Gladysh" <agladysh@xxxxxxxxx> wrote:
>> >>
>> >> Hi, list!
>> >>
>> >> We observe intermittent failures in ff.new, when attempting to
>> >> allocate a largeish chunk of memory on Linux (see below). C calls like
>> >> calloc work fine.
>> >>
>> >> Any clues?
>> >
>> > Look at the allocation code. For x64 it uses mmap with hints and needs
>> > to
>> > get low 2G addresses which can fail. Best to call mmap yourself for
huge
>> > allocations and allocate over 2G.
>>
>> Erm. What is the reason behind this implementation? That's FFI
>> allocation, not one for internal Lua works.
>>
>
> Erm maybe I am getting mixed up. What does strace say is happening?
read(0, =ffi.typeof('char[1000000000]')()
"=ffi.typeof('char[1000000000]')("..., 1024) = 34
rt_sigaction(SIGINT, {0x403980, [INT], SA_RESTORER|SA_RESTART,
0x7f16cdeb74a0}, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x7f16cdeb74a0},
8) = 0
mmap(NULL, 1000001536, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0) = 0x41633000
rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART,
0x7f16cdeb74a0}, {0x403980, [INT], SA_RESTORER|SA_RESTART, 0x7f16cdeb74a0},
8) = 0
write(1, "cdata<char [1000000000]>: 0x4163"..., 37cdata<char [1000000000]>:
0x41633018
) = 37
write(1, "> ", 2> ) = 2
read(0, =ffi.typeof('char[1000000000]')()
*"=ffi.typeof('char[1000000000]')("..., 1024) = 34rt_sigaction(SIGINT,
{0x403980, [INT], SA_RESTORER|SA_RESTART, 0x7f16cdeb74a0}, {SIG_DFL, [INT],
SA_RESTORER|SA_RESTART, 0x7f16cdeb74a0}, 8) = 0 mmap(NULL, 1000001536,
PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0) = -1
ENOMEM (Cannot allocate memory)mmap(NULL, 1000079360, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0) = -1 ENOMEM (Cannot allocate
memory) rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART,
0x7f16cdeb74a0}, {0x403980, [INT], SA_RESTORER|SA_RESTART, 0x7f16cdeb74a0},
8) = 0munmap(0x41633000, 1000001536) = 0write(2, "not enough
memory\n", 18not enough memory ) = 18*write(1, "> ", 2> )
= 2
read(0, =ffi.typeof('char[1000000000]')()
"=ffi.typeof('char[1000000000]')("..., 1024) = 34
rt_sigaction(SIGINT, {0x403980, [INT], SA_RESTORER|SA_RESTART,
0x7f16cdeb74a0}, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x7f16cdeb74a0},
8) = 0
mmap(NULL, 1000001536, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0) = 0x410b5000
rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART,
0x7f16cdeb74a0}, {0x403980, [INT], SA_RESTORER|SA_RESTART, 0x7f16cdeb74a0},
8) = 0
write(1, "cdata<char [1000000000]>: 0x410b"..., 37cdata<char [1000000000]>:
0x410b5018
) = 37
write(1, "> ", 2> ) = 2
read(0, =ffi.typeof('char[1000000000]')()
"=ffi.typeof('char[1000000000]')("..., 1024) = 34
rt_sigaction(SIGINT, {0x403980, [INT], SA_RESTORER|SA_RESTART,
0x7f16cdeb74a0}, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART, 0x7f16cdeb74a0},
8) = 0
mmap(NULL, 1000001536, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0) = -1 ENOMEM (Cannot allocate
memory)
mmap(NULL, 1000079360, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_ANONYMOUS|MAP_32BIT, -1, 0) = -1 ENOMEM (Cannot allocate
memory)
rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTORER|SA_RESTART,
0x7f16cdeb74a0}, {0x403980, [INT], SA_RESTORER|SA_RESTART, 0x7f16cdeb74a0},
8) = 0
munmap(0x410b5000, 1000001536) = 0
write(2, "not enough memory\n", 18not enough memory
) = 18
write(1, "> ", 2> ) = 2
HTH,
Alexander.
Other related posts:
