i might be totally wrong, but it seems to me you're asking two different things:
1.- why the signal handler isn't called?
2.- why can't I kill the LuaJIT space?
for #1, i can't help much. it's very likely that you're hitting an
implementation difference between different ports of the kernel. the
POSIX standards are way too lax and admit many different
interpretations. worse in parts that enough people consider the
standard "broken", so they might "fix" them differently. yes, even on
the same version of Linux, the architecture port might have different
behaviour.
as for #2: why can't the state be killed?
you ask "My understanding was the VMs are fully isolated. Is this
incorrect?" as any interesting question, the answer is "it depends".
specifically, it depends on the meaning of "fully". being on the same
thread, they share an address space, and (usually) the memory
allocator. to cleanly close the Lua state, lua_close() releases all
memory it holds, but as mentioned, lua_* functions aren't supposed to
be called at any moment.
The first "hard" case I imagine is about JITted code. there's no
guarantee that the state variables are up to date to the latest point.
there's of course enough information [1] for the code, if you could
let it run just a bit more, to return to a "sane" state, but if you
just try to release memory pointed by the Lua_State structure, you'd
either miss the last few objects, or (more likely) end up referencing
garbage pointers (becuase they haven't been updated).
is there any way out? i'm sure you can think of alternative
strategies. with the little i understand of your situation, the two
main ideas i'd explore are:
- don't kill, ask nicely to die: set some "quit" flag, or use sethook
to make the interpreter pause. at that point the interpreter is in a
sane state, and you can lua_close() it. (as suggested by Demi
Obenour)
- increase isolation. mainly by using a different allocator for each
Lua_State. then you can just kill the whole thing without any lua_*
function. either arena (allocate from a given address range) or pool
(register all allocations in a per-state linked list), might work.
hope this helps.
[1]: somewhere... dispersed between the Lua_State structure, the call
stack, the trace's snapshot chain, and any temporary register the code
might be using at the moment.
--
Javier
