Hello, I've recently completed a paper on internals of Lua 5.1 VM and API.
I chose Lua due to small but still real world relevant codebase and
language. The choice of 5.1 version was influenced by my indifference to
features in 5.2 and 5.3 and compatibility of LuaJIT with 5.1. LuaJIT is now
my next object of interest (not to write another paper but to learn about
VMs and optimizations better). I was very interested in how a full Lua
value could fit into 64 bits and now I have few questions about the design
of that particular piece of code.
I do understand the concept of NaN tagging fully. However:
1. I understand that LuaJIT works (in particular, in case of light user
data pointers) with the assumption that 16 top bits of a 64 bit pointer are
free for use due to this being the case in current implementations,
however, what if an implementation with more bits than that appears? I know
this is highly theoretical but still an interesting point. A more concrete
example is a situation in which the users themselves use the top 16 bit in
such way (as some sort of a tag or simply because they use the 64 bit
pointer as a 64 bit int) and then pass the pointer to LuaJIT and receive
something else back. A 'fix' could be making light userdata pointers that
don't come with 0s in top 16 bits become another kind of object, subject to
GC but behaving the same in the API, just like true and false seem to be
'types' internally for implementation reasons despite being two values of
one type 'bool' on the outside.
2. Why can't the above trick be used with GC objects. I know LuaJIT uses
only bottom 32 bit of the address space which leads to the infamous problem
with using strings to hold binary data. I remember coming across mentions
that GC objects "must be" in the bottom 4 GB of virtual address space. Why
is that?
3. In case the pointers to GC objects have to just be 32 bit values, could
a trick like the one in Java be used? HotSpot and some other JVMs are able
to make pointer aligned to 2, 4 or 8 bytes and then shift (and offset if
they don't start at zero) them left and right appropriately before use and
storage (because 1 to 3 bottom bits are always 0 due to alignment), this
gains them 1, 2 or 3 bits while keeping the pointer value 32 bit at the
cost of a single shift (and maybe addition) which Oracle claims is
negligible compared to the benefits. It's described better in here or by
simply googling 'Java compressed object pointers':
http://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html#compressedOop
