Re: Slow loadstring problem
- From: Sean Conner <sean@xxxxxxxxxx>
- To: luajit@xxxxxxxxxxxxx
- Date: Fri, 11 Jan 2013 01:10:00 -0500
It was thus said that the Great Ben Rog-Wilhelm once stated:
>
> I’ve got a 50mb lua file that takes about two seconds to load in Lua
> 5.2. Unfortunately, it takes about two *minutes* to load in Luajit 2.0.
> I haven’t been able to figure out what’s going on in detail. Lua 5.1.4
> errors out with “constant table overflow”, so maybe this isn’t
> intended to be supported on LuaJIT, but we’re actually using “can the
> lua environment load it” to tell us if the output file is parseable. As
> it is, a two-minute delay has turned into a rather large problem for us.
I had this same problem back in 2009 [1][2], only involving Lua itself,
not LuaJIT, and was due to a problem with garbage collection (not that this
is the issue here). During that time, I also encountered the "constant
table overflow" issue, both with Lua and LuaJIT.
Now, in my case, it was because I was trying to construct a large table:
intertwingle =
{
filelist =
{
[1] =
{
file = "/home/spc/LINUS/Archive.mail/20060607/received",
time = "Fri, 25 Mar 2005 21:00:24 GMT",
size = 4020741.000000,
},
},
mbox =
{
[1] =
{
received =
{
[1] = "(from myg@localhost) by linus.arg.armigeron.com (8.8.7/8.8.7)
id XAA01908 for spc; Mon, 17 Aug 1998 23:23:26 -0400",
},
info =
{
mboxfile = 1.000000,
ob =
{
[1] = 995.000000,
[2] = 212.000000,
},
oh =
{
[1] = 555.000000,
[2] = 439.000000,
},
},
resent =
{
},
-- ...
and so on, for about 131M worth of table. Both Lua and LuaJIT choked. I had
to construct the file as:
intertwingle = {
filelist = {
{ file = "/home/spc/LINUS/Archive.mail/20060607/received", size = 4020741,
time = "Fri, 25 Mar 2005 21:00:24 GMT"},
}
}
intertwingle.mbox = {}
local function init()
table.insert(intertwingle.mbox,{
info = { mboxfile = 1, oh={555, 439}, ob={995, 212}},
resent = {
},
received = {"(from myg@localhost) by linus.arg.armigeron.com (8.8.7/8.8.7)
id XAA01908 for spc; Mon, 17 Aug 1998 23:23:26 -0400",},
headers = {
['Return-Path'] = "<myg>",
['Message-ID'] = "<199808180323.XAA01908@xxxxxxxxxxxxxxxxxxxxxxx>",
['In-Reply-To'] = { },
-- ...
})
init()
local function init()
table.insert(...
That is, I had to break it up to limit the number of constants per
scope (it's 4,096 emails per init() function, but each email has a large
number of fields initialized, so I had to kind of find a figure that worked
for both Lua and LuaJIT).
Once I fixed the generation problem, and the GC bug was fixed in Lua
5.1.4, both Lua (5.1.5 now) and LuaJIT (2.0.0-beta10, dated Oct. 2012) load
it (on a 32bit 2.6GHz Pentium D) in about 11 seconds.
-spc
[1] http://lua-users.org/lists/lua-l/2009-11/msg00463.html
[2] http://www.lua.org/bugs.html#5.1.4-6
Other related posts:
