Re: Luajit string concatenation performance

  • From: Dimiter 'malkia' Stanev <malkia@xxxxxxxxx>
  • To: luajit@xxxxxxxxxxxxx
  • Date: Wed, 22 May 2013 13:43:29 -0700

On 5/22/2013 1:32 PM, Geoff Leyland wrote:
On 23/05/2013, at 7:43 AM, madigest i <madigest@xxxxxxxxxxxxx> wrote:

Hello.

I have had strange performance problems with many of my different Lua programs 
that handle big strings. I finally made a small test program to demonstrate the 
problem.

Here are the results in OSX:
LuaJIT 2.1.0-alpha: 46 seconds
LuaJIT 2.0.1: 39 seconds
Lua 5.1: 22 seconds
LuaJIT 2 doesn't compile string concatenation, and LuaJIT 2.1 doesn't compile 
the gmatch.

I realise this is a test case, not your real code, but why isn't your inner 
loop like:

local codeout = {}
local timeUsed = os.time()
for line in code:gmatch("[^\r\n]+") do
   codeout[#codeout+1] = line.."\n"
end
codeout = table.concat(codeout, "")
timeUsed = os.time() - timeUsed

That takes 0s on all versions.  (Use os.clock in preference to os.time, it has 
better resolution)



Yup. Increasing the linecount numbers, and I'm getting 0.3sec for lua, 0.2sec for luajit (-joff or not)

--  luajit_speed_test.lua
--local codeout = ""
local codeout = {}
local linecount = 20000
local strlen = 100
local line = string.rep("a", strlen).."\n"
local code = string.rep(line, 200000)
local i = 0
local timeUsed = os.clock()
for line in code:gmatch("[^\r\n]+") do
  i = i + 1
--for i=1,linecount do
  if i%10000 == 0 or i == 1 then
    print("  ... line: "..i.." / "..linecount)
  end
  --codeout = codeout..line.."\n"
  codeout[#codeout+1] = line
end
codeout = table.concat(codeout, "\n")
timeUsed = os.clock() - timeUsed
if jit then
  print(jit.version..": "..timeUsed.." seconds")
else
  print(_VERSION..": "..timeUsed.." seconds")
end
print(" -- luajit_speed_test.lua end -- ")
print()



Other related posts: