Using FFI for large string buffers and a suggestion
- From: QuentinC <webmaster@xxxxxxxxxxxx>
- To: luajit@xxxxxxxxxxxxx
- Date: Wed, 27 Jun 2012 08:28:26 +0200
Hello,
Because I'm in the context of a text editor, many useful lua scripts
operate on strings, sometimes possibly large.
I have tried 3 techniques to concat many pieces of strings together, and
the big winner is FFI buffers (script at the end of this message).
Using FFI buffer appear to be 300 times faster than the naive approach,
and 4-5 times faster than building a table and then using table.concat.
FFI is the winner, is that intended or is it just accidental ?
Am I really naive and is there even better solutions ?
Why not adding a function ffi.renew to enlarge an already existing
VLA/VLS using realloc internally ?
What about returning the number of bytes copied by ffi.copy when no
explicit length is given ?
Thank you for your answers.
Here is the compare script :
local ffi = require 'ffi'
local n = 50000
-- Version #1: naive approach
local t = os.clock()
local s = ''
for i=1,n do
s = s .. i
end
t = os.clock() -t
print(t, #s)
-- Version #2: FFI buffers
local l, p, c = 0, 0, 100
t = os.clock()
s = ffi.new('char[?]', c)
for i=1,n do
l = tostring(i)
if p+#l+1 > c then -- need buffer space
c = math.ceil(c * 3/2 +1)
local ns = ffi.new('char[?]', c)
ffi.copy(ns, s, p)
s = ns
end
ffi.copy(s+p, l)
p = p + #l
end
s[p]=0 -- Final NULL char
s = ffi.string(s)
t = os.clock() -t
print(t, #s)
-- Version #3: table.concat
t = os.clock()
local tab = {}
for i=1,n do
table.insert(tab,i)
end
s = table.concat(tab)
t = os.clock() -t
print(t, #s)
Other related posts:
