Re: Does anyone happen to have a good pure Lua/FFI qsort for int tuples?
- From: demetri <demetri.spanos@xxxxxxxxx>
- To: luajit@xxxxxxxxxxxxx
- Date: Tue, 21 Jul 2015 02:39:20 -0700
Thankfully I have resolved the problem, and I apologize to everyone
for having been stupid: there was a rare branch in the code that
resulted in an additional ffi.cast of a slice of an array that, over
large numbers of invocations, would generate a ton of garbage and
then tax the garbage collector.
It seems I spoke too soon. I got rid of the problem on medium size
(say 50MB) arrays but when I went back to testing on the really big
ones I still got a significant slowdown. This time memory usage is
tightly bounded by ~5MB even after I stopped the garbage collector.
Even with this fix, something that takes 30-60 seconds with a C
function takes 10+ minutes with my implementation.
If I reduce to the 1-tuple case I am able to use the code Shmuel
provided without modification but I get pretty rapid fallback to the
interpreter. So, unfortunately that doesn't provide an obvious path
forward either.
In any case, I sheepishly re-open my declaration of interest in a
pure Lua/FFI sort suitable for moderately large struct of int32_t
fields (8 fields per struct) in large (1GB+) arrays. If it is anywhere
in the ballpark of qsort with a C function for a 1GB array of
int32_t's pulled from dev/random, I will be completely satisfied
(any extra customization from that point on should be trivial, I'm
just trying to find a basic proof of concept from which to expand
and for whatever reason my copied implementation of the Apple /
BSD qsort is not anywhere close).
If nothing turns up I'll just live with compiling in a bunch of C
functions for now. Hardly a disaster, just an ongoing nuisance in
the build process every time the rest of the system evolves :)
Thanks again everyone,
Demetri
Other related posts:
