Huge performance loss when processing lots of data

  • From: Chris <chris@xxxxxxxxxx>
  • To: luajit@xxxxxxxxxxxxx
  • Date: Tue, 03 Jul 2012 14:24:54 -0400

I'm having a problem on a long running process that is churning through lots of data. At a certain constant point in processing it slows down massively (by at least an order of magnitude). It goes from taking 1 second to process a line item to 10+ seconds each, even for the exact same data that it just processed in 1 second.


I have sample code that shows the problem here:
http://luadev.com/uploads/slowdata01.lua

On my machine it runs quick for a while (~0.1 secs per) then at exactly k=102 iterations it starts taking 50+ times longer for each loop (~5+ secs per).

Weird thing is, it's always at 102 iterations, no matter the size of the data set. Not sure if that is machine dependent or possibly data dependent. I'm testing on a 64-bit Linux kernel 3.4.4, Q9550, 8 GB RAM using 32-bit LuaJIT HEAD from git.

Note that if the collectgarbage() call is removed it blows up (assert) because the VM is unaware of the amount of memory being allocated and can't clean up automatically (any way to fix that by the way?). That's a secondary issue though, I'm more concerned with the huge performance drop in the middle of processing.

Excuse any weirdness in the sample code, I had to reduce the problem to this simplified sample which is made from code ripped from a much larger base.

--
Chris   http://luadev.com

Other related posts: