Re: Data-dependent slowdown in loop involving io.lines()
- From: Coda Highland <chighland@xxxxxxxxx>
- To: luajit@xxxxxxxxxxxxx
- Date: Tue, 11 Nov 2014 16:55:11 -0800
Once again, you missed my point -- you didn't hit a bad hashing
SCHEME. You happened to hit a bad CASE in an otherwise-good scheme.
The "slightly nicer hashing scheme" that works better for you very
well might work more poorly for other use cases -- as I mentioned,
EVERY hashing scheme is going to have an issue one way or another, and
LuaJIT is unapologetically no-hand-holding.
If you want performance out of LuaJIT, you have to be willing to
actually optimize your code. LuaJIT isn't a magic bullet.
/s/ Adam
On Tue, Nov 11, 2014 at 4:49 PM, soumith <soumith@xxxxxxxxx> wrote:
> Rewriting the standard io.open/read via ffi just because we hit a bad
> hashing scheme is actually quite sad. The cases where the collisions occur
> are "very" real-world and not at all impractical cases (file paths, come
> on!).
>
> Either ways, we have a solution and we use it (a slightly nicer hashing
> scheme that doesn't break on such simple cases for a very very tiny increase
> in computation cycles), forking and monkey-patching it for ourselves is a
> solution for us, the people who don't know better would just give up.
>
> On Tue, Nov 11, 2014 at 5:41 PM, Coda Highland <chighland@xxxxxxxxx> wrote:
>>
>> On Tue, Nov 11, 2014 at 2:23 PM, soumith <soumith@xxxxxxxxx> wrote:
>> > Considering this as a non-issue is quite irresponsible especially when
>> > the
>> > case is quite general and would be encountered by quite a few people.
>>
>> This isn't really calling it a "non-issue." What's actually being said
>> is that it's essentially UNAVOIDABLE. There are tradeoffs to be made,
>> and in any string-hashing system there's ALWAYS going to be some
>> pathological worst-case behaviors that come up. And not hashing the
>> string would cause worse behaviors in other places. Somewhere has to
>> win, somewhere has to lose.
>>
>> If your behavior hits this path, there are ways to sidestep it. I gave you
>> one.
>>
>> /s/ Adam
>>
>
Other related posts:
