Hi Ludwig,
The tickets you linked relate to py-lmdb's potential to cause page
faults while the GIL is held, however as seen in #65, I committed
a simple change to minimize the possibility of this ever happening, and
under normal circumstances where the machine is not experiencing severe
memory pressure, e.g. newly touched pages are immediately flushed before
creating the Python string object, there should be no problem using a
thread to prefetch data within a multiprocessing app.
Let me know how you get on. :)
David
On Thu, Jun 16, 2016 at 02:17:53PM +0200, Ludwig Schmidt-Hackenberg wrote:
Hello,
I wanted to ask about using Python multiprocessing / threading to prefetch
data
using py-lmdb.
At the moment I have a loop that reads a chunk from a lmdb file using py-lmdb
and then performs a computation on that data. To speed up I would like to
'hide' the IO by reading the next chunk of the lmdb while the computation is
happening.
In the documentation I found the statement `Environments may be opened by
multiple processes on the same host, making it ideal for working around
Python’s GIL.` But I found no example code and reading (and only partly
understanding) https://github.com/dw/py-lmdb/issues/86 and https://github.com/
dw/py-lmdb/issues/65 I got the impression that I what I want is not 'easily'
possible.
Appreciate any pointers
Thanks
Ludwig
--
EyeEm
Ludwig Schmidt-Hackenberg
Computer Vision Engineer
@shackenberg
EyeEm Mobile GmbH
Kohlfurter Strasse 4I/43, I0999 Berlin, Germany
Geschäftsführer: Florian Meissner & Lorenz Aschoff
Handelsregister Amtsgericht Charlottenburg HRBI3253I
