[py-lmdb] Re: Multiple environments to avoid write lock?
- From: David Wilson <dw@xxxxxxxx>
- To: py-lmdb@xxxxxxxxxxxxx
- Date: Tue, 3 Feb 2015 20:03:47 +0000
Hi Jonatan!
There's not quite enough info to work with here, and probably it's
better if you just profile how your app behaves first. For example,
* if you're writing all those keys from a sequential range of keys, then
that will perform much better than random completely random keys. If
your write txns are fast enough, then probably it's not worth the
effort of splitting your data into separate environments. Often
sequential write is faster by a factor of 2 or more.
* Are your keys/values huge or tiny? To get an intuitive feel for
performance, it's sometimes helpful to think of txn size in terms of
total MB read/written.. this is especially true when the entire DB is
cached or on very fast media. Writing 100,000 8 byte key + 8 byte
values (1.52MiB) is going to be much faster than say, 512 byte values
(49.59MiB).
* if you're overwriting existing keys during the write txn, then that
changes things a lot too, since DB growth will be bounded.
* Similarly, the presence/absense of MDB_SYNC (if suitable for your app)
drastically changes write performance characteristics.
* How many concurrent writers do you need? Is it a web app? Another
option is to push your large writer off to a task queue.
* Perhaps tweak examples/dirtybench.py from the Git repository to create
keys/values roughly of your size. This will produce some nice example
numbers for you to work from for your hardware/disk combination.
Drop me a line if you need some more ideas, but probably it's best for
you to just write some throwaway code to profile your specific use case.
David
On Tue, Feb 03, 2015 at 08:50:43AM -0800, Jonatan Heyman wrote:
> Hi!
>
> I'm new to LMDB and python-lmdb, but so far I find it really promising and
> well
> documented.
>
> I'm writing an app where I plan to use LMDB to store records in three
> different
> named databases. One action in my app might result in hundreds or thousands of
> read/writes to these databases. If I've understood LMDB correctly, there is a
> global write lock for each Environment. Would it then make sense for me to
> store my different databases in three different Environments, in order to
> decrease the time spent waiting for write locks?
>
> Best,
> Jonatan
Other related posts:
