[py-lmdb] Re: Multiple environments to avoid write lock?

  • From: David Wilson <dw@xxxxxxxx>
  • To: py-lmdb@xxxxxxxxxxxxx
  • Date: Tue, 3 Feb 2015 20:03:47 +0000

Hi Jonatan!

There's not quite enough info to work with here, and probably it's
better if you just profile how your app behaves first. For example,

* if you're writing all those keys from a sequential range of keys, then
  that will perform much better than random completely random keys. If
  your write txns are fast enough, then probably it's not worth the
  effort of splitting your data into separate environments. Often
  sequential write is faster by a factor of 2 or more.

* Are your keys/values huge or tiny? To get an intuitive feel for
  performance, it's sometimes helpful to think of txn size in terms of
  total MB read/written.. this is especially true when the entire DB is
  cached or on very fast media. Writing 100,000 8 byte key + 8 byte
  values (1.52MiB) is going to be much faster than say, 512 byte values

* if you're overwriting existing keys during the write txn, then that
  changes things a lot too, since DB growth will be bounded.

* Similarly, the presence/absense of MDB_SYNC (if suitable for your app)
  drastically changes write performance characteristics.

* How many concurrent writers do you need? Is it a web app? Another
  option is to push your large writer off to a task queue.

* Perhaps tweak examples/dirtybench.py from the Git repository to create
  keys/values roughly of your size. This will produce some nice example
  numbers for you to work from for your hardware/disk combination.

Drop me a line if you need some more ideas, but probably it's best for
you to just write some throwaway code to profile your specific use case.


On Tue, Feb 03, 2015 at 08:50:43AM -0800, Jonatan Heyman wrote:
> Hi!
> I'm new to LMDB and python-lmdb, but so far I find it really promising and 
> well
> documented.
> I'm writing an app where I plan to use LMDB to store records in three 
> different
> named databases. One action in my app might result in hundreds or thousands of
> read/writes to these databases. If I've understood LMDB correctly, there is a
> global write lock for each Environment. Would it then make sense for me to
> store my different databases in three different Environments, in order to
> decrease the time spent waiting for write locks?
> Best,
> Jonatan

Other related posts: