[py-lmdb] Re: py-lmdb write performance

  • From: "Dinesh Vadhia" <dineshvadhia@xxxxxxxxxxx>
  • To: <py-lmdb@xxxxxxxxxxxxx>
  • Date: Wed, 28 May 2014 05:48:47 -0700

Generated sorted keys to create local dictionaries on each cluster machine; next, one machine merges each dictionary in sorted order into db; but it still takes ~2.5 hours to write ~1gb of data with append=True;

doesn't sound right does it or does it?


--------------------------------------------------
From: "David Wilson" <dw@xxxxxxxx>
Sent: Tuesday, May 27, 2014 8:38 AM
To: <py-lmdb@xxxxxxxxxxxxx>
Subject: [py-lmdb] Re: py-lmdb write performance

On Tue, May 27, 2014 at 08:33:24AM -0700, Dinesh Vadhia wrote:

for key, value in d.items():
   with env.begin(db, write=True) as txn:
       txn.put(key=key, value=value, append=True)

What am I missing?

From the docs:

   Appending a key that is not greater than the highest existing key
   will cause corruption.

You can only use append=True with sorted input.


David



Other related posts: