[py-lmdb] Re: py-lmdb write performance

  • From: David Wilson <dw@xxxxxxxx>
  • To: py-lmdb@xxxxxxxxxxxxx
  • Date: Tue, 27 May 2014 13:18:33 +0000

Hi Dinesh,

Your "divide and conquer" approach sounds interesting. In fact, assuming
the merge step is literally just combining the partitions into one
master database without any extra processing, LMDB includes a special
'append' mode that would speed this operation up.

A nice side effect of this approach is that the final database becomes
optimally packed in the merge step, since it is written sequentially.

Perhaps something like:

    def sorted_union(i1, i2):
        i1 = iter(i1)
        i2 = iter(i2)
        e1 = next(i1, None)
        e2 = next(i2, None)
        while e1 and e2:
            if e1 <= e2:
                yield e1
                e1 = next(i1, None)
                yield e2
                e2 = next(i2, None)

        for elem, it in (e1, i1), (e2, i2):
            if e:
                yield e
            for elem in it:
                yield elem

    def iterate_remote_db(num):
        """Do whatever necessary to call Cursor.iternext() on the remote
        database, returning an iterable of (key, value) pairs"""

    # Build a recursive union of all the cursor iterators
    merged = iter_local_db()
    for num in range(NUM_REMOTE_DBS):
        merged = sorted_union(merged, iterate_remote_db(num))

    # Write sequentially to the final DB
    with master_env.begin(write=True) as txn:
        curs = txn.cursor()
        curs.putmulti(merged, append=True)


On Tue, May 27, 2014 at 05:57:53AM -0700, Dinesh Vadhia wrote:
> The problem to solve is to create a very large db (> 1tb) of synthetic data
> using a cluster of machines.  Once created, the db will be accessed by one
> machine only for predominantly read-only use.  The filesystem is network
> attached.
> One method to create the db is for each machine to create a dictionary of
> data and save it on the filesystem - this is pretty fast.  Next, get one
> machine (only) to write each dictionary data to the db.  One machine writing
> to lmdb on filesystem across a network should be okay but slow - yes?

Other related posts: