[py-lmdb] Re: Allocating Max RAM memory to LMDB.

  • From: David Wilson <dw@xxxxxxxx>
  • To: py-lmdb@xxxxxxxxxxxxx
  • Date: Wed, 25 Mar 2015 22:05:27 +0000

Hi Offer,

On Tue, Mar 24, 2015 at 11:47:57AM -0400, Offer Sharabi wrote:

> We are planning to use the LMDB in our production environment where we
> have limited RAM resource. In our current implementation we are using
> the python LMDB client, and instantiate the DB from the python
> application.

> When testing LMDB we found that it used up to 99% RAM capacity before
> it started hitting disk. Unfortunately, this has severely slowed down
> the servers and affected all other services including LMDB swap which
> jumped from several ms to several seconds or even minutes.

I chatted briefly with Julian on IRC about the problem and discovered
you're using Centos 6. Over the past day I've been attempting to
replicate your issue using a CentOS 6 VM, and so far have been unable to
find a case where the default CentOS configuration results in Linux
going into swap.

Can you tell me a little more about your setup, e.g.:

* Is your LMDB load mostly read-only, or are you creating many dirty
  pages by writing to the database?

* Is your LMDB load mostly doing random read/write IO? (I presume this
  is the case), or e.g. is it append-only or similar.

* Were the other applications on your system previously swapping at all?
  e.g. is there a sizable "Used" value in /proc/swaps on machines that
  haven't been touched by your LMDB program?

* Are the other applications long-lived processes, or e.g. cron jobs? I
  don't know all the details, but Linux has a variety of metrics that
  influence the page reclaim process.

Could you possibly include the output of "cat /proc/123123/smaps", where
123123 is the PID of the largest of the other servers on the machine?
You might need to edit the output of this command to remove confidential
filenames from the output.

> I was hoping that there is some parameter that can be given to LMDB in
> order to allocate max memory usage, and in this way better control
> resources. However, I failed to find one in docs and was wondering : 
> 
> Did I just missed it ? 

> If not, is there a way to control memory allocation ? 

You did not miss it, LMDB does not directly have tunables that manage
memory. One cornerstone of LMDB's design is that all memory management
is delegated to the host operating system (Linux).


> If not, is there a plan to introduce one ? 

> Finally, is there any other solution that you can think of for the problem?

There are a variety of subtle ways you can control Linux memory
management behavior depending on version, but on modern Linux you have
basically two of interest:

* Control groups ("cgroups") and the "memory" controller. This allows
  you to precisely control how much page cache the LMDB process is
  entitled to, however it requires a much newer kernel than available in
  CentOS 6. :(

* /proc/sys/vm/swappiness. This influences the kernel's decisionmaking
  under memory pressure. Default/higher values will cause the kernel to
  favour paging anonymous memory (i.e. process heaps) to the configured
  swap space, while lower values will cause it to prefer emptying the
  page cache instead (i.e. file-backed mappings, like an LMDB database).


As far as I am aware, given you're on CentOS 6, your only available
control is swappiness, which is unfortunate, since it is a system-global
option and may have unintended consequences for other software running
on the machine, that might require a little more testing than usual.

In older releases of Linux, low swappiness values could trigger the
Linux OOM killer, which is absolutely not what you want on a production
machine. ;)


> Thanks in advance for your help on this, 

Sadly I can't offer much yet -- I can't trigger the situation where your
process is swapping. Are you sure that Linux really is pushing process
heaps into swap space? Or perhaps you can reveal some more information
about the other workloads on the machine that could hint at why they are
slowing down.

Could you perhaps include 60 seconds worth of "iostat -x 1" output from
an effected machine? On CentOS, iostat is included in the 'sysstat'
package.


David

> 
> Best ,
> 
> Offer Sharabi. 
> 
> MAGNE+IC
> 
> Offer Sharabi | Software Developer
> 
> magnetic.com  | blog  |  twitter  |  facebook  |  linkedin  |  youtube
> 

Other related posts: