Easy Steps to Limit Mongodb Memory Usage

When Using MongoDB in local computer or laptop for development we usually see mongoDB eating up all RAM and makes the machine very slow to operate, so in this article we will discuss on how to restrict MongoDB memory usage in a non standard way. Usually it is suggested not to restrict mongoDB memory as MongoDB defers to the operating system when loading data into memory from disk. It simply memory maps all its data files and relies on the operating system to cache data. The OS typically evicts the least-recently-used data from RAM when it runs low on memory. So in a production environment it is not recommended to restrict MongoDB memory usage as this might impact the performance and functionalities.

But when you are using the mongoDB as a test DB in your development environment it is frustrating to work while mongoDB running in backend as process as all the RAM gets used up and system becomes unusable till you stop/kill mongoDB. So to avoid this kind of situations we can follow below steps to Limit Mongodb Memory Usage and develop in peace even after MongoDB running in backend.

MongoDB running in Linux Servers:

Limit Mongodb Memory Usage using cgroups:

1. Create control group:

cgcreate -g memory:MongoLimitGroup

(make sure that cgroups binaries installed on your system)

2. Specify how much memory needs to be allocated for this group:

echo 1610612736 > /sys/fs/cgroup/memory/MongoLimitGroup/memory.limit_in_bytes

This command limits memory to 1.5G (This limits the memory for both malloc allocations and OS cache)

3. Drop pages already stayed in cache using below command:

sync; echo 3 > /proc/sys/vm/drop_caches

4. Now assign a server to be created control group:

cgclassify -g memory:MongoLimitGroup mongod_pid

This will assign a running mongod process to a group limited by only 1.5GB memory.

5. Now check your free memory using below command to make sure things are fine.

free -m

MongoDB running in Windows Servers:

Incase you are looking for Quick Steps To Limit Mongodb Memory Usage in windows servers too then you can follow below link for the same.

 

Bash Script Limit MongoDB Memory Usage :

We have created a bash script to automatically limit mongoDB memory usage. The script receives a single argument as how much MB of RAM needs to be allocated to mongoDB process.

./script.sh 1536

Note: 1536 is 1536MB. You can use any number of MB according to your requirement.

#!/bin/bash
# Script to limit mongoDB memory usage.
# Author: TechPaste.Com
############################################

export _arg="$1"
if [ -z "$_arg" ]
then
echo "Memory to Allocate is empty"
echo "Usage: ./script.sh 1536"
echo "Note: Here 1536 is 1536 MB of RAM"
exit 1;

fi

_check_tools() {
_cmd=$1
command -v $_cmd >/dev/null 2>&1 || { echo >&2 "Script requires $_cmd but it's not in path or not installed. Aborting."; exit 1; }

}
_pre_setup() {

_check_tools ps
_check_tools pmap
_check_tools cgcreate
_check_tools sync
_check_tools cgclassify
_check_tools tail
_check_tools free

memaloc=`echo "$(($_arg * 1024 * 1024))"`
mongopid=`ps -eaf | grep mongod |grep -v grep | awk -F" " '{print $2}'`
if [[ $mongopid == "" ]]; then

echo " Mongo DB is not running. Please start the Mongo DB service first.";
echo "Example start command: mongod --fork --dbpath /data/db/ --logpath /opt/mongodb/mongodb.log"
exit 1;

fi
echo
echo "Mongo DB Process: "
echo
echo "########################################################"
ps -eaf | grep mongod |grep -v grep
echo "########################################################"
echo "Current MongoDB RAM usage:"
echo "$(( `pmap -x $mongopid | tail -1 | awk -F" " '{print $3}'` ))KB= $(( `pmap -x $mongopid | tail -1 | awk -F" " '{print $3}'` / 1024 ))MB"
echo "########################################################"
echo
}
_mem_setup() {
echo
echo "1. Creating control group :MongoLimitGroup."
echo
echo "Running cgcreate -g memory:MongoLimitGroup"
cgcreate -g memory:MongoLimitGroup
echo
echo "2. Specifying $memaloc bytes memory needs to be allocated for this group"
echo
echo "echo $memaloc > /sys/fs/cgroup/memory/MongoLimitGroup/memory.limit_in_bytes"
echo $memaloc > /sys/fs/cgroup/memory/MongoLimitGroup/memory.limit_in_bytes
echo
echo "3. Dropping pages already stayed in cache..."
echo
echo "sync; echo 3 > /proc/sys/vm/drop_caches"
sync; echo 3 > /proc/sys/vm/drop_caches
echo
echo "4. Assigning a server to be created for control group"
echo
echo "cgclassify -g memory:MongoLimitGroup $mongopid"
cgclassify -g memory:MongoLimitGroup $mongopid
echo
echo "########################################################"
echo "Post Setup MongoDB RAM usage:"
echo "$(( `pmap -x $mongopid | tail -1 | awk -F" " '{print $3}'` ))KB= $(( `pmap -x $mongopid | tail -1 | awk -F" " '{print $3}'` / 1024 ))MB"
echo "########################################################"
}

Main() {
_pre_setup
_mem_setup
}
Main

UPDATE[09/29/2016] [OPTION 1] Setting Up MongoDB cache size in MB

For MongoDB 3.2 onwards there is cache_size option to define mongodb memory limit and reduce mongodb memory size and use mongoDB in low RAM development systems by following below hidden documentation of MongoDB.

You can set cache size in megabytes. It seems this is undocumented in the MongoDB documentation keep mongodb memory limit low.

For example, the following is valid startup parameter to set your cache size to 200MB :

--wiredTigerEngineConfigString="cache_size=200M"

Example:

# mongod –fork –storageEngine wiredTiger –wiredTigerEngineConfigString=”cache_size=200M” –dbpath /data/db/ –logpath /opt/mongodb.log

2016-09-30T00:33:30.273+0530 I STORAGE [main] Engine custom option: cache_size=200M
about to fork child process, waiting until server is ready for connections.
forked process: 7803
child process started successfully, parent exiting

Entries in mongodb.log

2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] MongoDB starting : pid=7803 port=27017 dbpath=/data/db/ 64-bit host=localhost
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] db version v3.2.4
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] git version: e2ee9ffcf9f5a94fad76802e28cc978718bb7a30
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] allocator: tcmalloc
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] modules: none
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] build environment:
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] distmod: rhel70
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] distarch: x86_64
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] target_arch: x86_64
2016-09-30T00:33:30.295+0530 I CONTROL [initandlisten] options: { processManagement: { fork: true }, storage: { dbPath: "/data/db/", engine: "wiredTiger", wiredTiger: { engineConfig: { cacheSizeGB: 1, configString: "cache_size=200M" } } }, systemLog: { destination: "file", path: "/opt/mongodb.log" } }
2016-09-30T00:33:30.354+0530 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),cache_size=200M

Output from MongoDB post retstart:

> db.serverStatus().wiredTiger.cache
{
"tracked dirty bytes in the cache" : 0,
"tracked bytes belonging to internal pages in the cache" : 360353,
"bytes currently in the cache" : 1741122,
"tracked bytes belonging to leaf pages in the cache" : 1380769,
"maximum bytes configured" : 209715200,
"tracked bytes belonging to overflow pages in the cache" : 0,
"bytes read into cache" : 1260264,
"bytes written from cache" : 148642,
"pages evicted by application threads" : 0,
"checkpoint blocked page eviction" : 0,
"unmodified pages evicted" : 0,
"page split during eviction deepened the tree" : 0,
"modified pages evicted" : 0,
"pages selected for eviction unable to be evicted" : 0,
"pages evicted because they exceeded the in-memory maximum" : 0,
"pages evicted because they had chains of deleted items" : 0,
"failed eviction of pages that exceeded the in-memory maximum" : 0,
"hazard pointer blocked page eviction" : 0,
"internal pages evicted" : 0,
"maximum page size at eviction" : 0,
"eviction server candidate queue empty when topping up" : 0,
"eviction server candidate queue not empty when topping up" : 0,
"eviction server evicting pages" : 0,
"eviction server populating queue, but not evicting pages" : 0,
"eviction server unable to reach eviction goal" : 0,
"internal pages split during eviction" : 0,
"leaf pages split during eviction" : 0,
"pages walked for eviction" : 0,
"eviction worker thread evicting pages" : 0,
"in-memory page splits" : 0,
"in-memory page passed criteria to be split" : 0,
"lookaside table insert calls" : 0,
"lookaside table remove calls" : 0,
"percentage overhead" : 8,
"tracked dirty pages in the cache" : 0,
"pages currently held in the cache" : 134,
"pages read into cache" : 133,
"pages read into cache requiring lookaside entries" : 0,
"pages written from cache" : 14,
"page written requiring lookaside records" : 0,
"pages written requiring in-memory restoration" : 0
}

Below are few stats which I have taken post setting the cache_size though not sure its completely reducing the mongodb memory usage or not.

Before Restart of MongoDB: ( Without setting cache_size)

           Kbytes RSS Dirty
total kB 1527432 114668 107220

Post fresh Restart of MongoDB:( Without setting cache_size)

           Kbytes RSS Dirty
total kB 381908 19916 14232

Post Accessing MongoDB services for few processing:( Without setting cache_size)

           Kbytes RSS Dirty
total kB 414692 39596 33188

Post Setting the cache_size:

mongod --fork --storageEngine wiredTiger --wiredTigerEngineConfigString="cache_size=200M" --dbpath /data/db/ --logpath /opt/mongodb.log

Post fresh Restart:

           Kbytes RSS Dirty
total kB 381900 16512 11088

Post Accessing MongoDB services for few processing:

           Kbytes RSS Dirty
total kB 410580 30580 24568

Its been 24+ hours and the MongoDB memory usage is bit low compared to old times of 1.5GB as shown below. Note: This time in using cache_size we have not used the MongoRAM.sh file to restrict the memory at OS level.

                  Kbytes  RSS      Dirty
total kB          639652  130380  125072

Will update the thread after some more days to see if there is any improvement in mongodb memory limit levels as this is helpful for developers to run MongoDB in there DEV boxes without a major performance hit.

UPDATE[10/03/2016] [OPTION 2] Using db.repairDatabase() to mongodb shrink database size

You can use below command to start the repair the database which will ultimately shrink the size of the database.

MongoDB compresses the files by:

• copying the files to a new location
• looping through the documents and re-ordering / re-solving them
• replacing the original files with the new files

You can compress by running mongod --repair or by connecting directly and running db.repairDatabase() like below.

bash$ printf "use MYDB\ndb.serverStatus().mem\ndb.repairDatabase()\ndb.serverStatus().mem" | mongo

Note: The above command will take longer time to complete depending on the size of your database.

Sample Output:

# printf "use MYDB\ndb.serverStatus().mem\ndb.repairDatabase()\ndb.serverStatus().mem" | mongo
MongoDB shell version: 3.2.4
connecting to: test
switched to db MYDB
{
        "bits" : 64,
        "resident" : 266,
        "virtual" : 10149,
        "supported" : true,
        "mapped" : 47402,
        "mappedWithJournal" : 65730
}
{ "ok" : 1 }
{
        "bits" : 64,
        "resident" : 155,
        "virtual" : 8149,
        "supported" : true,
        "mapped" : 47402,
        "mappedWithJournal" : 65740
}
bye

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.