Limiting Mongo’s Memory Usage

The database package used in AC-Hunter (up to version 6.4.0) is Mongodb. Mongo assumes it has access to as much memory as the system has – and occasionally even more than that. 🙂 If you’re finding that mongodb is occasionally crashing because it’s out of memory, please read on to see how to put a limit on Mongo’s memory usage.

Is Mongo Eating Up All Your Memory?

Run:

sudo dmesg -T | egrep -i '(Out of memory|oom)'

and look for mongod being killed in the results.

What Memory Limit to Use?

First, we need to confirm how much memory is available on your system. Please run:

free -h

on your AC-Hunter system. Here’s a sample output, showing just the “Total” column for memory and swap:

     total
Mem: 32Gi
Swap: 40Gi

This system has 32GB + 40GB = 72GB total memory space. We can’t use all of that since some is used by the kernel, libraries, other programs, and disk caching, so the amount we pick needs to be less than 72GB. Leaving 8GB for other uses means our limit for Mongo has to be 64GB or less.

Note that a lot of this memory is swap, so as Mongo goes beyond 32GB it will start to run more slowly. It should be fine to pick a number a little larger than 32GB (such as 36GB, 40GB, or 44GB, but you should be cautious about going too much beyond those.

In short, pick a number that’s up to 12GB more than your installed memory, but also less than (memory + swap – 8GB).

Putting This Limit in Place

Next, log in to the AC-Hunter system as a user that has sudo privileges and run:

sudo nano /opt/AC-Hunter/docker/db.mongo.yml

In this file, look down for a line that reads:

    command: "mongod --auth"

While this article may not show it, that line should be indented by exactly 4 spaces; as you’re editing it, please don’t change the spacing or save the file with any form of tabs.
Please add the following wiredTigerCacheSize parameter to that line inside the pair of double quotes. The number you place at the end is the maximum number of gigabytes that Mongo should be allowed to use:

    command: "mongod --auth --wiredTigerCacheSizeGB 44"

Again, make sure not to change the number of spaces to the left of “command:…” or convert it to tabs. Both of the command line options start with two simple dashes each (to the right of “0” on an English language keyboard.) The odd capitalization in “–wiredTigerCacheSizeGB” is actually correct.

Once you’ve made the change, save and exit with ctrl-o and ctrl-x, respectively, and then run:

hunt down
hunt up -d --force-recreate

Closing Thoughts

To watch Mongo’s memory usage, run “top”. Press “M” to sort the processes by the amount of memory they use. If you plan to leave top running for a few days to watch it, press “s120<Enter>” (without the quotes) to tell top to only refresh its output once every 120 seconds.

There’s one side note to this. The limit you set above handles most of Mongo’s memory usage but not the memory it temporarily uses while creating indexes. This is why Mongo may occasionally go above the limit you set.

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=8234