The database package used in AC-Hunter (up to version 6.4.0) is Mongodb. Mongo assumes it has access to as much memory as the system has – and occasionally even more than that. 🙂 If you’re finding that mongodb is occasionally crashing because it’s out of memory, please read on to see how to put a limit on Mongo’s memory usage.

 

Is Mongo Eating Up All Your Memory?

Run:

sudo dmesg -T | egrep -i '(Out of memory|oom)'

and look for mongod being killed in the results.

 

What Memory Limit to Use?

First, we need to confirm how much memory is available on your system. Please run:

free -h

on your AC-Hunter system. Here’s a sample output, showing just the “Total” column for memory and swap:

     total
Mem: 32Gi
Swap: 40Gi

This system has 32GB + 40GB = 72GB total memory space. We can’t use all of that since some is used by the kernel, libraries, other programs, and disk caching, so the amount we pick needs to be less than 72GB. Leaving 8GB for other uses means our limit for Mongo has to be 64GB or less.

Note that a lot of this memory is swap, so as Mongo goes beyond 32GB it will start to run more slowly. It should be fine to pick a number a little larger than 32GB (such as 36GB, 40GB, or 44GB, but you should be cautious about going too much beyond those.

In short, pick a number that’s up to 12GB more than your installed memory, but also less than (memory + swap – 8GB).

 

Putting This Limit in Place

Next, log in to the AC-Hunter system as a user that has sudo privileges and run:

sudo nano /opt/AC-Hunter/docker/db.mongo.yml

In this file, look down for a line that reads:

    command: "mongod --auth"

While this article may not show it, that line should be indented by exactly 4 spaces; as you’re editing it, please don’t change the spacing or save the file with any form of tabs.
Please add the following wiredTigerCacheSize parameter to that line inside the pair of double quotes. The number you place at the end is the maximum number of gigabytes that Mongo should be allowed to use:

    command: "mongod --auth --wiredTigerCacheSizeGB 44"

Again, make sure not to change the number of spaces to the left of “command:…” or convert it to tabs. Both of the command line options start with two simple dashes each (to the right of “0” on an English language keyboard.) The odd capitalization in “–wiredTigerCacheSizeGB” is actually correct.

Once you’ve made the change, save and exit with ctrl-o and ctrl-x, respectively, and then run:

hunt down
hunt up -d --force-recreate

 

Closing Thoughts

To watch Mongo’s memory usage, run “top”. Press “M” to sort the processes by the amount of memory they use. If you plan to leave top running for a few days to watch it, press “s120<Enter>” (without the quotes) to tell top to only refresh its output once every 120 seconds.

There’s one side note to this. The limit you set above handles most of Mongo’s memory usage but not the memory it temporarily uses while creating indexes. This is why Mongo may occasionally go above the limit you set.

 

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=8234

AC-Hunter normally imports the most recent 24 hours of logs for you, discarding old logs when new ones show up. There may be times when you want to import logs by hand, such as when:

– you’re bringing over logs from a sensor that doesn’t normally submit them.
– you’ve upgraded AC-Hunter and have a note that a database needs to be re-imported.
– you’d like to look at a time range smaller than a day.

 

Before you create a database, make sure the name you want to use doesn’t exist by checking the output of:

rita list

If that database name does exist, you can delete it with:

rita delete database_name

Finally, create a database from these logs by running the following command:

rita import /directory/that/holds/the/logs/ database_name

This command may take a while – the import process uses a lot of processor time, memory, and disk.

 

Note that this is the same command for both RITA and AC-Hunter. Once that has finished running, there will be a new database that you can use in future RITA commands or find in AC-Hunter’s database list (go to Dashboard, Gear icon, Database tab).

If you’re looking to replace a rolling database (one whose name ends in “-rolling”), please contact [email protected].

 

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=7732

When you run out of space, it’s time to add a new drive to your AC-Hunter system. Included in the below PDF document are the steps to make that happen. It will involve a brief outage, so plan for a time when AC-Hunter isn’t being actively used.

Moving_AC-Hunter_Storage_to_a_Larger_Drive.pdf

 

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=7640

Intro

Zeek, RITA and AC-Hunter all do a good job of summarizing your network traffic; these summaries take up far less space than the original network packets. That said, they will still fill up a network drive if left unchecked.

There are 4 directories you need to monitor:

(1) On the Zeek system: /opt/zeek/logs/ . This holds the logs collected by this copy of Zeek.

(2) On the AC-Hunter system: /opt/zeek/logs/ , but only if you are running Zeek, Espy, or ActiveFlow on the AC-Hunter system itself. This directory will be empty if you are not running one of these on the AC-Hunter system.

(3) On the AC-Hunter system: /opt/zeek/remotelogs/ . This holds a second copy of the logs for each reporting Zeek sensor. The directories immediately under “remotelogs” will be the names of each sensor (like “Zeek1__10117”), and under each of those directories you’ll find the same dated directories (like “2022-08-13”) that you’d find on the Zeek system. Note that since AC-Hunter only needs some of the Zeek logs (those starting with conn, dns, http, ssl, x509, and known_certs), only these logs are transferred over to AC-Hunter.

(4) On the AC-Hunter system: /var/lib/docker/volumes/ . We use a docker volume, a virtual drive, to hold the AC-Hunter databases. Each database usually corresponds to a 24 hour block of logs. The “-rolling” databases hold the most recent 24 complete hours of logs. The databases ending in a yyyy-mm-dd date stamp hold the logs for that calendar day.

 

Space Used

To see how much disk space is being used, run the following command on your Zeek sensor(s):

sudo du -sh /opt/zeek/logs/

which will report on (1), and the following command on your AC-Hunter system(s):

sudo du -sh /opt/zeek/logs/ /opt/zeek/remotelogs /var/lib/docker/volumes/

which will report on (2), (3), and (4).

All 4 numbers will be printed in “human readable” format, so numbers in “megabytes” will end in “M”, numbers in gigabytes will end in “G”, and numbers in Terabytes will end in “T”. Here’s an example from a very lightly loaded Zeek server:

sudo du -sh /opt/zeek/logs/
320M /opt/zeek/logs/

This shows that we’re using 320 megabytes of storage for our logs on this Zeek sensor.

Here’s an example from an AC-Hunter server:

sudo du -sh /opt/zeek/logs/ /opt/zeek/remotelogs /var/lib/docker/volumes/
4.0K /opt/zeek/logs/
89G /opt/zeek/remotelogs
29G /var/lib/docker/volumes/

Numbers like “4.0K” (4 kilobytes) are essentially 0, so you can read the first line of output as “No locally generated logs”.

The second and third lines of output show that we’re using 89 gigabytes of space for the logs sent from remote Zeek sensors, and 29 gigabytes for the AC-Hunter databases.

 

Space Available

To see how much space is available, run this on your Zeek sensor(s):

df -h /opt/zeek/logs/
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 26G 610M 25G 3% /opt/bro

This tells us we’re using 610 megabytes of space out of 26 total gigabytes on that disk (3% of the total space). We have another 25 gigabytes available.

On your AC-Hunter server(s), run:

sudo df -h /opt/zeek/logs/ /opt/zeek/remotelogs /var/lib/docker/volumes/
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 310G 156G 155G 51% /
/dev/vda1 310G 156G 155G 51% /opt/bro/remotelogs
/dev/vda1 310G 156G 155G 51% /

All three show the same amount of space, in this case because they’re all stored on the same 310 gigabyte virtual drive. The total amount of space used is 156 gigabytes (51% of the total) and we have another 155 gigabytes available for all uses (logs, databases, and everything else we store on that system.)

 

How Much Should I Keep?

The first answer comes from your retention policy, if your organization has one. This should tell you how many days of data you’re expected to keep.

Databases

For active threat hunting for the last 24 hours, you’ll need the “rolling” database(s) for each sensor. The daily snapshots are handy if you need to do historical research, such as “I just found this infected system – when was it first infected?” If you don’t expect to need to look back further than, for example, 90 days, you don’t need to keep more than that many days of databases.

Zeek Logs

Because of the way log files are sent from Zeek sensors to AC-Hunter, you should always keep at least 4 days of Zeek logs on the AC-Hunter system.

Your Zeek logs (1 to 3) get automatically imported into databases (4). Once they’ve been imported, you no longer need them for generating AC-Hunter databases. While it’s not part of normal threat hunting, people occasionally refer back to the original Zeek logs for more details on particular systems or conversations.

Beyond the 4 day minimum you can remove the logs on the Zeek sensor(s), the AC-Hunter system(s), or both. You can either delete the logs or migrate them over to secondary storage to satisfy your data retention requirements.

Pruning Manually

(1) and (2) https://portal.activecountermeasures.com/support/faq/?Display_FAQ=870

(3) https://portal.activecountermeasures.com/support/faq/?Display_FAQ=860

(4) You can delete all databases older than a number of days by logging in to AC-Hunter, going to the Dashboard tab, clicking on the gear (settings) icon, and scrolling to the bottom of the Database screen. There are two buttons; choose “Database Removal _by age_” (not delete all, please!). There you can remove all databases older than a certain number of days. You also have the option of deleting individual databases by clicking on the “x” to the right of the database name.

Pruning Automatically

(1) This is a copy of the “Log Maintenance” section of the AC-Hunter User Guide:

Zeek logs that accumulate on the Zeek system can be configured to expire and be automatically deleted after a certain amount of time. The setting can be found in the zeekctl configuration file (/opt/zeek/etc/zeekctl.cfg – see the note below).

# Expiration interval for archived log files in LogDir. 
# Files older than this will be deleted by "broctl cron". 
# The interval is an integer followed by one of these time units: 
# day, hr, min. A value of 0 means that logs never expire.
LogExpireInterval = 0

For instance, if you wanted logs to automatically be deleted after 30 days you would modify the setting to be:

LogExpireInterval = 30 day

This will automatically remove log files located in the /opt/zeek/logs/ directory on your Zeek system.

Note: if you do not have a file called /opt/zeek/etc/zeekctl.cfg , please see the instructions at https://portal.activecountermeasures.com/support/faq/?Display_FAQ=3132 first.

(2) If you are running Zeek on the AC-Hunter system, follow the same steps in (1), above. If you are running Espy or ActiveFlow on the AC-Hunter system, we do not have an automatic pruning tool for their logs at this time.

(3) https://portal.activecountermeasures.com/support/faq/?Display_FAQ=860

(4) This is a copy of the “Deleting RITA Logs/Databases” section of the User Guide:

Once the Zeek logs are copied to the RITA/AC-Hunter system, they are processed by RITA and then passed to AC-Hunter. In the “Managing Databases” section, we identified how to delete databases from within AC-Hunter. This method is if you cannot access the AC-Hunter interface or prefer to use a script.

First, connect to the RITA/AC-Hunter system with ssh. Run the command:

df -h

to see how much space is available. If this amount is low, run

rita list

The command will then display all of the databases that are currently stored in RITA. Once you’ve decided which ones to delete, you can specify either a specific database to delete on the command line or a pattern that matches a number of them. For example, to delete all databases from July 2018, run:

rita delete -m RITA-2018-07

You can specify more than one pattern like this:

rita delete -r '(^RITA-2017|^RITA-2018-0[123])'

which would delete all databases from 2017 and January through March, 2018.

To see if you have enough free space, run the following command again:

df -h

You can repeat the above steps as required.

As an alternative, AC-Hunter 6.2.0 will provide a new script, /usr/local/bin/rita_delete_old_dbs.sh . This allows you to manually or automatically delete databases that are N days old or older. Here are the instructions for use at the top of the script:

This script deletes RITA databases older than a given number of days.

The following example performs a dry run, displaying RITA databases 300 days old or older:

rita_delete_old_dbs.sh 300

To actually delete them, add “-a” to the command line after the number of days:

rita_delete_old_dbs.sh 300 -a

After testing, you can set this up to run automatically by creating the file /etc/cron.d/cron-delete-sensors
with the following line (with the initial “#” and spaces removed):

# 50 2 * * * root /usr/local/bin/rita_delete_old_dbs.sh 300 -a

You’ll want to adjust the “300” to the maximum number of days to keep.

Once this is in place, run the following 2 commands (again, without “#” or spaced) to load the new cron job:

sudo service cron reload 2>/dev/null
sudo service crond reload 2>/dev/null

 

Automatically Removing Logs and Databases With One Command

One of our developers has written a script that will remove both old log file directories and databases with a single script. Please see ​​https://github.com/activecm/zeek-log-clean for the tool and instructions on how to use it.

To cover all 4 storage locations you’ll need to install it on all Zeek sensors (to handle category (1)), all AC-Hunter systems (to handle categories (3) and (4)), and you’ll need to run it twice on all AC-Hunter systems (once for /opt/zeek/logs/ and once for /opt/zeek/remotelogs/ to also handle category (2).

We hope this is useful to you!

 

Adding Drive Space

There’s one more possibility. If you chose to use “LVM” (Logical Volume Management) when you first installed the Linux operating system on either the Zeek sensor(s) or the AC-Hunter system(s), you may be able to add drive space.

To check if you’re using LVM, run:

sudo vgdisplay

If this comes back empty, you are not using LVM and do not have this option. If this comes back with the names and details of one or more Volume Groups, you may be able to add another drive to the system and add the space to the filesystem that’s full. Contact [email protected] for more details.

 

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=4970

If you’d like to have your sensors connect to a new instance of AC-Hunter (in addition to continuing to send to the old server or as a replacement for the old server) here are the steps.

On each Zeek server, perform the following steps:

(1) Edit /etc/cron.d/zeek_log_transport

• • If you’re sending logs to a second server: copy the existing line and edit the new line. In the new line, replace the old hostname/IP with the new hostname/IP.
  • If you’re sending logs to a new AC-Hunter server and no longer want to send the log to the old server: edit the existing line and replace the hostname/IP with the new hostname/IP.
  • Remember the user under which the file transfer is being run (field after the last * in that file, like “5 * * * * jparker ……”).

 

(2) Under that user, ssh to the new AC-Hunter server, like:

jparker$ ssh dataimport@new_achunter_hostname

and accept the ssh key.

If you get an error message indicating the “host key has changed”, jot down the known_hosts file offending line number mentioned in the error message. Your new server has either the same IP address or the same hostname as the old server, so you’ll need to tell ssh to use the new host key. The error message (“…the host key has changed…”) will mention a line number in ~/.ssh/known_hosts. Edit that file with your preferred editor and scroll down to the mentioned line. Delete the line and save the file.

Now retry the ssh command; the “host key has changed” error message should disappear and you should be asked to accept the new ssh host key. Please do so.

 

If your new server has both a different hostname and a different IP address from the old server, continue on.

 

(3) As that same user, send your ssh host keys to the new server with the following commands. Both commands (one starting with “cat” and the other starting with “ssh”) should each be on a single command line, even though they may wrap in this blog. You’ll be asked for the dataimport user’s password on that remote system by the first command, but not the second. Please replace “newserver” with the name/IP of the new server.

cat ~/.ssh/id_rsa_dataimport 2>/dev/null | ssh "dataimport@newserver" 'mkdir -p .ssh ; cat >>.ssh/authorized_keys ; chmod go-rwx ./ .ssh/ .ssh/authorized_keys'

ssh -i "$HOME/.ssh/id_rsa_dataimport" -o 'PasswordAuthentication=no' -o 'PreferredAuthentications=publickey' "dataimport@newserver" 'true' && echo "Key appears to be successfully installed"

If you get back “Key appears to be successfully installed” , you’re done.

 

(4) If you have more than 1 Zeek sensor that you wish to feed to this new server, please log in to the next sensor and restart at step 1.

 

If the steps above refuse to work, you can always rerun the script that connects a Zeek sensor to AC-Hunter. Follow the commands under “Automated instructions” at https://portal.activecountermeasures.com/support/faq/?Display_FAQ=863 .

If you continue to find problems, please feel free to contact us at [email protected] .

 

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=4947

Why?

While swap is critically important when you’re low on memory, it’s not immediately obvious that Linux systems really want at least a modest amount of swap space even when you have lots and lots of ram. This allows the kernel to push out unused data, freeing up memory to do a better job caching disk access.

 

How Much?

Generally 1GB or more. If you’re extremely low on disk space, try to come up with at least a 512MB or 256MB swap.

Going up to 4GB is a good idea on a larger server. Going a good deal beyond that probably won’t add much benefit (except in special cases where you know you’ll have lots of idle processes).

 

Checking Existing Memory Settings

To find out how much memory you have installed, run this:

sudo lshw -short -C memory

The line with “System memory” is the one you care about. Here are some example output lines from systems with 8GB and 4GB of memory, respectively:

/0/5 memory 7811MiB System memory
/0/1000 memory 4GiB System Memory

BTW, lshw is great at reporting on all your main system specs; try this at some point:

sudo lshw -short | less

To see how much swap space you have, run:

cat /proc/swaps

Sample output:

cat /proc/swaps

Filename Type Size Used Priority

/var/swap/swap1 file 1048572 6952 -1

The Size column is in kilobytes, so that’s a 1GB swap space.

Alternatively, run “top” and look at the 5th line of output; the number immediately following “Swap:” is the swap space size. If the line starts with “KiB Swap:”, this number is in kilobytes, or if it starts with “MiB Swap:”, the number is in megabytes.

If there are no lines other than the banner line, you have no swap space and should do the following.

 

Adding Swap

If you’ve broken up your disk into small partitions and “/var/” is low on space, you’re welcome to put the swap directory somewhere else, adjusting the following commands to match.

sudo mkdir -p /var/swap

Make sure there isn’t already a “swap1” file; if there is, use a different name:

ls -al

The “dd” command will make a 2048 * 1MB = 2GB swap space. Adjust 2048 up or down as needed.

sudo dd if=/dev/zero of=/var/swap/swap1 bs=1048576 count=2048

Now format that file as swap space and make it “live”, available for immediate use:

sudo chown root.root /var/swap/swap1
sudo chmod 600 /var/swap/swap1
sudo mkswap /var/swap/swap1
sudo swapon /var/swap/swap1

Check that it’s activated. You should see a new line in the following output:

cat /proc/swaps

You should also see this swap space available when running top; look at the value following “Swap:” on the 5th line.

Once this is done, bring up /etc/fstab in your favorite editor under sudo as well to make the change permanent. Please type carefully – recovering from a typo in fstab can be very complicated.

sudo nano /etc/fstab

The line to add is:

​​/var/swap/swap1 swap swap defaults 0 0

The spacing between elements isn’t critical, but please double check that you have all components, there are no spaces in the file name, and there’s a linefeed at the end of the file. This will bring up your swap automatically on the next boot. Save and exit. (You do not need to reboot to activate the swap; the “swapon” command above did that, and the fstab line is only used for the next boot.)

 

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=4470

If the underlying Linux system runs out of space, that can lead to processes dying unexpectedly or tasks not completing. To check if you’re running low, run:

df -h

This shows the amount of free space on each of your partitions. Example – your output will be different:

df -h

Filesystem Size Used Avail Use% Mounted on

/dev/vda1 60G 9.9G 51G 17% /

/dev/vda2 1.0T 101G 899G 10% /var

devtmpfs 1.9G 0 1.9G 0% /dev

tmpfs 1.9G 180K 1.9G 1% /dev/shm

tmpfs 1.9G 193M 1.7G 11% /run

tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup

tmpfs 380M 0 380M 0% /run/user/1001

tmpfs 380M 0 380M 0% /run/user/0

The important lines are the ones associated with the root filesystem(“/”), the home partition (“/home/”, if there is one), and “/var/”, if there is one. If any of these are very low on space, you’ll need to free some up.

A common tool for this is delete-databases.sh , available on the Zeek and AC-Hunter/Rita systems. When run with no command line parameters, it lists the available databases.

Once you’ve found some you no longer need to keep, run the tool again with the name(s) of one or more of them to remove. We recommend running the same command on both systems to remove them from both.

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=870

---

If you have trouble running delete-databases.sh , such as:

./delete-databases.sh
exception: login failed
No pattern specified on the command line (such as ./delete-databases.sh '2018-05-31', so we will just list databases available to delete. Press enter when ready to see list.


2019-04-25T11:46:18.589-0400 Error: 18 { ok: 0.0, errmsg: "auth failed", code: 18, codeName: "AuthenticationFailed" } at src/mongo/shell/db.js:1292
Exiting.

This may be a result of a mismatch between the mongo command line client and the mongo server software. To check, run:

mongo --version

and see if you have version 3.6. If not, follow the instructions to upgrade the client at https://docs.mongodb.com/v3.6/administration/install-on-linux/ . It amounts to adding a third party package repo and then installing the “mongodb-org-shell” package.

You can tell if you were successful by running:

mongo --version

and checking that the MongoDB shell version is v3.6.x.

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=862

---