Intro

Zeek, RITA and AC-Hunter all do a good job of summarizing your network traffic; these summaries take up far less space than the original network packets. That said, they will still fill up a network drive if left unchecked.

There are 4 directories you need to monitor:

(1) On the Zeek system: /opt/zeek/logs/ . This holds the logs collected by this copy of Zeek.

(2) On the AC-Hunter system: /opt/zeek/logs/ , but only if you are running Zeek, Espy, or ActiveFlow on the AC-Hunter system itself. This directory will be empty if you are not running one of these on the AC-Hunter system.

(3) On the AC-Hunter system: /opt/zeek/remotelogs/ . This holds a second copy of the logs for each reporting Zeek sensor. The directories immediately under “remotelogs” will be the names of each sensor (like “Zeek1__10117”), and under each of those directories you’ll find the same dated directories (like “2022-08-13”) that you’d find on the Zeek system. Note that since AC-Hunter only needs some of the Zeek logs (those starting with conn, dns, http, ssl, x509, and known_certs), only these logs are transferred over to AC-Hunter.

(4) On the AC-Hunter system: /var/lib/docker/volumes/ . We use a docker volume, a virtual drive, to hold the AC-Hunter databases. Each database usually corresponds to a 24 hour block of logs. The “-rolling” databases hold the most recent 24 complete hours of logs. The databases ending in a yyyy-mm-dd date stamp hold the logs for that calendar day.

 

Space Used

To see how much disk space is being used, run the following command on your Zeek sensor(s):

sudo du -sh /opt/zeek/logs/

which will report on (1), and the following command on your AC-Hunter system(s):

sudo du -sh /opt/zeek/logs/ /opt/zeek/remotelogs /var/lib/docker/volumes/

which will report on (2), (3), and (4).

All 4 numbers will be printed in “human readable” format, so numbers in “megabytes” will end in “M”, numbers in gigabytes will end in “G”, and numbers in Terabytes will end in “T”. Here’s an example from a very lightly loaded Zeek server:

sudo du -sh /opt/zeek/logs/
320M /opt/zeek/logs/

This shows that we’re using 320 megabytes of storage for our logs on this Zeek sensor.

Here’s an example from an AC-Hunter server:

sudo du -sh /opt/zeek/logs/ /opt/zeek/remotelogs /var/lib/docker/volumes/
4.0K /opt/zeek/logs/
89G /opt/zeek/remotelogs
29G /var/lib/docker/volumes/

Numbers like “4.0K” (4 kilobytes) are essentially 0, so you can read the first line of output as “No locally generated logs”.

The second and third lines of output show that we’re using 89 gigabytes of space for the logs sent from remote Zeek sensors, and 29 gigabytes for the AC-Hunter databases.

 

Space Available

To see how much space is available, run this on your Zeek sensor(s):

df -h /opt/zeek/logs/
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 26G 610M 25G 3% /opt/bro

This tells us we’re using 610 megabytes of space out of 26 total gigabytes on that disk (3% of the total space). We have another 25 gigabytes available.

On your AC-Hunter server(s), run:

sudo df -h /opt/zeek/logs/ /opt/zeek/remotelogs /var/lib/docker/volumes/
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 310G 156G 155G 51% /
/dev/vda1 310G 156G 155G 51% /opt/bro/remotelogs
/dev/vda1 310G 156G 155G 51% /

All three show the same amount of space, in this case because they’re all stored on the same 310 gigabyte virtual drive. The total amount of space used is 156 gigabytes (51% of the total) and we have another 155 gigabytes available for all uses (logs, databases, and everything else we store on that system.)

 

How Much Should I Keep?

The first answer comes from your retention policy, if your organization has one. This should tell you how many days of data you’re expected to keep.

Databases

For active threat hunting for the last 24 hours, you’ll need the “rolling” database(s) for each sensor. The daily snapshots are handy if you need to do historical research, such as “I just found this infected system – when was it first infected?” If you don’t expect to need to look back further than, for example, 90 days, you don’t need to keep more than that many days of databases.

Zeek Logs

Because of the way log files are sent from Zeek sensors to AC-Hunter, you should always keep at least 4 days of Zeek logs on the AC-Hunter system.

Your Zeek logs (1 to 3) get automatically imported into databases (4). Once they’ve been imported, you no longer need them for generating AC-Hunter databases. While it’s not part of normal threat hunting, people occasionally refer back to the original Zeek logs for more details on particular systems or conversations.

Beyond the 4 day minimum you can remove the logs on the Zeek sensor(s), the AC-Hunter system(s), or both. You can either delete the logs or migrate them over to secondary storage to satisfy your data retention requirements.

Pruning Manually

(1) and (2) https://portal.activecountermeasures.com/support/faq/?Display_FAQ=870

(3) https://portal.activecountermeasures.com/support/faq/?Display_FAQ=860

(4) You can delete all databases older than a number of days by logging in to AC-Hunter, going to the Dashboard tab, clicking on the gear (settings) icon, and scrolling to the bottom of the Database screen. There are two buttons; choose “Database Removal _by age_” (not delete all, please!). There you can remove all databases older than a certain number of days. You also have the option of deleting individual databases by clicking on the “x” to the right of the database name.

Pruning Automatically

(1) This is a copy of the “Log Maintenance” section of the AC-Hunter User Guide:

Zeek logs that accumulate on the Zeek system can be configured to expire and be automatically deleted after a certain amount of time. The setting can be found in the zeekctl configuration file (/opt/zeek/etc/zeekctl.cfg – see the note below).

# Expiration interval for archived log files in LogDir. 
# Files older than this will be deleted by "broctl cron". 
# The interval is an integer followed by one of these time units: 
# day, hr, min. A value of 0 means that logs never expire.
LogExpireInterval = 0

For instance, if you wanted logs to automatically be deleted after 30 days you would modify the setting to be:

LogExpireInterval = 30 day

This will automatically remove log files located in the /opt/zeek/logs/ directory on your Zeek system.

Note: if you do not have a file called /opt/zeek/etc/zeekctl.cfg , please see the instructions at https://portal.activecountermeasures.com/support/faq/?Display_FAQ=3132 first.

(2) If you are running Zeek on the AC-Hunter system, follow the same steps in (1), above. If you are running Espy or ActiveFlow on the AC-Hunter system, we do not have an automatic pruning tool for their logs at this time.

(3) https://portal.activecountermeasures.com/support/faq/?Display_FAQ=860

(4) This is a copy of the “Deleting RITA Logs/Databases” section of the User Guide:

Once the Zeek logs are copied to the RITA/AC-Hunter system, they are processed by RITA and then passed to AC-Hunter. In the “Managing Databases” section, we identified how to delete databases from within AC-Hunter. This method is if you cannot access the AC-Hunter interface or prefer to use a script.

First, connect to the RITA/AC-Hunter system with ssh. Run the command:

df -h

to see how much space is available. If this amount is low, run

rita list

The command will then display all of the databases that are currently stored in RITA. Once you’ve decided which ones to delete, you can specify either a specific database to delete on the command line or a pattern that matches a number of them. For example, to delete all databases from July 2018, run:

rita delete -m RITA-2018-07

You can specify more than one pattern like this:

rita delete -r '(^RITA-2017|^RITA-2018-0[123])'

which would delete all databases from 2017 and January through March, 2018.

To see if you have enough free space, run the following command again:

df -h

You can repeat the above steps as required.

As an alternative, AC-Hunter 6.2.0 will provide a new script, /usr/local/bin/rita_delete_old_dbs.sh . This allows you to manually or automatically delete databases that are N days old or older. Here are the instructions for use at the top of the script:

This script deletes RITA databases older than a given number of days.

The following example performs a dry run, displaying RITA databases 300 days old or older:

rita_delete_old_dbs.sh 300

To actually delete them, add “-a” to the command line after the number of days:

rita_delete_old_dbs.sh 300 -a

After testing, you can set this up to run automatically by creating the file /etc/cron.d/cron-delete-sensors
with the following line (with the initial “#” and spaces removed):

# 50 2 * * * root /usr/local/bin/rita_delete_old_dbs.sh 300 -a

You’ll want to adjust the “300” to the maximum number of days to keep.

Once this is in place, run the following 2 commands (again, without “#” or spaced) to load the new cron job:

sudo service cron reload 2>/dev/null
sudo service crond reload 2>/dev/null

 

Automatically Removing Logs and Databases With One Command

One of our developers has written a script that will remove both old log file directories and databases with a single script. Please see ​​https://github.com/activecm/zeek-log-clean for the tool and instructions on how to use it.

To cover all 4 storage locations you’ll need to install it on all Zeek sensors (to handle category (1)), all AC-Hunter systems (to handle categories (3) and (4)), and you’ll need to run it twice on all AC-Hunter systems (once for /opt/zeek/logs/ and once for /opt/zeek/remotelogs/ to also handle category (2).

We hope this is useful to you!

 

Adding Drive Space

There’s one more possibility. If you chose to use “LVM” (Logical Volume Management) when you first installed the Linux operating system on either the Zeek sensor(s) or the AC-Hunter system(s), you may be able to add drive space.

To check if you’re using LVM, run:

sudo vgdisplay

If this comes back empty, you are not using LVM and do not have this option. If this comes back with the names and details of one or more Volume Groups, you may be able to add another drive to the system and add the space to the filesystem that’s full. Contact [email protected] for more details.

 

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=4970

If you’d like to have your sensors connect to a new instance of AC-Hunter (in addition to continuing to send to the old server or as a replacement for the old server) here are the steps.

On each Zeek server, perform the following steps:

(1) Edit /etc/cron.d/zeek_log_transport

• • If you’re sending logs to a second server: copy the existing line and edit the new line. In the new line, replace the old hostname/IP with the new hostname/IP.
  • If you’re sending logs to a new AC-Hunter server and no longer want to send the log to the old server: edit the existing line and replace the hostname/IP with the new hostname/IP.
  • Remember the user under which the file transfer is being run (field after the last * in that file, like “5 * * * * jparker ……”).

 

(2) Under that user, ssh to the new AC-Hunter server, like:

jparker$ ssh dataimport@new_achunter_hostname

and accept the ssh key.

If you get an error message indicating the “host key has changed”, jot down the known_hosts file offending line number mentioned in the error message. Your new server has either the same IP address or the same hostname as the old server, so you’ll need to tell ssh to use the new host key. The error message (“…the host key has changed…”) will mention a line number in ~/.ssh/known_hosts. Edit that file with your preferred editor and scroll down to the mentioned line. Delete the line and save the file.

Now retry the ssh command; the “host key has changed” error message should disappear and you should be asked to accept the new ssh host key. Please do so.

 

If your new server has both a different hostname and a different IP address from the old server, continue on.

 

(3) As that same user, send your ssh host keys to the new server with the following commands. Both commands (one starting with “cat” and the other starting with “ssh”) should each be on a single command line, even though they may wrap in this blog. You’ll be asked for the dataimport user’s password on that remote system by the first command, but not the second. Please replace “newserver” with the name/IP of the new server.

cat ~/.ssh/id_rsa_dataimport 2>/dev/null | ssh "dataimport@newserver" 'mkdir -p .ssh ; cat >>.ssh/authorized_keys ; chmod go-rwx ./ .ssh/ .ssh/authorized_keys'

ssh -i "$HOME/.ssh/id_rsa_dataimport" -o 'PasswordAuthentication=no' -o 'PreferredAuthentications=publickey' "dataimport@newserver" 'true' && echo "Key appears to be successfully installed"

If you get back “Key appears to be successfully installed” , you’re done.

 

(4) If you have more than 1 Zeek sensor that you wish to feed to this new server, please log in to the next sensor and restart at step 1.

 

If the steps above refuse to work, you can always rerun the script that connects a Zeek sensor to AC-Hunter. Follow the commands under “Automated instructions” at https://portal.activecountermeasures.com/support/faq/?Display_FAQ=863 .

If you continue to find problems, please feel free to contact us at [email protected] .

 

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=4947

Why?

While swap is critically important when you’re low on memory, it’s not immediately obvious that Linux systems really want at least a modest amount of swap space even when you have lots and lots of ram. This allows the kernel to push out unused data, freeing up memory to do a better job caching disk access.

 

How Much?

Generally 1GB or more. If you’re extremely low on disk space, try to come up with at least a 512MB or 256MB swap.

Going up to 4GB is a good idea on a larger server. Going a good deal beyond that probably won’t add much benefit (except in special cases where you know you’ll have lots of idle processes).

 

Checking Existing Memory Settings

To find out how much memory you have installed, run this:

sudo lshw -short -C memory

The line with “System memory” is the one you care about. Here are some example output lines from systems with 8GB and 4GB of memory, respectively:

/0/5 memory 7811MiB System memory
/0/1000 memory 4GiB System Memory

BTW, lshw is great at reporting on all your main system specs; try this at some point:

sudo lshw -short | less

To see how much swap space you have, run:

cat /proc/swaps

Sample output:

cat /proc/swaps

Filename Type Size Used Priority

/var/swap/swap1 file 1048572 6952 -1

The Size column is in kilobytes, so that’s a 1GB swap space.

Alternatively, run “top” and look at the 5th line of output; the number immediately following “Swap:” is the swap space size. If the line starts with “KiB Swap:”, this number is in kilobytes, or if it starts with “MiB Swap:”, the number is in megabytes.

If there are no lines other than the banner line, you have no swap space and should do the following.

 

Adding Swap

If you’ve broken up your disk into small partitions and “/var/” is low on space, you’re welcome to put the swap directory somewhere else, adjusting the following commands to match.

sudo mkdir -p /var/swap

Make sure there isn’t already a “swap1” file; if there is, use a different name:

ls -al

The “dd” command will make a 2048 * 1MB = 2GB swap space. Adjust 2048 up or down as needed.

sudo dd if=/dev/zero of=/var/swap/swap1 bs=1048576 count=2048

Now format that file as swap space and make it “live”, available for immediate use:

sudo chown root.root /var/swap/swap1
sudo chmod 600 /var/swap/swap1
sudo mkswap /var/swap/swap1
sudo swapon /var/swap/swap1

Check that it’s activated. You should see a new line in the following output:

cat /proc/swaps

You should also see this swap space available when running top; look at the value following “Swap:” on the 5th line.

Once this is done, bring up /etc/fstab in your favorite editor under sudo as well to make the change permanent. Please type carefully – recovering from a typo in fstab can be very complicated.

sudo nano /etc/fstab

The line to add is:

​​/var/swap/swap1 swap swap defaults 0 0

The spacing between elements isn’t critical, but please double check that you have all components, there are no spaces in the file name, and there’s a linefeed at the end of the file. This will bring up your swap automatically on the next boot. Save and exit. (You do not need to reboot to activate the swap; the “swapon” command above did that, and the fstab line is only used for the next boot.)

 

Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=4470