Freeing Drive Space on Zeek and AC-Hunter
Intro
Zeek, RITA and AC-Hunter all do a good job of summarizing your network traffic; these summaries take up far less space than the original network packets. That said, they will still fill up a network drive if left unchecked.
There are 4 directories you need to monitor:
(1) On the Zeek system: /opt/zeek/logs/ . This holds the logs collected by this copy of Zeek.
(2) On the AC-Hunter system: /opt/zeek/logs/ , but only if you are running Zeek, Espy, or ActiveFlow on the AC-Hunter system itself. This directory will be empty if you are not running one of these on the AC-Hunter system.
(3) On the AC-Hunter system: /opt/zeek/remotelogs/ . This holds a second copy of the logs for each reporting Zeek sensor. The directories immediately under “remotelogs” will be the names of each sensor (like “Zeek1__10117”), and under each of those directories you’ll find the same dated directories (like “2022-08-13”) that you’d find on the Zeek system. Note that since AC-Hunter only needs some of the Zeek logs (those starting with conn, dns, http, ssl, x509, and known_certs), only these logs are transferred over to AC-Hunter.
(4) On the AC-Hunter system: /var/lib/docker/volumes/ . We use a docker volume, a virtual drive, to hold the AC-Hunter databases. Each database usually corresponds to a 24 hour block of logs. The “-rolling” databases hold the most recent 24 complete hours of logs. The databases ending in a yyyy-mm-dd date stamp hold the logs for that calendar day.
Space Used
To see how much disk space is being used, run the following command on your Zeek sensor(s):
sudo du -sh /opt/zeek/logs/
which will report on (1), and the following command on your AC-Hunter system(s):
sudo du -sh /opt/zeek/logs/ /opt/zeek/remotelogs /var/lib/docker/volumes/
which will report on (2), (3), and (4).
All 4 numbers will be printed in “human readable” format, so numbers in “megabytes” will end in “M”, numbers in gigabytes will end in “G”, and numbers in Terabytes will end in “T”. Here’s an example from a very lightly loaded Zeek server:
sudo du -sh /opt/zeek/logs/ 320M /opt/zeek/logs/
This shows that we’re using 320 megabytes of storage for our logs on this Zeek sensor.
Here’s an example from an AC-Hunter server:
sudo du -sh /opt/zeek/logs/ /opt/zeek/remotelogs /var/lib/docker/volumes/ 4.0K /opt/zeek/logs/ 89G /opt/zeek/remotelogs 29G /var/lib/docker/volumes/
Numbers like “4.0K” (4 kilobytes) are essentially 0, so you can read the first line of output as “No locally generated logs”.
The second and third lines of output show that we’re using 89 gigabytes of space for the logs sent from remote Zeek sensors, and 29 gigabytes for the AC-Hunter databases.
Space Available
To see how much space is available, run this on your Zeek sensor(s):
df -h /opt/zeek/logs/ Filesystem Size Used Avail Use% Mounted on /dev/sda2 26G 610M 25G 3% /opt/bro
This tells us we’re using 610 megabytes of space out of 26 total gigabytes on that disk (3% of the total space). We have another 25 gigabytes available.
On your AC-Hunter server(s), run:
sudo df -h /opt/zeek/logs/ /opt/zeek/remotelogs /var/lib/docker/volumes/ Filesystem Size Used Avail Use% Mounted on /dev/vda1 310G 156G 155G 51% / /dev/vda1 310G 156G 155G 51% /opt/bro/remotelogs /dev/vda1 310G 156G 155G 51% /
All three show the same amount of space, in this case because they’re all stored on the same 310 gigabyte virtual drive. The total amount of space used is 156 gigabytes (51% of the total) and we have another 155 gigabytes available for all uses (logs, databases, and everything else we store on that system.)
How Much Should I Keep?
The first answer comes from your retention policy, if your organization has one. This should tell you how many days of data you’re expected to keep.
Databases
For active threat hunting for the last 24 hours, you’ll need the “rolling” database(s) for each sensor. The daily snapshots are handy if you need to do historical research, such as “I just found this infected system – when was it first infected?” If you don’t expect to need to look back further than, for example, 90 days, you don’t need to keep more than that many days of databases.
Zeek Logs
Because of the way log files are sent from Zeek sensors to AC-Hunter, you should always keep at least 4 days of Zeek logs on the AC-Hunter system.
Your Zeek logs (1 to 3) get automatically imported into databases (4). Once they’ve been imported, you no longer need them for generating AC-Hunter databases. While it’s not part of normal threat hunting, people occasionally refer back to the original Zeek logs for more details on particular systems or conversations.
Beyond the 4 day minimum you can remove the logs on the Zeek sensor(s), the AC-Hunter system(s), or both. You can either delete the logs or migrate them over to secondary storage to satisfy your data retention requirements.
Pruning Manually
(1) and (2) https://portal.activecountermeasures.com/support/faq/?Display_FAQ=870
(3) https://portal.activecountermeasures.com/support/faq/?Display_FAQ=860
(4) You can delete all databases older than a number of days by logging in to AC-Hunter, going to the Dashboard tab, clicking on the gear (settings) icon, and scrolling to the bottom of the Database screen. There are two buttons; choose “Database Removal _by age_” (not delete all, please!). There you can remove all databases older than a certain number of days. You also have the option of deleting individual databases by clicking on the “x” to the right of the database name.
Pruning Automatically
(1) This is a copy of the “Log Maintenance” section of the AC-Hunter User Guide:
Zeek logs that accumulate on the Zeek system can be configured to expire and be automatically deleted after a certain amount of time. The setting can be found in the zeekctl configuration file (/opt/zeek/etc/zeekctl.cfg – see the note below).
# Expiration interval for archived log files in LogDir. # Files older than this will be deleted by "broctl cron". # The interval is an integer followed by one of these time units: # day, hr, min. A value of 0 means that logs never expire. LogExpireInterval = 0
For instance, if you wanted logs to automatically be deleted after 30 days you would modify the setting to be:
LogExpireInterval = 30 day
This will automatically remove log files located in the /opt/zeek/logs/ directory on your Zeek system.
Note: if you do not have a file called /opt/zeek/etc/zeekctl.cfg , please see the instructions at https://portal.activecountermeasures.com/support/faq/?Display_FAQ=3132 first.
(2) If you are running Zeek on the AC-Hunter system, follow the same steps in (1), above. If you are running Espy or ActiveFlow on the AC-Hunter system, we do not have an automatic pruning tool for their logs at this time.
(3) https://portal.activecountermeasures.com/support/faq/?Display_FAQ=860
(4) This is a copy of the “Deleting RITA Logs/Databases” section of the User Guide:
Once the Zeek logs are copied to the RITA/AC-Hunter system, they are processed by RITA and then passed to AC-Hunter. In the “Managing Databases” section, we identified how to delete databases from within AC-Hunter. This method is if you cannot access the AC-Hunter interface or prefer to use a script.
First, connect to the RITA/AC-Hunter system with ssh. Run the command:
df -h
to see how much space is available. If this amount is low, run
rita list
The command will then display all of the databases that are currently stored in RITA. Once you’ve decided which ones to delete, you can specify either a specific database to delete on the command line or a pattern that matches a number of them. For example, to delete all databases from July 2018, run:
rita delete -m RITA-2018-07
You can specify more than one pattern like this:
rita delete -r '(^RITA-2017|^RITA-2018-0[123])'
which would delete all databases from 2017 and January through March, 2018.
To see if you have enough free space, run the following command again:
df -h
You can repeat the above steps as required.
As an alternative, AC-Hunter 6.2.0 will provide a new script, /usr/local/bin/rita_delete_old_dbs.sh . This allows you to manually or automatically delete databases that are N days old or older. Here are the instructions for use at the top of the script:
This script deletes RITA databases older than a given number of days.
The following example performs a dry run, displaying RITA databases 300 days old or older:
rita_delete_old_dbs.sh 300
To actually delete them, add “-a” to the command line after the number of days:
rita_delete_old_dbs.sh 300 -a
After testing, you can set this up to run automatically by creating the file /etc/cron.d/cron-delete-sensors
with the following line (with the initial “#” and spaces removed):
# 50 2 * * * root /usr/local/bin/rita_delete_old_dbs.sh 300 -a
You’ll want to adjust the “300” to the maximum number of days to keep.
Once this is in place, run the following 2 commands (again, without “#” or spaced) to load the new cron job:
sudo service cron reload 2>/dev/null sudo service crond reload 2>/dev/null
Automatically Removing Logs and Databases With One Command
One of our developers has written a script that will remove both old log file directories and databases with a single script. Please see https://github.com/activecm/zeek-log-clean for the tool and instructions on how to use it.
To cover all 4 storage locations you’ll need to install it on all Zeek sensors (to handle category (1)), all AC-Hunter systems (to handle categories (3) and (4)), and you’ll need to run it twice on all AC-Hunter systems (once for /opt/zeek/logs/ and once for /opt/zeek/remotelogs/ to also handle category (2).
We hope this is useful to you!
Adding Drive Space
There’s one more possibility. If you chose to use “LVM” (Logical Volume Management) when you first installed the Linux operating system on either the Zeek sensor(s) or the AC-Hunter system(s), you may be able to add drive space.
To check if you’re using LVM, run:
sudo vgdisplay
If this comes back empty, you are not using LVM and do not have this option. If this comes back with the names and details of one or more Volume Groups, you may be able to add another drive to the system and add the space to the filesystem that’s full. Contact [email protected] for more details.
Direct Link to this FAQ Item: https://portal.activecountermeasures.com/support/faq/?Display_FAQ=4970