Rootkits behavior detection

ThreatLockDown uses the Rootcheck module to detect anomalies that might indicate the presence of malware in an endpoint. By default, Rootcheck scans run every 12 hours. However, you can configure a custom frequency based on your needs. Check the Rootcheck frequency option on the documentation to learn how to do this. For more information on Rootcheck, see the Rootcheck section of the documentation.

How it works

The image below describes the steps ThreatLockDown performs to detect malicious behavior on a monitored endpoint.

  1. After you install a ThreatLockDown agent on an endpoint, the agent uses the Rootcheck module to monitor specified directories, registry entries, and system calls for abnormal behavior. The module scans the endpoint using out-of-the-box rootkit signatures. You can also define custom rootkit signatures.

  2. The ThreatLockDown manager analyzes the logs sent by the agent. The analysis includes pre-decoding, decoding, and matching the logs to predefined rules.

  3. When a log matches a rule, the ThreatLockDown manager creates alerts in the /var/ossec/logs/alerts/alerts.log and /var/ossec/logs/alerts/alerts.json files.

You can enable the <logall> option in the ThreatLockDown server configuration file, /var/ossec/etc/ossec.conf. When this option is enabled, the ThreatLockDown manager saves all the logs from monitored endpoints, whether they trigger alerts or not, to the /var/ossec/logs/archives/archives.log and /var/ossec/logs/archives/archives.json files.

We describe Rootcheck capabilities in the sections below.

Check running processes

A malicious process can prevent itself from being seen in a system list of running processes, for example, by replacing the ps command with a trojan version. Rootcheck inspects all process IDs (PID), looking for discrepancies using different system calls such as getsid and getpgid.

For example, diamorphine is a kernel-mode rootkit that can hide itself and other processes from the ps command. Suppose you install this rootkit and hide a process. In that case, you get an alert like this:

** Alert 1668497757.1838662: - ossec,rootcheck,
2022 Nov 15 09:35:57 (Ubuntu) any->rootcheck
Rule: 521 (level 11) -> 'Possible kernel level rootkit'
Process '659' hidden from /proc. Possible kernel level rootkit.
title: Process '659' hidden from /proc.

Check hidden ports

Malware might use hidden ports to communicate with attackers. Rootcheck scans every port in the system using the C language function bind(). If it can’t bind to a port and that port isn’t in the netstat output, it means malware might be present.

Check unusual files and permissions

ThreatLockDown scans the entire file system looking for unusual files and permissions. An example is a file owned by the root user, which has write permissions for other user accounts. Rootcheck inspects suid files, hidden directories, and other files to check for unusual permissions.

Check hidden files using system calls

ThreatLockDown scans the entire system, comparing the differences between the stat size and the file size when using the fopen + read calls. ThreatLockDown also compares the number of nodes in each directory with the output of opendir + readdir. If any results don't match, malware might be present.

Below, we show an example of an alert generated by Rootcheck when it detects an anomaly in the filesystem:

** Alert 1668497750.1838326: - ossec,rootcheck,pci_dss_10.6.1,gdpr_IV_35.7.d,
2022 Nov 15 09:35:50 (Ubuntu) any->rootcheck
Rule: 510 (level 7) -> 'Host-based anomaly detection event (rootcheck).'
Rootkit 't0rn' detected by the presence of file '/usr/bin/.t0rn'.
title: Rootkit 't0rn' detected by the presence of file '/usr/bin/.t0rn'.

By default, the Rootcheck module monitors the following directories:

Unix

/bin, /sbin, /usr/bin, /usr/sbin, /dev, /lib, /etc, /root, /var/log, /var/mail, /var/lib, /var/www, /usr/lib, /usr/include, /tmp, /boot, /usr/local, /var/tmp and /sys

Windows

C:\WINDOWS and C:\Program Files

Scan the /dev directory

The /dev directory should only contain device-specific files. ThreatLockDown inspects all files in this directory because malware can use this partition to hide files.

For example, if you create a hidden file in the /dev directory, it triggers an alert in ThreatLockDown because there is a hidden file in a directory that should only contain device-specific files. The following is an example of such an alert:

** Alert 1668498534.1862633: - ossec,rootcheck,pci_dss_10.6.1,gdpr_IV_35.7.d,
2022 Nov 15 09:48:54 (Ubuntu) any->rootcheck
Rule: 510 (level 7) -> 'Host-based anomaly detection event (rootcheck).'
File '/dev/.hidden' present on /dev. Possible hidden file.
title: File present on /dev.
file: /dev/.hidden

Scan network interfaces

ThreatLockDown scans for any network interface in a system with promiscuous mode enabled. It checks the output of the ifconfig command. If an interface is in promiscuous mode, it triggers an alert. A network interface in promiscuous mode might be an indicator that malware is present.

Rootkit checks

Rootcheck performs several checks using its database of rootkit signatures: rootkit_files.txt, rootkit_trojans.txt, and win_malware_rcl.txt. You can update these signatures by adding your signatures to perform rootkit checks using the latest information available. We describe how you can do this in the next section.

How to configure options for Rootcheck

Basic example

To configure the options for Rootcheck, edit the /var/ossec/etc/ossec.conf file on the ThreatLockDown server. For more information about the configuration options, read the Rootcheck section of the documentation. Also, check the following sections: frequency, rootkit_files, and rootkit_trojans.

Here is a basic example of configuring the databases Rootcheck uses to detect rootkits, specifically rootkit files, and trojans. The rootkit_trojans.txt and rootkit_files.txt databases must contain trojan and rootkit signatures:

<rootcheck>
  <rootkit_files>etc/shared/rootkit_files.txt</rootkit_files>
  <rootkit_trojans>etc/shared/rootkit_trojans.txt</rootkit_trojans>
</rootcheck>

Use the rootkit_files.txt to specify known filepaths to files used by rootkits. Rootcheck scans these filepaths and generates alerts whenever it detects any of these files. You can add multiple entries to the file and each entry must be in the following format:

# This is a comment
file_path ! Rootkit_Name ::

The example below detects Reptile rootkit:

#Reptile
reptile/reptile_cmd         ! Reptile rootkit ::
reptile/reptile_rc          ! Reptile rootkit ::
reptile/reptile_shell       ! Reptile rootkit ::
reptile/reptile_start       ! Reptile rootkit ::
lib/udev/reptile            ! Reptile rootkit ::

The rootkit_trojans.txt contains signatures of files that rootkits have trojaned. Rootcheck scans the files in this database for specified strings and generates alerts when there is a match. The signatures must be in the following format:

# This is a comment
file_name !string_to_search!Description

The example below detects trojan versions of common Linux binaries:

# Common binaries and public trojan entries
ls          !bash|^/bin/sh|dev/[^clu]|\.tmp/lsfile|duarawkz|/prof|/security|file\.h!
env         !bash|^/bin/sh|file\.h|proc\.h|/dev/|^/bin/.*sh!
echo        !bash|^/bin/sh|file\.h|proc\.h|/dev/[^cl]|^/bin/.*sh!
chown       !bash|^/bin/sh|file\.h|proc\.h|/dev/[^cl]|^/bin/.*sh!
chmod       !bash|^/bin/sh|file\.h|proc\.h|/dev/[^cl]|^/bin/.*sh!
chgrp       !bash|^/bin/sh|file\.h|proc\.h|/dev/[^cl]|^/bin/.*sh!
cat         !bash|^/bin/sh|file\.h|proc\.h|/dev/[^cl]|^/bin/.*sh!

Ignoring false positives

In case of false positives, create a rule with level 0 and use a regex pattern matching specifically the logs that are false positives. You can see an example below.

<rule id="100100" level="0">
  <if_group>rootcheck</if_group>
  <match>/dev/.blkid.tab</match>
  <description>Ignore false positive for /dev/.blkid.tab</description>
</rule>