File integrity monitoring and YARA

You can combine the capabilities of the ThreatLockDown FIM module with YARA to detect malware. YARA is an open source tool that identifies and classifies malware artifacts based on textual or binary patterns. These patterns are indicators found in malware samples and are defined in the YARA rules file. The YARA community updates the YARA rules file to include newly discovered malware signatures.

How it works

This technique detects malware in several stages and is achieved in the following way:

  1. The ThreatLockDown FIM module scans monitored directories on endpoints to detect changes such as file creation and modifications.

  2. When FIM detects a change in the monitored directory or file, it triggers a YARA scan active response. The active response module automatically executes YARA using the yara.sh script. YARA then scans the file that triggered the FIM alert against its ruleset to determine whether it is malware.

  3. When the YARA rules match a file, the scan results are forwarded to the ThreatLockDown manager for decoding, analysis and alerting. These scan results are not decoded out-of-the-box so you have to add the decoders to your ThreatLockDown server.

This implementation of YARA scans with the active response module concentrates the scans on new or recently modified files in monitored directories, thus optimizing resource consumption on the monitored endpoints.

The diagram below illustrates the flow of events between the different components.

You can configure YARA for malware detection on Linux and Windows endpoints. In the use cases below, we show how you can configure the FIM module to monitor changes in the /root/ directory of a Linux endpoint. We also show how you can obtain the same results on Windows endpoints.

Use case: Detecting malware on Linux endpoints using YARA

In this use case, we demonstrate how to configure YARA with ThreatLockDown to detect malware on Linux endpoints.

Infrastructure

Endpoint

Description

Ubuntu 22.04

The active response module runs YARA scans on new or modified files whenever the ThreatLockDown FIM module alerts about them.

Linux endpoint configuration

Perform the following steps to configure YARA and the FIM module on the monitored endpoint.

  1. Download, compile, and install YARA:

    $ sudo apt update
$ sudo apt install -y make gcc autoconf libtool libssl-dev pkg-config
$ sudo curl -LO https://github.com/VirusTotal/yara/archive/v4.2.3.tar.gz
$ sudo tar -xvzf v4.2.3.tar.gz -C /usr/local/bin/ && rm -f v4.2.3.tar.gz
$ cd /usr/local/bin/yara-4.2.3/
$ sudo ./bootstrap.sh && sudo ./configure && sudo make && sudo make install && sudo make check

  2. Test that YARA is running properly.

    $ yara

    Expected output:

    yara: wrong number of arguments
Usage: yara [OPTION]... [NAMESPACE:]RULES_FILE... FILE | DIR | PID

Try `--help` for more options

    If the error message below is displayed:

    /usr/local/bin/yara: error while loading shared libraries: libyara.so.9: cannot open shared object file: No such file or directory.

    This means that the loader doesn’t find the libyara library usually located in /usr/local/lib. Add the /usr/local/lib path to the /etc/ld.so.conf loader configuration file to solve this.

    $ sudo su
# echo "/usr/local/lib" >> /etc/ld.so.conf
# ldconfig

    Switch back to the previous user.

  3. Download YARA detection rules:

    $ sudo mkdir -p /tmp/yara/rules
$ sudo curl 'https://valhalla.nextron-systems.com/api/v1/get' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Language: en-US,en;q=0.5' \
--compressed \
-H 'Referer: https://valhalla.nextron-systems.com/' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' \
--data 'demo=demo&apikey=1111111111111111111111111111111111111111111111111111111111111111&format=text' \
-o /tmp/yara/rules/yara_rules.yar

  4. Create a /var/ossec/active-response/bin/yara.sh file and add the content below:

    #!/bin/bash
# ThreatLockDown - Yara active response
# Copyright (C) 2015-2022, ThreatLockDown Inc.
#
# This program is free software; you can redistribute it
# and/or modify it under the terms of the GNU General Public
# License (version 2) as published by the FSF - Free Software
# Foundation.


#------------------------- Gather parameters -------------------------#

# Extra arguments
read INPUT_JSON
YARA_PATH=$(echo $INPUT_JSON | jq -r .parameters.extra_args[1])
YARA_RULES=$(echo $INPUT_JSON | jq -r .parameters.extra_args[3])
FILENAME=$(echo $INPUT_JSON | jq -r .parameters.alert.syscheck.path)

# Set LOG_FILE path
LOG_FILE="logs/active-responses.log"

size=0
actual_size=$(stat -c %s ${FILENAME})
while [ ${size} -ne ${actual_size} ]; do
    sleep 1
    size=${actual_size}
    actual_size=$(stat -c %s ${FILENAME})
done

#----------------------- Analyze parameters -----------------------#

if [[ ! $YARA_PATH ]] || [[ ! $YARA_RULES ]]
then
    echo "wazuh-yara: ERROR - Yara active response error. Yara path and rules parameters are mandatory." >> ${LOG_FILE}
    exit 1
fi

#------------------------- Main workflow --------------------------#

# Execute Yara scan on the specified filename
yara_output="$("${YARA_PATH}"/yara -w -r "$YARA_RULES" "$FILENAME")"

if [[ $yara_output != "" ]]
then
    # Iterate every detected rule and append it to the LOG_FILE
    while read -r line; do
        echo "wazuh-yara: INFO - Scan result: $line" >> ${LOG_FILE}
    done <<< "$yara_output"
fi

exit 0;

    This is the active response script that executes YARA scans when FIM detects changes in the monitored directory. The active response script receives these parameters from the generated FIM alerts:

    • The file path contained in the alert that triggered the active response. The parameters.alert.syscheck.path key of the JSON alert holds the value of the file path. The path in this use case is /root/.

    • YARA_PATH: This variable specifies the path to the directory where the YARA executable is located. We installed YARA in the /usr/local/bin directory as shown in step 2 above.

    • YARA_RULES: This variable specifies the path to the file containing the YARA rules used for the scan.

    This snippet of the script uses the parameters above to perform a YARA scan and appends the results to a log file called active-responses.log:

    # Iterate every detected rule and append it to the LOG_FILE
while read -r line; do
    echo "wazuh-yara: INFO - Scan result: $line" >> ${LOG_FILE}
done <<< "$yara_output"

    For every line in the output of the YARA scan, the script appends an event to the active response log, /var/ossec/logs/active-responses.log, with the following format:

    wazuh-yara: INFO - Scan result: yara_rule file_path

    Note

    There's no need to configure the ThreatLockDown agent to monitor the active response log as it’s part of the agent default configuration.

  5. Change the script ownership and permissions with the following commands:

    $ sudo chmod 750 /var/ossec/active-response/bin/yara.sh
$ sudo chown root:wazuh /var/ossec/active-response/bin/yara.sh

  6. Install the jq utility to process the JSON data from the FIM alerts:

    $ sudo apt install -y jq

  7. Add the following within the <syscheck> block of the ThreatLockDown agent /var/ossec/etc/ossec.conf configuration file to monitor the /root/ directory:

    <directories realtime="yes">/root/</directories>

  8. Restart the ThreatLockDown agent to apply the configuration changes:

    $ sudo systemctl restart wazuh-agent

ThreatLockDown server configuration

Perform the following steps to configure ThreatLockDown FIM to alert file changes in a monitored directory on the endpoint. Then, configure the active response module to trigger a YARA scan on detection of a new or modified file.

  1. Add the following custom decoders to the /var/ossec/etc/decoders/local_decoder.xml file. This extracts information from the YARA scan results:

    <decoder name="yara_decoder">
  <prematch>wazuh-yara:</prematch>
</decoder>

<decoder name="yara_decoder1">
  <parent>yara_decoder</parent>
  <regex>wazuh-yara: (\S+) - Scan result: (\S+) (\S+)</regex>
  <order>log_type, yara_rule, yara_scanned_file</order>
</decoder>

  2. Add the following custom rules to the /var/ossec/etc/rules/local_rules.xml file:

    <group name="syscheck,">
  <rule id="100200" level="7">
    <if_sid>550</if_sid>
    <field name="file">/root/</field>
    <description>File modified in /root directory.</description>
  </rule>
  <rule id="100201" level="7">
    <if_sid>554</if_sid>
    <field name="file">/root/</field>
    <description>File added to /root directory.</description>
  </rule>
</group>

<group name="yara,">
  <rule id="108000" level="0">
    <decoded_as>yara_decoder</decoded_as>
    <description>Yara grouping rule</description>
  </rule>
  <rule id="108001" level="12">
    <if_sid>108000</if_sid>
    <match>wazuh-yara: INFO - Scan result: </match>
    <description>File "$(yara_scanned_file)" is a positive match. Yara rule: $(yara_rule)</description>
  </rule>
</group>

    The rules group syscheck detects FIM events in the monitored directory. The rules group yara alerts when the YARA integration finds malware in the monitored endpoints directory. You can modify the rules to detect file creation and modification events from other directories.

  3. Configure the execution of the YARA script when files are added or modified to a monitored directory. Edit the ThreatLockDown server /var/ossec/etc/ossec.conf configuration file and add the following:

    <ossec_config>
  <command>
    <name>yara_linux</name>
    <executable>yara.sh</executable>
    <extra_args>-yara_path /usr/local/bin -yara_rules /tmp/yara/rules/yara_rules.yar</extra_args>
    <timeout_allowed>no</timeout_allowed>
  </command>

  <active-response>
    <command>yara_linux</command>
    <location>local</location>
    <rules_id>100200,100201</rules_id>
  </active-response>
</ossec_config>

    The <command> block describes information about the action to be executed on the ThreatLockDown agent. It includes the following parameters:

    • <name>: This uniquely identifies it as the yara_linux command.

    • <executable>: This specifies the active response script or executable that must execute after a trigger. In this case, yara.sh.

    • <extra_args>: This specifies where the YARA binary and rules are located.

    • <timeout_allowed>: This specifies if the active response needs to undo an action after a specified period. It is set to no, which represents a stateless active response.

    The <active-response> block defines the criteria used to execute a specific command:

    • <command>: The previously identified yara_linux command.

    • <location>: This specifies where the command executes. Using the local value means the command executes on the ThreatLockDown agent that reported the event.

    • <rules_id>: This represents the rule IDs that trigger the command:

      • Rule 100200: File modified on the /root directory.

      • Rule 100201: New file added to the /root directory.

  4. Restart the ThreatLockDown manager to apply the configuration changes:

    $ sudo systemctl restart wazuh-manager

Test the configuration

To test that everything is working correctly, we use the Mirai and Xbash malware samples.

Warning

These malicious files are dangerous so use them for testing purposes only. Do not install them in production environments.

  1. Download the malware samples and move them into the /root/ directory of the monitored endpoint.

    $ curl https://wazuh-demo.s3-us-west-1.amazonaws.com/mirai --output ~/mirai
$ curl https://wazuh-demo.s3-us-west-1.amazonaws.com/xbash --output ~/Xbash
$ sudo mv ~/mirai /root/
$ sudo mv ~/Xbash /root/

Visualize the alerts

You can see these alerts on the ThreatLockDown dashboard. To do this, go to the Threat Hunting module of the ThreatLockDown dashboard to view the alerts.

Use case: Detecting malware on Windows endpoints using YARA

In this use case, we demonstrate how to configure YARA with ThreatLockDown to detect malware on Windows endpoints.

Infrastructure

Endpoint

Description

Windows 11

The active response module runs YARA scans on new or modified files whenever the ThreatLockDown FIM module alerts about them.

Windows endpoint configuration

Configure Python and YARA

Perform the following steps to install Python, and YARA, and download YARA rules.

  1. Download Python executable installer from the official Python website.

  2. Run the Python installer once downloaded and make sure to check the following boxes:

    • Install launcher for all users

    • Add Python 3.X to PATH. This places the interpreter in the execution path.

  3. Download and install the latest Visual C++ Redistributable package.

  4. Open PowerShell with administrator privileges to download and extract YARA:

    > Invoke-WebRequest -Uri https://github.com/VirusTotal/yara/releases/download/v4.2.3/yara-4.2.3-2029-win64.zip -OutFile v4.2.3-2029-win64.zip
> Expand-Archive v4.2.3-2029-win64.zip; Remove-Item v4.2.3-2029-win64.zip

  5. Create a directory called C:\Program Files (x86)\ossec-agent\active-response\bin\yara\ and copy the YARA executable into it:

    > mkdir 'C:\Program Files (x86)\ossec-agent\active-response\bin\yara\'
> cp .\v4.2.3-2029-win64\yara64.exe 'C:\Program Files (x86)\ossec-agent\active-response\bin\yara\'

  6. Install the valhallaAPI module:

    > pip install valhallaAPI

  7. Copy the following script and save it as download_yara_rules.py:

    from valhallaAPI.valhalla import ValhallaAPI

v = ValhallaAPI(api_key="1111111111111111111111111111111111111111111111111111111111111111")
response = v.get_rules_text()

with open('yara_rules.yar', 'w') as fh:
    fh.write(response)

  8. Run the following commands to download the rules and place them in the C:\Program Files (x86)\ossec-agent\active-response\bin\yara\rules\ directory:

    > python.exe download_yara_rules.py
> mkdir 'C:\Program Files (x86)\ossec-agent\active-response\bin\yara\rules\'
> cp yara_rules.yar 'C:\Program Files (x86)\ossec-agent\active-response\bin\yara\rules\'

Configure active response and FIM

Perform the steps below to configure the ThreatLockDown FIM and a YARA scan active response script for the detection of malicious files on the endpoint.

  1. Create the yara.bat script in the C:\Program Files (x86)\ossec-agent\active-response\bin\ directory. This is necessary for the Wazuh-YARA active response scans:

    @echo off

setlocal enableDelayedExpansion

reg Query "HKLM\Hardware\Description\System\CentralProcessor\0" | find /i "x86" > NUL && SET OS=32BIT || SET OS=64BIT


if %OS%==32BIT (
    SET log_file_path="%programfiles%\ossec-agent\active-response\active-responses.log"
)

if %OS%==64BIT (
    SET log_file_path="%programfiles(x86)%\ossec-agent\active-response\active-responses.log"
)

set input=
for /f "delims=" %%a in ('PowerShell -command "$logInput = Read-Host; Write-Output $logInput"') do (
    set input=%%a
)


set json_file_path="C:\Program Files (x86)\ossec-agent\active-response\stdin.txt"
set syscheck_file_path=
echo %input% > %json_file_path%

for /F "tokens=* USEBACKQ" %%F in (`Powershell -Nop -C "(Get-Content 'C:\Program Files (x86)\ossec-agent\active-response\stdin.txt'|ConvertFrom-Json).parameters.alert.syscheck.path"`) do (
set syscheck_file_path=%%F
)

del /f %json_file_path%
set yara_exe_path="C:\Program Files (x86)\ossec-agent\active-response\bin\yara\yara64.exe"
set yara_rules_path="C:\Program Files (x86)\ossec-agent\active-response\bin\yara\rules\yara_rules.yar"
echo %syscheck_file_path% >> %log_file_path%
for /f "delims=" %%a in ('powershell -command "& \"%yara_exe_path%\" \"%yara_rules_path%\" \"%syscheck_file_path%\""') do (
    echo wazuh-yara: INFO - Scan result: %%a >> %log_file_path%
)

exit /b

  2. Add the C:\Users\<USER_NAME>\Downloads directory for monitoring within the <syscheck> block in the ThreatLockDown agent configuration file C:\Program Files (x86)\ossec-agent\ossec.conf. Replace <USER_NAME> with the username of the endpoint:

    <directories realtime="yes">C:\Users\<USER_NAME>\Downloads</directories>

  3. Restart the ThreatLockDown agent to apply the configuration changes:

    > Restart-Service -Name wazuh

ThreatLockDown server configuration

Perform the following steps on the ThreatLockDown server. This allows alerting for changes in the endpoint monitored directory and configuring a YARA scan active response script to trigger whenever it detects a suspicious file.

  1. Add the following decoders to the ThreatLockDown server /var/ossec/etc/decoders/local_decoder.xml file. This allows extracting the information from YARA scan results:

    <decoder name="yara_decoder">
    <prematch>wazuh-yara:</prematch>
</decoder>

<decoder name="yara_decoder1">
    <parent>yara_decoder</parent>
    <regex>wazuh-yara: (\S+) - Scan result: (\S+) (\S+)</regex>
    <order>log_type, yara_rule, yara_scanned_file</order>
</decoder>

  2. Add the following rules to the ThreatLockDown server /var/ossec/etc/rules/local_rules.xml file. Replace <USER_NAME> with the username of the endpoint. The rules detect FIM events in the monitored directory. They also alert when malware is found by the YARA integration:

    <group name="syscheck,">
  <rule id="100303" level="7">
    <if_sid>550</if_sid>
    <field name="file">C:\\Users\\<USER_NAME>\\Downloads</field>
    <description>File modified in C:\Users\<USER_NAME>\Downloads directory.</description>
  </rule>
  <rule id="100304" level="7">
    <if_sid>554</if_sid>
    <field name="file">C:\\Users\\<USER_NAME>\\Downloads</field>
    <description>File added to C:\Users\<USER_NAME>\Downloads  directory.</description>
  </rule>
</group>

<group name="yara,">
  <rule id="108000" level="0">
    <decoded_as>yara_decoder</decoded_as>
    <description>Yara grouping rule</description>
  </rule>

  <rule id="108001" level="12">
    <if_sid>108000</if_sid>
    <match>wazuh-yara: INFO - Scan result: </match>
    <description>File "$(yara_scanned_file)" is a positive match. Yara rule: $(yara_rule)</description>
  </rule>
</group>

  3. Add the following configuration to the ThreatLockDown server /var/ossec/etc/ossec.conf file:

    <ossec_config>
  <command>
    <name>yara_windows</name>
    <executable>yara.bat</executable>
    <timeout_allowed>no</timeout_allowed>
  </command>

  <active-response>
    <command>yara_windows</command>
    <location>local</location>
    <rules_id>100303,100304</rules_id>
  </active-response>
</ossec_config>

  4. Restart the ThreatLockDown manager to apply the configuration changes:

    $ sudo systemctl restart wazuh-manager

Test the configuration

Note

For testing purposes, we download the EICAR anti-malware test file as shown below. We recommend testing in a sandbox, not in a production environment.

Download a malware sample on the monitored Windows endpoint:

  1. Turn off Microsoft Virus and threat protection temporarily to avoid the EICAR file removal.

  2. Download the EICAR zip file:

    > Invoke-WebRequest -Uri https://secure.eicar.org/eicar_com.zip -OutFile eicar.zip

  3. Unzip it:

    > Expand-Archive .\eicar.zip

  4. Copy the EICAR file to the monitored directory:

    > cp .\eicar\eicar.com C:\Users\<USER_NAME>\Downloads

Visualize the alerts

You can visualize the alert data in the ThreatLockDown dashboard. To do this, go to the Threat Hunting module and add the filters in the search bar to query the alerts.

  • rule.groups:yara