Command output analysis

ThreatLockDown is shipped with a default ruleset that analyzes events from monitored endpoints and generates security alerts. The ThreatLockDown server processes the output of commands executed on the endpoints by the Command or the Logcollector module. ThreatLockDown processes the output like it processes logs from other sources. There are several stages of log processing in ThreatLockDown including pre-decoding, decoding, and rule matching.

Decoding and rule matching

Decoders extract defined fields from logs sent to the ThreatLockDown server. The pre-decoding phase of analysis extracts information such as timestamp, hostname, and program name from the log header. After this phase, the analysis engine looks for a decoder matching a log. When a decoder is found, it extracts the fields of the logs and matches the appropriate rules to generate security alerts. When a decoder is not found, the log is ignored.

Utilize the Wazuh-Logtest utility to simulate and visualize the process of decoding and matching logs against rules. When a decoder is found, it is used to extract the defined fields, otherwise, it throws the message, No decoder matched.

Viewing the raw logs

The ThreatLockDown archives provide the option to view the raw log format of executed commands that the ThreatLockDown server processes. This feature allows you to gather log information required to develop decoders when no decoders are available to process specific log formats.

When you enable the ThreatLockDown archives, all the logs the ThreatLockDown server receives are stored in the /var/ossec/logs/archives/archives.log and /var/ossec/logs/archives/archives.json files on the ThreatLockDown server. By default, this feature is disabled on the ThreatLockDown server to prevent the server from storing raw logs and filling the disk storage.

To view the output of executed commands, you need to enable the ThreatLockDown archives on the ThreatLockDown server using the following steps:

  1. Set the value of the <logall> and <logall_json> tags to yes within the <global> block of the ThreatLockDown server /var/ossec/etc/ossec.conf file.

  2. Restart the ThreatLockDown manager to apply the configuration changes:

    # systemctl restart wazuh-manager

  3. Open the /var/ossec/logs/archives/archives.log and /var/ossec/logs/archives/archives.json files on the ThreatLockDown server to view the newly received logs.

Visit the logall and logall_json options of the global configuration section to get detailed information.

Note

As earlier stated, the ThreatLockDown archives retain logs collected from all monitored endpoints, therefore consuming significant storage resources on the ThreatLockDown server over time. So, you might want to disable it after successful log analyses.

Creating a custom ruleset

The ThreatLockDown agent collects logs and events that contain specific information on monitored endpoints. You need to create a custom ruleset to extract relevant information from the collected logs.

To demonstrate the process of creating a custom ruleset, we configure the Command module to obtain the amount of physical RAM left unused in a Linux endpoint every five minutes. In this example, the module executes a command to generate memory usage events on the Linux endpoint and forward the logs to the ThreatLockDown server.

Perform the steps below to generate the events and create a custom ruleset.

ThreatLockDown server

  1. Enable the ThreatLockDown archives to store the raw logs by setting the value of <logall_json> to yes within the <global> block of the /var/ossec/etc/ossec.conf file.

  2. Restart the ThreatLockDown manager to apply the configuration changes:

    # systemctl restart wazuh-manager

Linux endpoint

  1. Add the Command module configuration below within the <ossec_config> block of the /var/ossec/etc/ossec.conf file:

    <wodle name="command">
  <disabled>no</disabled>
  <tag>unused_memory</tag>
  <command>grep MemFree /proc/meminfo</command>
  <interval>5m</interval>
  <ignore_output>no</ignore_output>
  <run_on_start>yes</run_on_start>
  <timeout>0</timeout>
</wodle>

  2. Restart the ThreatLockDown agent to apply the configuration changes:

    # systemctl restart wazuh-agent

ThreatLockDown server

  1. Run the command below to obtain the log received from the monitored Linux endpoint:

    # grep "unused_memory" /var/ossec/logs/archives/archives.json

    {"timestamp":"2023-07-26T09:06:08.947+0000","agent":{"id":"002","name":"Ubuntu-22-LTS","ip":"10.0.2.15"},"manager":{"name":"wazuh-server"},"id":"1690362368.662599","full_log":"MemFree:          90008 kB","decoder":{},"location":"command_unused_memory"}

    The full_log field in the log contains the actual event generated on the monitored Linux endpoint.

  2. Run the /var/ossec/bin/wazuh-logtest program and paste the value of the full_log field in the prompt:

    # /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.9.0
Type one log per line

MemFree:          90008 kB

**Phase 1: Completed pre-decoding.
          full event: 'MemFree:           90008 kB'

**Phase 2: Completed decoding.
          No decoder matched.

    You can see that there is no decoder available to decode the log as highlighted. You need to create a decoder to extract information from the log.

  3. To extract information such as the free memory and unit of measurement from the log, add the following custom decoder to the /var/ossec/etc/decoders/local_decoder.xml file:

    <decoder name="unused-memory">
  <prematch>^MemFree: </prematch>
  <regex offset="after_prematch">\t*(\S+)\s(\S+)</regex>
  <order>free_memory, unit_of_measurment</order>
</decoder>

  4. Run the /var/ossec/bin/wazuh-logtest program and paste the value of the full_log field in the log again. This time you can see that the log is properly decoded by the decoder unused-memory and the fields free_memory and unit_of_measurment are extracted accordingly:

    # /var/ossec/bin/wazuh-logtest
Starting wazuh-logtest v4.9.0
Type one log per line

MemFree:          90008 kB

**Phase 1: Completed pre-decoding.
          full event: 'MemFree:           90008 kB'

**Phase 2: Completed decoding.
          name: 'unused-memory'
          free_memory: '90008'
          unit_of_measurment: 'kB'

  5. The decoded output of the command needs to trigger a rule to generate an alert. Add the custom rule below to the /var/ossec/etc/rules/local_rules.xml file to generate an alert when the Command module executes the grep MemFree /proc/meminfo command. The rule uses the <decoded_as> tag to reference the unused-memory decoder created above:

    <group name="unused_memory">
  <rule id="100003" level="5">
    <decoded_as>unused-memory</decoded_as>
    <description>The system’s free memory is $(free_memory) $(unit_of_measurment).</description>
  </rule>
</group>

  6. Restart the ThreatLockDown manager to apply the configuration changes:

    # systemctl restart wazuh-manager

  7. Run the command below to see the JSON alert generated when the Command module runs the grep MemFree /proc/meminfo command:

    # grep "unused_memory" /var/ossec/logs/alerts/alerts.json | /var/ossec/framework/python/bin/python3 -mjson.tool

    {
    "timestamp": "2023-07-26T09:21:07.928+0000",
    "rule": {
        "level": 5,
        "description": "The system\u2019s free memory is 88456 kB.",
        "id": "100003",
        "firedtimes": 1,
        "mail": false,
        "groups": [
            "unused_memory"
        ]
    },
    "agent": {
        "id": "002",
        "name": "Ubuntu-22-LTS",
        "ip": "10.0.2.15"
    },
    "manager": {
        "name": "wazuh-server"
    },
    "id": "1690363267.663636",
    "full_log": "MemFree:           88456 kB",
    "decoder": {
        "name": "unused-memory"
    },
    "data": {
        "free_memory": "88456",
        "unit_of_measurment": "kB"
    },
    "location": "command_unused_memory"
}

ThreatLockDown dashboard

Click the upper-left menu icon ☰ to open the options, go to Server management > Endpoints summary and select the monitored Linux endpoint. Then, navigate to the Threat Hunting module to view the alerts generated when the Command module runs the grep MemFree /proc/meminfo command.

System's free memory alert