OpenSearch integration

Installing Logstash

Perform the following steps to install Logstash and the required plugins. Ensure your Logstash and OpenSearch versions are compatible.

1. Follow the Elastic documentation to install Logstash.

2. Install the logstash-input-opensearch plugin and the logstash-output-opensearch plugin using the following command. These plugins allow reading the data from the ThreatLockDown indexer into the Logstash pipeline and writing the data into OpenSearch.

   $ sudo /usr/share/logstash/bin/logstash-plugin install logstash-input-opensearch logstash-output-opensearch
   

3. Copy the ThreatLockDown indexer and OpenSearch root certificates on the Logstash server.

   Note

   You can add the certificates to any directory of your choice. For example, we added them in /etc/logstash/wazuh-indexer-certs and /etc/logstash/opensearch-certs respectively.

4. Give the logstash user the necessary permissions to read the copied certificates:

   $ sudo chmod -R 755 </PATH/TO/LOCAL/WAZUH-INDEXER/CERTIFICATE>/root-ca.pem
   $ sudo chmod -R 755 </PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem
   

   Replace </PATH/TO/LOCAL/WAZUH-INDEXER/CERTIFICATE>/root-ca.pem and </PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem with your ThreatLockDown indexer and Opensearch certificate local path on the Logstash endpoint respectively.

Configuring new indexes

You must define the mappings between the data and the index types to ensure OpenSearch indexes your data correctly. OpenSearch can infer these mappings, but we recommend that you explicitly configure them. ThreatLockDown provides a set of mappings to ensure OpenSearch indexes the data correctly.

You need to use the logstash/os_template.json template to configure this index initialization for your OpenSearch platform.

Create a /etc/logstash/templates/ directory and download the template as wazuh.json using the following commands:

# mkdir /etc/logstash/templates
# curl -o /etc/logstash/templates/wazuh.json https://packages.wazuh.com/integrations/opensearch/4.x-2.x/dashboards/wz-os-4.x-2.x-template.json

In OpenSearch, the indexes support up to 1000 fields by default. However, ThreatLockDown logs might contain even more than this number of fields. To solve this issue, the provided wazuh.json template has the fields set to 10000 by default as shown below:

...
"template": {
  ...
  "settings": {
        ...
        "mapping": {
         "total_fields": {
            "limit": 10000
         }
        }
        ...
  }
  ...
}
...

You can further increase this value by following the creating an index template documentation.

Configuring a pipeline

A Logstash pipeline allows Logstash to use plugins to read the data from the ThreatLockDown indexes and send them to OpenSearch.

The Logstash pipeline requires access to the following secret values:

• ThreatLockDown indexer credentials

• OpenSearch credentials

We use the Logstash keystore to securely store these values.

1. Run the following commands on your Logstash server to set a keystore password:

   $ set +o history
   $ echo 'LOGSTASH_KEYSTORE_PASS="<MY_KEYSTORE_PASSWORD>"'| sudo tee /etc/sysconfig/logstash
   $ export LOGSTASH_KEYSTORE_PASS=<MY_KEYSTORE_PASSWORD>
   $ set -o history
   $ sudo chown root /etc/sysconfig/logstash
   $ sudo chmod 600 /etc/sysconfig/logstash
   $ sudo systemctl start logstash
   

   Where <MY_KEYSTORE_PASSWORD> is your keystore password.

   Note

   You need to create the /etc/sysconfig folder if it does not exist on your server.

2. Run the following commands to securely store the credentials of the ThreatLockDown indexer and OpenSearch in the Logstash keystore.

   Note

   When you run each of the commands, you will be prompted to enter your credentials and the credentials will not be visible as you enter them.

   OPENSEARCH_USERNAME, OPENSEARCH_PASSWORD, WAZUH_INDEXER_USERNAME, and WAZUH_INDEXER_PASSWORD are keys representing the secret values you are adding to the Logstash keystore. These keys will be used in the Logstash pipeline.

   1. Create a new Logstash keystore:

      $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
      

   2. Store your OpenSearch username and password:

      $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add OPENSEARCH_USERNAME
      $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add OPENSEARCH_PASSWORD
      

   3. Store your ThreatLockDown indexer administrator username and password:

      $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add WAZUH_INDEXER_USERNAME
      $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add WAZUH_INDEXER_PASSWORD
      

   Where:

   • OPENSEARCH_USERNAME and OPENSEARCH_PASSWORD are keys representing your OpenSearch username and password respectively.

   • WAZUH_INDEXER_USERNAME and WAZUH_INDEXER_PASSWORD are keys representing your ThreatLockDown indexer administrator username and password respectively.

3. Perform the following steps to configure the Logstash pipeline.

   1. Create the configuration file wazuh-opensearch.conf in /etc/logstash/conf.d/ folder:

      $ sudo touch /etc/logstash/conf.d/wazuh-opensearch.conf
      

   2. Add the following configuration to the wazuh-opensearch.conf file. This sets the parameters required to run Logstash.

      input {
        opensearch {
         hosts =>  ["<WAZUH_INDEXER_ADDRESS>:9200"]
         user  =>  "${WAZUH_INDEXER_USERNAME}"
         password  =>  "${WAZUH_INDEXER_PASSWORD}"
         index =>  "wazuh-alerts-4.x-*"
         ssl => true
         ca_file => "</PATH/TO/LOCAL/WAZUH-INDEXER/CERTIFICATE>/root-ca.pem"
         query =>  '{
             "query": {
                "range": {
                   "@timestamp": {
                      "gt": "now-1m"
                   }
                }
             }
         }'
         schedule => "* * * * *"
        }
      }
      
      output {
          opensearch {
            hosts => ["<OPENSEARCH_ADDRESS>"]
            auth_type => {
               type => 'basic'
               user => '${OPENSEARCH_USERNAME}'
               password => '${OPENSEARCH_PASSWORD}'
            }
            index  => "wazuh-alerts-4.x-%{+YYYY.MM.dd}"
            cacert => "</PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem"
            ssl => true
            template => "/etc/logstash/templates/wazuh.json"
            template_name => "wazuh"
            template_overwrite => true
            legacy_template => false
          }
      }
      

      Where:

      • <WAZUH_INDEXER_ADDRESS> is your ThreatLockDown indexer address or addresses in case of cluster deployment.

      • <OPENSEARCH_ADDRESS> is your OpenSearch address.

      • </PATH/TO/LOCAL/WAZUH-INDEXER/CERTIFICATE>/root-ca.pem is your ThreatLockDown indexer certificate local path on the ThreatLockDown server. For example, you can use /etc/logstash/wazuh-indexer-certs/root-ca.pem which is the ThreatLockDown indexer root certificate that was copied earlier.

      • </PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem is your OpenSearch certificate local path on the ThreatLockDown server. For example, you can use /etc/logstash/opensearch-certs/root-ca.pem which is the OpenSearch certificate that was copied earlier.

      Note

      For testing purposes, you can avoid SSL verification by replacing cacert => "</PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem" with ssl_certificate_verification => false.

      If you aren't using composable index templates and the _index_template API, remove the legacy_template => false parameter.

Running Logstash

1. Once you have everything set, run Logstash from CLI with your configuration:

   $ sudo systemctl stop logstash
   $ sudo -E /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/wazuh-opensearch.conf --path.settings /etc/logstash/
   

   Make sure to use your own paths for the Logstash executable, the pipeline, and the configuration files.

   Ensure that ThreatLockDown indexer RESTful API port (9200) is open on your ThreatLockDown indexer. To verify that the necessary ports for ThreatLockDown component communication are open, refer to the list of required ports.

2. After confirming that the configuration loads correctly without errors, cancel the command and run Logstash as a service. This way Logstash is not dependent on the lifecycle of the terminal it's running on. You can now enable and run Logstash as service:

   $ sudo systemctl enable logstash
   $ sudo systemctl start logstash
   

Check Elastic documentation for more details on setting up and running Logstash.

After Logstash is successfully running, check how to configure the ThreatLockDown alert index pattern and verify the integration.

Installing Logstash

We use Logstash to forward security data in the /var/ossec/logs/alerts/alerts.json alerts file from the ThreatLockDown server to the OpenSearch indexes.

Perform the following steps to install Logstash and the required plugin.

1. Follow the Elastic documentation to install Logstash on the ThreatLockDown server.

2. Run the following command to install the logstash-output-opensearch plugin. This plugin allows Logstash to write the data into OpenSearch.

   $ sudo /usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch
   

3. Copy the OpenSearch root certificate to the ThreatLockDown server. You can add the certificate to any directory of your choice. In our case, we add it in the /etc/logstash/opensearch-certs directory.

4. Give the logstash user the necessary permissions to read the copied certificates:

   $ sudo chmod -R 755 </PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem
   

   Replace </PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem with your OpenSearch certificate local path on the ThreatLockDown server.

Configuring new indexes

You must define the mappings between the data and the index types to ensure Opensearch indexes your data correctly. Opensearch can infer these mappings, but we recommend that you explicitly configure them. ThreatLockDown provides a set of mappings to ensure Opensearch indexes the data correctly.

You need to use the logstash/os_template.json template to configure this index initialization for your Opensearch platform. The refresh_interval is set to 5s in the template we provide.

Create a /etc/logstash/templates/ directory and download the template as wazuh.json using the following commands:

# mkdir /etc/logstash/templates
# curl -o /etc/logstash/templates/wazuh.json https://packages.wazuh.com/integrations/opensearch/4.x-2.x/dashboards/wz-os-4.x-2.x-template.json

In OpenSearch, the indexes support up to 1000 fields by default. However, ThreatLockDown logs might contain even more than this number of fields. To solve this issue, the provided wazuh.json template has the fields set to 10000 by default as shown below:

...
"template": {
  ...
  "settings": {
        ...
        "mapping": {
         "total_fields": {
            "limit": 10000
         }
        }
        ...
  }
  ...
}
...

You can further increase this value by following the creating an index template documentation.

Configuring a pipeline

A Logstash pipeline allows Logstash to use plugins to read the data in the ThreatLockDown /var/ossec/logs/alerts/alerts.json alerts file and send them to OpenSearch.

The Logstash pipeline requires access to your OpenSearch credentials.

We use the Logstash keystore to securely store these values.

1. Run the following commands on your Logstash server to set a keystore password:

   $ set +o history
   $ echo 'LOGSTASH_KEYSTORE_PASS="<MY_KEYSTORE_PASSWORD>"'| sudo tee /etc/sysconfig/logstash
   $ export LOGSTASH_KEYSTORE_PASS=<MY_KEYSTORE_PASSWORD>
   $ set -o history
   $ sudo chown root /etc/sysconfig/logstash
   $ sudo chmod 600 /etc/sysconfig/logstash
   $ sudo systemctl start logstash
   

   Where <MY_KEYSTORE_PASSWORD> is your keystore password.

   Note

   You need to create the /etc/sysconfig folder if it does not exist on your server.

2. Run the following commands to securely store the credentials of OpenSearch.

   Note

   When you run each of the commands, you will be prompted to enter your credentials and the credentials will not be visible as you enter them.

   OPENSEARCH_USERNAME and OPENSEARCH_PASSWORD are keys representing the secret values you are adding to the Logstash keystore. These keys will be used in the Logstash pipeline.

   1. Create a new Logstash keystore:

      $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
      

   2. Store your OpenSearch username and password:

      $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add OPENSEARCH_USERNAME
      $ sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add OPENSEARCH_PASSWORD
      

      Where OPENSEARCH_USERNAME and OPENSEARCH_PASSWORD are keys representing your OpenSearch username and password respectively.

3. Perform the following steps to configure the Logstash pipeline.

   1. Create the configuration file wazuh-opensearch.conf in /etc/logstash/conf.d/ folder:

      $ sudo touch /etc/logstash/conf.d/wazuh-opensearch.conf
      

   2. Add the following configuration to the wazuh-opensearch.conf file. This sets the parameters required to run Logstash.

      input {
        file {
          id => "wazuh_alerts"
          codec => "json"
          start_position => "beginning"
          stat_interval => "1 second"
          path => "/var/ossec/logs/alerts/alerts.json"
          mode => "tail"
          ecs_compatibility => "disabled"
        }
      }
      
      output {
          opensearch {
            hosts => ["<OPENSEARCH_ADDRESS>"]
            auth_type => {
               type => 'basic'
               user => '${OPENSEARCH_USERNAME}'
               password => '${OPENSEARCH_PASSWORD}'
            }
            index  => "wazuh-alerts-4.x-%{+YYYY.MM.dd}"
            cacert => "</PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem"
            ssl => true
            template => "/etc/logstash/templates/wazuh.json"
            template_name => "wazuh"
            template_overwrite => true
            legacy_template => false
          }
      }
      

      Where:

      • <OPENSEARCH_ADDRESS> is your OpenSearch IP address.

      • </PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem is your OpenSearch certificate local path on the ThreatLockDown server. In our case, we used /etc/logstash/opensearch-certs/root-ca.pem.

      Note

      For testing purposes, you can avoid SSL verification by replacing cacert => "</PATH/TO/LOCAL/OPENSEARCH/CERTIFICATE>/root-ca.pem" with ssl_certificate_verification => false.

      If you aren't using composable index templates and the _index_template API, remove the legacy_template => false parameter.

4. By default the /var/ossec/logs/alerts/alerts.json file is owned by the wazuh user with restrictive permissions. You must add the logstash user to the wazuh group so it can read the file when running Logstash as a service:

   $ sudo usermod -a -G wazuh logstash
   

Running Logstash

1. Once you have everything set, run Logstash from CLI with your configuration:

   $ sudo systemctl stop logstash
   $ sudo -E /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/wazuh-opensearch.conf --path.settings /etc/logstash/
   

   Make sure to use your own paths for the executable, the pipeline, and the configuration files.

   Ensure that ThreatLockDown server RESTful API port (55000) is open on your ThreatLockDown server. To verify that the necessary ports for ThreatLockDown component communication are open, refer to the list of required ports.

2. After confirming that the configuration loads correctly without errors, cancel the command and run Logstash as a service. This way Logstash is not dependent on the lifecycle of the terminal it's running on. You can now enable and run Logstash as a service:

   $ sudo systemctl enable logstash
   $ sudo systemctl start logstash
   

Check Elastic documentation for more details on setting up and running Logstash.