Custom Logs Buckets

New in version 4.7.0.

Amazon Simple Queue Service (Amazon SQS) is a fully managed message queuing service. It offers secure, durable, and available hosted queues to decouple and scale software systems and components. It allows sending, storing, and receiving messages between software components at any volume, without losing messages or requiring other services to be available. These features make it an optimal component to associate with Amazon S3 buckets to consume any type of log.

Combining Amazon SQS with Amazon S3 buckets allows ThreatLockDown to fetch JSON, CSV, and plain text logs from any custom path. The origin of these logs don't even need to be AWS.

Note

To properly process CSV logs, they must include column headers.

To set up the ThreatLockDown integration for Custom Logs Buckets, you need to do the following:

  1. Create an AWS SQS Queue.

  2. Configure an S3 bucket. For every object creation event, the bucket sends notifications to the queue.

AWS configuration

Amazon Simple Queue Service

  1. Set up a Standard type SQS Queue with the default configurations. You can apply an Access Policy similar to the following example, where <region>, <account-id>, and <s3-bucket> are the region, account ID, and the name you are going to provide to the S3 bucket.

    {
"Version": "2012-10-17",
"Id": "example-ID",
"Statement": [
  {
    "Sid": "example-access-policy",
    "Effect": "Allow",
    "Principal": {
      "Service": "s3.amazonaws.com"
    },
    "Action": "SQS:SendMessage",
    "Resource": "arn:aws:sqs:<region>:<account-id>:<s3-bucket>",
    "Condition": {
      "StringEquals": {
        "aws:SourceAccount": "<account-id>"
      },
      "ArnLike": {
        "aws:SourceArn": "arn:aws:s3:*:*:<s3-bucket>"
      }
    }
  }
]
}

    You can make your access policy to accept S3 notifications from different account IDs and to apply different conditions. More information in Managing access in Amazon SQS.

Amazon S3 and Event Notifications

To configure an S3 bucket that reports creation events, do the following.

  1. Configure an S3 bucket as defined in the Configuring an S3 Bucket section. Provide the name you decided in the previous section.

  2. Once created, go to Event notifications inside the Properties tab. Select Create event notification.

  3. In Event Types, select All object create events. This generates notifications for any type of event that results in the creation of an object in the bucket.

  4. In the Destination section, select the following options:

    • SQS queue

    • Choose from your SQS queues

  5. Choose the queue you created previously.

ThreatLockDown Configuration

Warning

Every message sent to the queue is read and deleted. Make sure you only use the queue for bucket notifications.

  1. Edit the /var/ossec/etc/ossec.conf file. Add the SQS name and your Configuration parameters for the buckets service. Set this inside <subscriber type="buckets">. For example:

    <wodle name="aws-s3">
    <disabled>no</disabled>
    <interval>1h</interval>
    <run_on_start>yes</run_on_start>
    <subscriber type="buckets">
        <sqs_name>sqs-queue</sqs_name>
        <aws_profile>default</aws_profile>
    </subscriber>
</wodle>

    Check the AWS S3 module reference manual to learn more about the available settings.

    Note

    The amount of notifications present in the queue affects the execution time of the AWS S3 module. If the <interval> value for the waiting time between executions is too short, the Interval overtaken warning is logged into the ossec.log file.

  2. Restart the ThreatLockDown manager to apply the changes.

    # systemctl restart wazuh-manager

Configuration parameters

Configure the following fields to set the queue and authentication configuration. For more information, check the Subscribers reference.

Queue

  • <sqs_name>: The name of the queue.

  • Optional – <service_endpoint>: The AWS S3 endpoint URL for data downloading from the bucket. Check Using non-default AWS endpoints for more information about VPC and FIPS endpoints.

Authentication

The available authentication methods are the following:

These authentication methods require using the /root/.aws/credentials file to provide credentials. You can find more information in Configuring AWS credentials.

The available authentication configuration parameters are the following:

  • <aws_profile>: A valid profile name from a Shared Credential File or AWS Config File with the permission to read logs from the bucket.

  • <iam_role_arn>: ARN for the corresponding IAM role to assume.

  • Optional – <iam_role_duration>: The session duration in seconds.

  • Optional – <sts_endpoint>: The URL of the VPC endpoint of the AWS Security Token Service.