wodle name="aws-s3"

After adding an aws-s3 section, it is mandatory to define at least one bucket, service or subscriber. It is possible to configure multiple buckets, services and subscribers inside the same aws-s3 section.

The options available to use inside the aws-s3 section are the following:

disabled

Disables the AWS-S3 wodle.

Default value	no
Allowed values	yes, no
Mandatory	yes

skip_on_error

When unable to process and parse a log, skip it and continue processing. If set to no, the module will abort the execution once it encounters an error.

Default value	yes
Allowed values	yes, no
Mandatory	no

run_on_start

Run the module immediately after the ThreatLockDown service starts.

Default value	yes
Allowed values	yes, no
Mandatory	no

interval

The amount of time the module will wait for before running again.

Default value	10m
Allowed values	A positive number that must contain a suffix character indicating a time unit, such as, s (seconds), m (minutes), h (hours), d (days), M (months).
Mandatory	no

day

Day of the month to run the scan.

Default value	N/A
Allowed values	Day of the month [1..31]
Mandatory	no

Note

When the day option is set, the interval value must be a multiple of months. By default, the interval is set to a month.

wday

Day of the week to run the scan. This option is not compatible with the day option.

Default value	N/A
Allowed values	Day of the week: sunday/sun monday/mon tuesday/tue wednesday/wed thursday/thu friday/fri saturday/sat
Mandatory	no

Note

When the wday option is set, the interval value must be a multiple of weeks. By default, the interval is set to a week.

time

Time of the day to run the scan. It has to be in the hh:mm format.

Default value	N/A
Allowed values	Time of day [hh:mm]
Mandatory	no

Note

If only the time option is set, the interval value must be a multiple of days, weeks, or months. By default, the interval is set to a day.

Buckets

It is necessary to specify the type as an attribute of the bucket tag to indicate the service configured. More information about the supported services and their associated types on AWS supported services.

<bucket type="cloudtrail">

</bucket>

The available types are: cloudtrail, guardduty, vpcflow, config, custom, cisco_umbrella, waf, alb, clb, nlb, and server_access.

Options	Allowed values	Mandatory/Optional
name	Any valid bucket name	Mandatory
aws_account_id	Comma-separated list of AWS Accounts	Optional (only works with CloudTrail buckets)
aws_account_alias	Any string	Optional
access_key	Alphanumerical key	Optional
secret_key	Alphanumerical key	Optional
aws_profile	Any string	Optional
iam_role_arn	IAM role ARN	Optional
iam_role_duration	Number of seconds between 900 and 3600	Optional (if set, it requires an iam_role_arn to be provided)
path	Prefix for S3 bucket key	Optional
path_suffix	Suffix for S3 bucket key	Optional
only_logs_after	Date (YYYY-MMM-DDD, for example 2018-AUG-21)	Optional
regions	Comma-separated list of AWS regions	Optional (only works with CloudTrail buckets)
aws_organization_id	Name of AWS organization	Optional (only works with CloudTrail buckets)
discard_regex	A regex to determine if an event must be discarded	Optional
remove_from_bucket	A value to determine if each log file is deleted once it has been collected by the module	Optional
sts_endpoint	The AWS Security Token Service VPC endpoint URL	Optional
service_endpoint	The AWS S3 endpoint URL	Optional

name

Name of the S3 bucket from where logs are read.

Default value	N/A
Allowed values	Any valid bucket name

aws_account_id

The AWS Account ID for the bucket logs. Only works with CloudTrail buckets.

Default value	All accounts
Allowed values	Comma-separated list of 12 digit AWS Account IDs

aws_account_alias

A user-friendly name for the AWS account.

Default value	N/A
Allowed values	Any string

access_key

Deprecated since version 4.4.0.

The access key ID for the IAM user with the permission to read logs from the bucket.

Default value	N/A
Allowed values	Any alphanumerical key

secret_key

Deprecated since version 4.4.0.

The secret key created for the IAM user with the permission to read logs from the bucket.

Default value	N/A
Allowed values	Any alphanumerical key

aws_profile

A valid profile name from a Shared Credential File or AWS Config File with the permission to read logs from the bucket.

Default value	N/A
Allowed values	Valid profile name

iam_role_arn

A valid role ARN with permission to read logs from the bucket.

Default value	N/A
Allowed values	Valid role ARN

iam_role_duration

A valid number of seconds that defines the duration of the session assumed when using the provided iam_role_arn.

Default value	N/A
Allowed values	Number of seconds between 900 and 3600

path

If defined, the path or prefix for the bucket.

Default value	N/A
Allowed values	Valid path

path_suffix

If defined, the suffix for the bucket. Only works with buckets that contain the folder named AWSLogs (Cloudtrail, VPC, and Macie).

Default value	N/A
Allowed values	Valid path

only_logs_after

A valid date, in YYYY-MMM-DD format. Only logs from that date onwards will be parsed.

Default value	Date of execution at `00:00:00`
Allowed values	Valid date

regions

A comma-separated list of regions to limit parsing of logs. Only works with CloudTrail buckets.

Default value	All regions
Allowed values	Comma-separated list of valid regions

aws_organization_id

Name of AWS organization. Only works with CloudTrail buckets.

Default value	N/A
Allowed values	Valid AWS organization name

discard_regex

A regular expression to determine if an event must be discarded. It requires a mandatory field attribute. If the field is present in the event log, the regex is applied to it. For example, userIdentity.principalID for the following AWS CloudTrail log example:

{
   "eventVersion": "1.09",
   "userIdentity": {
       "type": "IAMUser",
       "principalId": "EXAMPLE6E4XEGITWATV6R",
       "arn": "arn:aws:iam::123456789012:user/Mary_Major",
       "accountId": "123456789012",
       "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
       "userName": "Mary_Major",
       "sessionContext": {
           "attributes": {
               "creationDate": "2023-07-19T21:11:57Z",
               "mfaAuthenticated": "false"
           }
       }
   },
   "eventTime": "2023-07-19T21:33:41Z",
   "eventSource": "cloudtrail.amazonaws.com",
   "eventName": "StartLogging",
   "awsRegion": "us-east-1",
   "sourceIPAddress": "192.0.2.0",
   "userAgent": "aws-cli/2.13.5 Python/3.11.4 Linux/4.14.255-314-253.539.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off command/cloudtrail.start-logging",
   "requestParameters": {
       "name": "myTrail"
   },
   "responseElements": null,
   "requestID": "9d478fc1-4f10-490f-a26b-EXAMPLE0e932",
   "eventID": "eae87c48-d421-4626-94f5-EXAMPLEac994",
   "readOnly": false,
   "eventType": "AwsApiCall",
   "managementEvent": true,
   "recipientAccountId": "123456789012",
   "eventCategory": "Management",
   "tlsDetails": {
       "tlsVersion": "TLSv1.2",
       "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
       "clientProvidedHostHeader": "cloudtrail.us-east-1.amazonaws.com"
   },
   "sessionCredentialFromConsole": "true"
}

Note

This log is the raw event log fetched from the AWS Bucket.

Default value	N/A
Allowed values	Any regex or sregex expression

Attributes:

field	The event field where to apply the regex
	Default value	N/A
	Allowed values	A str containing the full field name path

Usage example for the cloudtrail bucket type:

<discard_regex field="userIdentity.principalID">EXAMPLE6E4XEGITWATV6R</discard_regex>

remove_from_bucket

A value to determine if each log file is deleted once it has been collected by the module.

Default value	no
Allowed values	yes, no

sts_endpoint

The AWS Security Token Service VPC endpoint URL to be used when an IAM role is provided as the authentication method. Check the Considerations for configuration page to learn more about VPC endpoints.

Default value	N/A
Allowed values	Any valid VPC endpoint URL for STS

service_endpoint

The AWS S3 endpoint URL to be used to download the data from the bucket. Check the Considerations for configuration page to learn more about VPC and FIPS endpoints.

Default value	N/A
Allowed values	Any valid endpoint URL for S3

Services

It is necessary to specify the type as an attribute of the service tag to indicate the service configured. More information about the supported services and their associated types on AWS supported services.

<service type="cloudwatchlogs">

</service>

The available types are: cloudwatchlogs, and inspector.

Options	Allowed values	Mandatory/Optional
aws_account_id	Comma-separated list of 12 digit AWS Account IDs	Optional
aws_account_alias	Any string	Optional
aws_log_groups	Comma-separated list of valid log group names	Mandatory for CloudWatch Logs
access_key	Any alphanumerical key	Optional
secret_key	Any alphanumerical key	Optional
aws_profile	Valid profile name	Optional
discard_regex	A regex to determine if an event must be discarded	Optional
iam_role_arn	Valid role ARN	Optional
iam_role_duration	Number of seconds between 900 and 3600	Optional (if set, it requires an iam_role_arn to be provided)
only_logs_after	Valid date in YYYY-MMM-DD format	Optional
regions	Comma-separated list of valid regions	Optional
remove_log_streams	yes, no	Optional
sts_endpoint	Any valid VPC endpoint URL for STS	Optional
service_endpoint	Any valid endpoint URL for the AWS Service	Optional

aws_account_id

The AWS Account ID for accessing the service.

Default value	All accounts
Allowed values	Comma-separated list of 12 digit AWS Account IDs

aws_account_alias

A user-friendly name for the AWS account.

Default value	N/A
Allowed values	Any string

access_key

The access key ID for the IAM user with the permission to access the service.

Default value	N/A
Allowed values	Any alphanumerical key

aws_log_groups

A comma-separated list of log group names from where the logs should be extracted. This option is mandatory for CloudWatch Logs, and only works with that service.

Default value	N/A
Allowed values	Comma-separated list of valid log group names

secret_key

The secret key created for the IAM user with the permission to access the service.

Default value	N/A
Allowed values	Any alphanumerical key

aws_profile

A valid profile name from a Shared Credential File or AWS Config File with the permission to access the service.

Default value	N/A
Allowed values	Valid profile name

discard_regex

A regular expression to determine if an event must be discarded.

For inspector, it requires a mandatory field attribute which must be present in the fetched event. The regex is applied to the event field specified with this attribute.
For cloudwatchlogs, the field attribute is optional. You can omit it, for example, when monitoring Cloudwatch logs in JSON format and plain text.

Default value	N/A
Allowed values	Any regex or sregex expression

Attributes:

field	The event field where to apply the regex
	Default value	N/A
	Allowed values	A str containing the full field name path

Usage example for the inspector service type:

<discard_regex field="assetAttributes.agentId">i-instanceID</discard_regex>

Usage example only for cloudwatchlogs:

<discard_regex>.*Log:.*</discard_regex>

iam_role_arn

A valid role ARN with permission to access the service.

Default value	N/A
Allowed values	Valid role ARN

iam_role_duration

A valid number of seconds that defines the duration of the session assumed when using the provided iam_role_arn.

Default value	N/A
Allowed values	Number of seconds between 900 and 3600

only_logs_after

A valid date, in YYYY-MMM-DD format. Only logs from that date onwards will be parsed. This option is only available for the CloudWatch Logs service.

Default value	Date of execution at `00:00:00`
Allowed values	Valid date in YYYY-MMM-DD format

regions

A comma-separated list of regions to limit parsing of logs.

Default value	All regions
Allowed values	Comma-separated list of valid regions

remove_log_streams

Define whether or not to remove the log streams from the log groups after they are read by the module. Only works for CloudWatch Logs service.

Default value	no
Allowed values	yes, no

sts_endpoint

Default value	N/A
Allowed values	Any valid VPC endpoint URL for STS

service_endpoint

The endpoint URL for the required AWS Service to be used to download the data from it. Check the Considerations for configuration page to learn more about VPC and FIPS endpoints.

Default value	N/A
Allowed values	Any valid endpoint URL for the AWS Service

Subscribers

New in version 4.4.2.

It is necessary to specify the type as an attribute of the subscriber tag to indicate the service configured. More information about the supported services and their associated types on AWS supported services.

<subscriber type="security_lake">

</subscriber>

The currently available types are: security_lake and buckets.

Options	Allowed values	Mandatory/Optional
sqs_name	Any valid SQS name	Mandatory
iam_role_arn	Valid role ARN	Mandatory for Amazon Security Lake Subscription
external_id	Valid external ID	Mandatory for Amazon Security Lake Subscription (not available for Custom Logs Buckets)
aws_profile	Valid profile name	Optional
iam_role_duration	Number of seconds between 900 and 3600	Optional (if set, it requires an iam_role_arn to be provided)
discard_regex	A regex value to determine if an event must be discarded	Optional (only available for Custom Logs Buckets)
sts_endpoint	Any valid VPC endpoint URL for STS	Optional
service_endpoint	Any valid endpoint URL for S3	Optional

sqs_name

Name of the SQS from where notifications are pulled.

Default value	N/A
Allowed values	Any valid SQS name

external_id

External ID to use when assuming the role.

Default value	N/A
Allowed values	Valid external ID

iam_role_arn

A valid role ARN with permission to access the service.

Default value	N/A
Allowed values	Valid role ARN

iam_role_duration

A valid number of seconds that defines the duration of the session assumed when using the provided iam_role_arn.

Default value	N/A
Allowed values	Number of seconds between 900 and 3600

aws_profile

A valid profile name from a Shared Credential File or AWS Config File with the permission to access the service.

Default value	N/A
Allowed values	Valid profile name

discard_regex

A regular expression to determine if an event must be discarded. JSON and CSV logs require a mandatory field attribute. If the field is present in the event log, the regex is applied to it. For example, userIdentity.principalID for the following AWS CloudTrail log example:

{
   "eventVersion": "1.09",
   "userIdentity": {
       "type": "IAMUser",
       "principalId": "EXAMPLE6E4XEGITWATV6R",
       "arn": "arn:aws:iam::123456789012:user/Mary_Major",
       "accountId": "123456789012",
       "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
       "userName": "Mary_Major",
       "sessionContext": {
           "attributes": {
               "creationDate": "2023-07-19T21:11:57Z",
               "mfaAuthenticated": "false"
           }
       }
   },
   "eventTime": "2023-07-19T21:33:41Z",
   "eventSource": "cloudtrail.amazonaws.com",
   "eventName": "StartLogging",
   "awsRegion": "us-east-1",
   "sourceIPAddress": "192.0.2.0",
   "userAgent": "aws-cli/2.13.5 Python/3.11.4 Linux/4.14.255-314-253.539.amzn2.x86_64 exec-env/CloudShell exe/x86_64.amzn.2 prompt/off command/cloudtrail.start-logging",
   "requestParameters": {
       "name": "myTrail"
   },
   "responseElements": null,
   "requestID": "9d478fc1-4f10-490f-a26b-EXAMPLE0e932",
   "eventID": "eae87c48-d421-4626-94f5-EXAMPLEac994",
   "readOnly": false,
   "eventType": "AwsApiCall",
   "managementEvent": true,
   "recipientAccountId": "123456789012",
   "eventCategory": "Management",
   "tlsDetails": {
       "tlsVersion": "TLSv1.2",
       "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
       "clientProvidedHostHeader": "cloudtrail.us-east-1.amazonaws.com"
   },
   "sessionCredentialFromConsole": "true"
}

Note

This log is the raw event log fetched from the AWS Bucket.

Default value	N/A
Allowed values	Any regex or sregex expression

Attributes:

field	The event field where to apply the regex
	Default value	N/A
	Allowed values	A str containing the full field name path

Usage example for Cloudtrail fetched events:

<discard_regex field="userIdentity.principalID">EXAMPLE6E4XEGITWATV6R</discard_regex>

Usage example only for plain text logs:

<discard_regex>.*Log:.*</discard_regex>

sts_endpoint

Default value	N/A
Allowed values	Any valid VPC endpoint URL for STS

service_endpoint

The AWS S3 endpoint URL to be used to download the data from the bucket. Check the Considerations for configuration page to learn more about VPC and FIPS endpoints.

Default value	N/A
Allowed values	Any valid endpoint URL for S3

Example of configuration

<wodle name="aws-s3">
    <disabled>no</disabled>
    <interval>10m</interval>
    <run_on_start>no</run_on_start>
    <skip_on_error>no</skip_on_error>
    <bucket type="cloudtrail">
        <name>s3-dev-bucket</name>
        <aws_profile>default</aws_profile>
        <only_logs_after>2018-JUN-01</only_logs_after>
        <regions>us-east-1,us-west-1,eu-central-1</regions>
        <path>/dev1/</path>
        <aws_account_id>123456789012</aws_account_id>
        <aws_account_alias>dev1-account</aws_account_alias>
        <discard_regex field="userIdentity.userName">john.doe</discard_regex>
        <remove_from_bucket>yes<remove_from_bucket>
    </bucket>
    <bucket type="cloudtrail">
        <name>s3-dev-bucket</name>
        <aws_profile>default</aws_profile>
        <only_logs_after>2018-JUN-01</only_logs_after>
        <regions>us-east-1,us-west-1,eu-central-1</regions>
        <path>/dev2/</path>
        <aws_account_id>112233445566</aws_account_id>
        <aws_account_alias>dev2-account</aws_account_alias>
        <discard_regex field="userIdentity.userName">john.smith</discard_regex>
        <service_endpoint>https://bucket.xxxxxx.s3.us-east-2.vpce.amazonaws.com</service_endpoint>
    </bucket>
    <bucket type="custom">
        <name>s3-stage-bucket</name>
        <aws_profile>stage-creds</aws_profile>
        <aws_account_id>111222333444</aws_account_id>
        <aws_account_alias>stage-account</aws_account_alias>
        <discard_regex field="detail.check-item-detail.Status">Green</discard_regex>
    </bucket>
    <bucket type="custom">
        <name>s3-prod-bucket</name>
        <iam_role_arn>arn:aws:iam::010203040506:role/ROLE_SVC_Log-Parser</iam_role_arn>
        <iam_role_duration>1300</iam_role_duration>
        <aws_account_id>11112222333</aws_account_id>
        <aws_account_alias>prod-account</aws_account_alias>
        <discard_regex field="detail.status">OK</discard_regex>
        <remove_from_bucket>yes<remove_from_bucket>
    </bucket>
    <service type="cloudwatchlogs">
        <aws_profile>default</aws_profile>
        <aws_log_groups>log_group1,log_group2</aws_log_groups>
        <only_logs_after>2018-JUN-01</only_logs_after>
        <regions>us-east-1,us-west-1,eu-central-1</regions>
        <discard_regex>.*Log Hostname1:.*</discard_regex>
    </service>
    <subscriber type="security_lake">
      <sqs_name>sqs-security-lake-main-queue</sqs_name>
      <external_id>wazuh-external-id-value</external_id>
      <iam_role_arn>arn:aws:iam::010203040506:role/ASL-Role</iam_role_arn>
    </subscriber>
    <subscriber type="buckets">
      <sqs_name>sqs-custom-logs-queue</sqs_name>
      <aws_profile>dev</aws_profile>
    </subscriber>
</wodle>