AWS S3 Integration

You can use AWS S3 as a source or a destination. Continue below to integrate as a source, or click here to integrate as a destination.

AWS S3 as a Source

Send logs from S3 bucket to Realm

Create SQS Queue for notifications

1. Go to SQS Queue
2. Create Queue
3. Fill out details

Name: rlm-s3-event-notifications

4. Copy the ARN of the queue

Create IAM policy

1. Go to IAM > Policy
2. Create Policy
3. Click JSON
4. Paste the following and replace s3_bucket_arn with the name of your s3 bucket and sqs_queue_arn with the SQS queue from above

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "s3ReadObjects",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "<s3_bucket_arn>",
                "<s3_bucket_arn>/*"
            ]
        },
        {
            "Sid": "sqsEventNotifications",
            "Effect": "Allow",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Resource": [
                "<sqs_queue_arn>"
            ]
        }
    ]
}

5. Click next.
6. Enter policy name: rlm-s3-notifications-and-read
7. Enter description: Grant read access to s3 cloud trail bucket as well as receiving s3 notifications from the SQS queue.
8. Click Create Policy.

Create User with credentials

1. Go to IAM > Users > Create User
2. Enter Name: rlm-s3-read-user
3. Click Next
4. Click Attach Policy Directly
5. Select the policy created in the previous step
6. Click Next
7. Click Create user
8. Select the user that was just created
9. Copy the ARN of the user, and save it to a safe location - you will need it in the next step
10. Go to Security Credentials
11. Click Create Access Key
12. Select Third party Service
13. Check the confirmation checkbox
14. Click Next
15. Enter description: Credentials for Realm.Security to read Cloudtrail logs from S3 bucket
16. Copy and save the Access Key and Secret access key in a safe location, you will need to paste these two values in Realm console when configuring the S3 input feed

Update SQS policy

1. Go to SQS Queues
2. Select the rlm-event-notification queue
3. Go to Queue Policies > Edit Access policy
4. Replace the policy JSON with the following

{
  "Version": "2012-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "Stmt1737666508309",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "<sqs_queue_arn>"
    },
    {
      "Sid": "Stmt1737666814690",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<iam_user_arn>"
      },
      "Action": [
        "sqs:ChangeMessageVisibility",
        "sqs:DeleteMessage",
        "sqs:ReceiveMessage"
      ],
      "Resource": "<sqs_queue_arn>"
    }
  ]
}

5. Click Save

Configure notifications for s3 bucket

1. Login to AWS console
2. Go to S3 buckets
3. Select the bucket
4. Go to Properties
5. Go to Event Notifications and click on Create Event Notification
6. Fill out the event notification details

Event name: RlmCreateEvents
Select All object create events checkbox
Destination: Select SQS Queue
In the drop down, select SQS queue that you created in the step above

7. Click Save Changes

AWS S3 as a Destination

Send logs from Realm to S3 bucket

1. Create S3 Bucket.
2. Login to AWS console.
3. Go to S3 > Buckets.
4. Click Create Bucket.
5. Enter bucket name:

rlm-demo-output

6. Click Create bucket.
7. Copy name of the bucket.

Note: Versioning is disabled by default.

Create policy that grants write permission to the above S3 bucket

1. In a new Tab, Go to IAM > Policies.
2. Click on Create Policy.
3. Click JSON.
4. Paste the following policy and replace <BUCKET_NAME> with the name of the bucket created above.

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "S3WriteRealmBucket",
			"Effect": "Allow",
			"Action": [
				"s3:PutObject",
				"s3:ListBucket",
				"s3:PutObjectAcl"
			],
			"Resource": [
        "arn:aws:s3:::<BUCKET_NAME>",
        "arn:aws:s3:::<BUCKET_NAME>/*"
      ]
		}
	]
}

Create IAM user

1. Go to IAM > Users.
2. Click Create User.
3. Give it a name:

<tenant_name>-realm.security-s3-output

4. Click Next.
5. Click Attach policies directly.
6. From the list, search and select Policy created above.
7. Click Next.
8. Click Create user.
9. Select user that was just created.
10. Click Security Credentials.
11. Click Create Access Keys.
12. Click Third Party Service.
13. Select "I understand" checkbox.
14. Click Next.
15. Enter description:

Allows <tenant_name> in stage to write to <bucket_name> s3 bucket

Structure

Within the configured S3 bucket, the data will be stored in the following structure by default s3://{bucket_name}/{destination_name}/{source_name}/YYYY/MM/DD/*.log.gz.

To use a different S3 prefix structure, update the Key Prefix field for output feed. For example, if Key Prefix field is set to foo, the bucket structure will be s3://{bucket_name}/foo/*.log.gz.

You can include date with key_prefix using special place holders %Y, %m, %d. For example

• when key_prefix is set to foo/year=%Y/month=%m/day=%d/, it will create following structure in S3 s3://{bucket_name}/foo/year=2025/month=10/day=23/.
• when key_prefix is set to foo/%Y/%m/%d/, it will create following structure in S3 s3://{bucket_name}/foo/2025/10/23/.

Each file uses zstd compression and the file contains events that are separated by new lines. The format of each event (RAW vs JSON) is determined by the selected Format for the AWS S3 output feed.