AWS S3 Integration

You can use AWS S3 as a source or a destination. Continue below to integrate as a source, or click here to integrate as a destination.

AWS S3 as a Source

Realm supports two approaches for receiving logs from S3 bucket:

Option 1 — Via SQS (S3 → SQS): Configure S3 to send event notifications directly to a new SQS queue.
Option 2 — Via SNS & SQS (S3 → SNS → SQS): Subscribe a new SQS queue to an existing SNS topic that already receives S3 events.

Both approaches share the same SQS queue, IAM policy, and IAM user setup. Follow the relevant section below based on your setup.

Option 1: Via SQS (S3 → SQS)

Use this approach if your S3 bucket does not already have an SNS topic configured for event notifications.

Step 1: Create SQS Queue

Go to SQS > Queues
Click Create Queue
Enter the queue name:

Name: rlm-s3-event-notifications

Click Create Queue
Copy the ARN of the queue — you will need it in the steps below

Step 2: Create IAM Policy

Go to IAM > Policies
Click Create Policy
Click JSON and replace the contents with the following, substituting <s3_bucket_arn> with your S3 bucket ARN and <sqs_queue_arn> with the SQS queue ARN from Step 1

json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "s3ReadObjects",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "<s3_bucket_arn>",
                "<s3_bucket_arn>/*"
            ]
        },
        {
            "Sid": "sqsEventNotifications",
            "Effect": "Allow",
            "Action": [
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Resource": [
                "<sqs_queue_arn>"
            ]
        }
    ]
}

Click Next
Enter the policy details:

Name: rlm-s3-notifications-and-read
Description: Grant read access to S3 bucket and permission to receive S3 notifications from the SQS queue.

Click Create Policy

Step 3: Create IAM User

Go to IAM > Users and click Create User
Enter the username:

Name: rlm-s3-read-user

Click Next
Select Attach policies directly
Search for and select the rlm-s3-notifications-and-read policy created in Step 2
Click Next, then Create user
Open the user that was just created and copy its ARN — you will need it in Step 4
Go to Security credentials and click Create access key
Select Third-party service, check the confirmation checkbox, then click Next
Enter a description:

Credentials for Realm.Security to read logs from S3 bucket

Click Create access key
Copy and save the Access Key and Secret access key — you will need these when configuring the S3 input feed in Realm

Step 4: Update SQS Access Policy

The SQS queue policy must allow S3 to publish notifications to the queue and allow the IAM user to read from it.

Go to SQS > Queues
Select the rlm-s3-event-notifications queue
Go to Queue policies and click Edit
Replace the policy JSON with the following, substituting <sqs_queue_arn> with the SQS queue ARN and <iam_user_arn> with the IAM user ARN from Step 3

json

{
  "Version": "2012-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "AllowS3Publish",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "<sqs_queue_arn>"
    },
    {
      "Sid": "AllowRealmUser",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<iam_user_arn>"
      },
      "Action": [
        "sqs:ChangeMessageVisibility",
        "sqs:DeleteMessage",
        "sqs:ReceiveMessage"
      ],
      "Resource": "<sqs_queue_arn>"
    }
  ]
}

Click Save

Step 5: Configure S3 Event Notifications

Go to S3 > Buckets and select your bucket
Go to Properties > Event notifications
Click Create event notification
Fill out the details:

Event name: RlmCreateEvents
Check All object create events
Destination: SQS Queue
Select rlm-s3-event-notifications from the dropdown

Click Save changes

Option 2: Via SNS → SQS

Use this approach to subscribe a new SQS queue to an SNS topic that receives S3 ObjectCreate notifications, then connect that queue to Realm.

Skip this step if you already have an SNS topic configured to receive S3 ObjectCreate notifications. Copy its ARN and proceed to Step 2.

Create the SNS topic:

Go to SNS > Topics
Click Create topic
Select Standard as the type
Enter the topic name:

Name: rlm-s3-event-notifications-topic

Click Create topic
Copy the ARN of the topic — you will need it in the steps below

Update the SNS topic access policy:

The SNS topic policy must allow S3 to publish to it.

On the topic details page, go to Access policy and click Edit
Replace the policy JSON with the following, substituting <sns_topic_arn> with the SNS topic ARN and <s3_bucket_arn> with your S3 bucket ARN

json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3Publish",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sns:Publish",
      "Resource": "<sns_topic_arn>",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "<s3_bucket_arn>"
        }
      }
    }
  ]
}

Click Save changes

Configure S3 to send ObjectCreate notifications to the SNS topic:

Go to S3 > Buckets and select your bucket
Go to Properties > Event notifications
Click Create event notification
Fill out the details:

Event name: RlmS3ToSnsEvents
Check All object create events
Destination: SNS Topic
Select rlm-s3-event-notifications-topic from the dropdown

Click Save changes

Step 2: Create SQS Queue

Follow the same steps as Step 1 in Option 1 above to create a queue named rlm-s3-event-notifications and copy its ARN.

Step 3: Create IAM Policy

Follow the same steps as Step 2 in Option 1 above to create the rlm-s3-notifications-and-read policy.

Step 4: Create IAM User

Follow the same steps as Step 3 in Option 1 above. Save the IAM user ARN, Access Key, and Secret access key.

Step 5: Update SQS Access Policy

The SQS queue policy must allow the SNS topic to publish messages to the queue and allow the IAM user to read from it.

Go to SQS > Queues
Select the rlm-s3-event-notifications queue
Go to Queue policies and click Edit
Replace the policy JSON with the following, substituting <sqs_queue_arn> with the SQS queue ARN, <sns_topic_arn> with the SNS topic ARN from Step 1, and <iam_user_arn> with the IAM user ARN from Step 4

json

{
  "Version": "2012-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "AllowSNSPublish",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "<sqs_queue_arn>",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "<sns_topic_arn>"
        }
      }
    },
    {
      "Sid": "AllowRealmUser",
      "Effect": "Allow",
      "Principal": {
        "AWS": "<iam_user_arn>"
      },
      "Action": [
        "sqs:ChangeMessageVisibility",
        "sqs:DeleteMessage",
        "sqs:ReceiveMessage"
      ],
      "Resource": "<sqs_queue_arn>"
    }
  ]
}

Click Save

Go to SNS > Topics
Select the SNS topic from Step 1
Click Create subscription
Fill out the details:

Protocol: Amazon SQS
Endpoint: <sqs_queue_arn>

Click Create subscription

The subscription will be automatically confirmed since both resources are in the same AWS account.

Step 7: Verify the S3 → SNS → SQS Pipeline

Navigate to S3 and open the bucket tied to your SNS topic.
Go to Properties → Event notifications and confirm the existing notification is pointing to your SNS topic for s3:ObjectCreated:* events.
Upload a test object to the S3 bucket.
Go back to your SQS queue and click Send and receive messages.
Click Poll for messages — you should see a message appear containing the S3 event payload.

AWS S3 as a Destination

Send logs from Realm to S3 bucket

Step 1: Create S3 Bucket

Go to S3 > Buckets
Click Create bucket
Enter the bucket name:

Bucket name: rlm-demo-output

Click Create bucket
Copy the bucket name — you will need it in the next step

Note: Versioning is disabled by default.

Step 2: Create IAM Policy

Go to IAM > Policies
Click Create policy
Click JSON and replace the contents with the following, substituting <BUCKET_NAME> with the bucket name from Step 1

json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3WriteRealmBucket",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::<BUCKET_NAME>",
                "arn:aws:s3:::<BUCKET_NAME>/*"
            ]
        }
    ]
}

Click Next
Enter the policy details:

Name: rlm-s3-output-write
Description: Grant Realm write access to the S3 output bucket.

Click Create policy

Step 3: Create IAM User

Go to IAM > Users and click Create user
Enter the username:

Name: <tenant_name>-realm.security-s3-output

Click Next
Select Attach policies directly
Search for and select the rlm-s3-output-write policy created in Step 2
Click Next, then Create user
Open the user that was just created
Go to Security credentials and click Create access key
Select Third-party service, check the confirmation checkbox, then click Next
Enter a description:

Allows <tenant_name> to write to <bucket_name> S3 bucket

Click Create access key
Copy and save the Access Key and Secret access key — you will need these when configuring the S3 output feed in Realm

Output File Structure

By default, data is stored in the following path structure:

s3://{bucket_name}/{destination_name}/{source_name}/YYYY/MM/DD/*.log.gz

To use a custom prefix, update the Key Prefix field on the output feed. For example, setting Key Prefix to foo produces:

s3://{bucket_name}/foo/*.log.gz

You can include date components in the prefix using the placeholders %Y, %m, and %d:

Key Prefix	Resulting path
`foo/year=%Y/month=%m/day=%d/`	`s3://{bucket_name}/foo/year=2025/month=10/day=23/`
`foo/%Y/%m/%d/`	`s3://{bucket_name}/foo/2025/10/23/`

Each file is compressed with zstd and contains newline-delimited events. The event format (RAW or JSON) is set by the Format field on the S3 output feed.

AWS S3 Integration ​

AWS S3 as a Source ​

Option 1: Via SQS (S3 → SQS) ​

Step 1: Create SQS Queue ​

Step 2: Create IAM Policy ​

Step 3: Create IAM User ​

Step 4: Update SQS Access Policy ​

Step 5: Configure S3 Event Notifications ​

Option 2: Via SNS → SQS ​

Step 1: Create and Configure SNS Topic ​

Step 2: Create SQS Queue ​

Step 3: Create IAM Policy ​

Step 4: Create IAM User ​

Step 5: Update SQS Access Policy ​

Step 6: Subscribe SQS Queue to SNS Topic ​

Step 7: Verify the S3 → SNS → SQS Pipeline ​

AWS S3 as a Destination ​

Send logs from Realm to S3 bucket ​

Step 1: Create S3 Bucket ​

Step 2: Create IAM Policy ​

Step 3: Create IAM User ​

Output File Structure ​

AWS S3 Integration

AWS S3 as a Source

Option 1: Via SQS (S3 → SQS)

Step 1: Create SQS Queue

Step 2: Create IAM Policy

Step 3: Create IAM User

Step 4: Update SQS Access Policy

Step 5: Configure S3 Event Notifications

Option 2: Via SNS → SQS

Step 1: Create and Configure SNS Topic

Step 2: Create SQS Queue

Step 3: Create IAM Policy

Step 4: Create IAM User

Step 5: Update SQS Access Policy

Step 6: Subscribe SQS Queue to SNS Topic

Step 7: Verify the S3 → SNS → SQS Pipeline

AWS S3 as a Destination

Send logs from Realm to S3 bucket

Step 1: Create S3 Bucket

Step 2: Create IAM Policy

Step 3: Create IAM User

Output File Structure