Realm.Security Documentation

Appearance

Elasticsearch Integration ​

Realm supports sending log data to Elasticsearch and AWS OpenSearch via the Bulk API or Data Streams. This guide covers configuring the Realm output feed.

Note: This integration supports Elasticsearch API v7 and v8 only.

Prerequisites ​

  • Administrator access to your Elasticsearch or OpenSearch instance
  • An index or data stream configured to receive log data (or auto-creation enabled)
  • Basic auth: a user with write permissions to the target index or data stream, and their username and password — see Managing native users for user creation via Kibana
  • AWS auth: an IAM role with write access to your OpenSearch domain, and the region of the domain

Elasticsearch User Permissions ​

Whether using Basic Auth through Elasticsearch or AWS Auth through the AWS OpenSearch, the Elasticsearch user must have the following minimum set of permissions:

Cluster privileges:

  • cluster:monitor/health
  • indices:data/write/bulk

Index privileges:

  • indices:admin/create

IP Allow Listing ​

If your Elasticsearch instance restricts inbound connections via an IP allow list, contact Realm Security for the IP addresses to allow.

Configure an Elasticsearch Output Feed in Realm ​

  1. Navigate to the Destinations page and select an existing destination or create a new one for Elasticsearch.

  2. Add a new Output Feed and select Elasticsearch as the output type.

  3. Enter the Endpoint URL of your Elasticsearch instance.

    Example for Cloud hosted Elasticsearch/OpenSearch:

    text
    https://<deployment-id>.<region>.es.amazonaws.com

    Example for self-hosted:

    text
    https://elasticsearch.example.com:9200

  4. OpenSearch Mode (optional) — if using AWS OpenSearch, select the appropriate mode:

    • Managed — for provisioned OpenSearch Service domains
    • Serverless — for OpenSearch Serverless collections

    Note: The API Version field does not need to be set when using OpenSearch Serverless mode.

  5. Select the API Version:

    • v7 — for Elasticsearch 7.x clusters
    • v8 — for Elasticsearch 8.x clusters

  6. Ingest Pipeline (optional) — if you have an Elasticsearch ingest pipeline to process incoming documents, enter the pipeline name in the Pipeline Name field.

  7. Select the Auth type and enter your credentials:

    • Basic — enter the username and password of your Elasticsearch user
    • AWS — enter the AWS Access Key ID and Secret Access Key of your IAM user, as well as the AWS region of your Elasticsearch/Opensearch domain

  8. Select the Mode — see the sections below for details on each option.

Bulk Mode ​

Bulk mode writes documents to named Elasticsearch indices. Configure an Index Mapping to route each Realm source to a specific index.

Each entry in the index mapping is:

  • Key: the name of a Realm Source (e.g., fortigate-prod)
  • Value: the target Elasticsearch index name (e.g., logs-fortigate)

Data Stream Mode ​

Data Stream mode writes documents to Elasticsearch data streams. Configure the following fields:

Type: logs
Dataset: <dataset name> (e.g., fortigate)

Configure a Namespace Mapping to route each Realm source to a specific data stream namespace:

  • Key: the name of a Realm Source (e.g., fortigate-prod)
  • Value: the data stream namespace (e.g., production)

Documents will be written to a data stream following the naming pattern:

text
<type>-<dataset>-<namespace>

For example: logs-fortigate-production