Realm.Security Documentation

Appearance

Elasticsearch Integration ​

Realm supports sending log data to Elasticsearch and AWS OpenSearch via the Bulk API or Data Streams. This guide covers configuring the Realm output feed.

Note: This integration supports Elasticsearch API v7 and v8 only.

Prerequisites ​

  • Administrator access to your Elasticsearch or OpenSearch instance
  • An index or data stream configured to receive log data (or auto-creation enabled)
  • Basic auth: a user with write permissions to the target index or data stream, and their username and password — see Managing native users for user creation via Kibana
  • AWS auth: an IAM role with write access to your OpenSearch domain, and the region of the domain

Elasticsearch User Permissions ​

Whether using Basic Auth through Elasticsearch or AWS Auth through the AWS OpenSearch, the Elasticsearch user must have the following minimum set of permissions:

Cluster privileges:

  • cluster:monitor/health
  • indices:data/write/bulk

Index privileges:

  • indices:admin/create

IP Allow Listing ​

If your Elasticsearch instance restricts inbound connections via an IP allow list, contact Realm Security for the IP addresses to allow.

Configure an Elasticsearch Output Feed in Realm ​

  1. Navigate to the Destinations page and select an existing destination or create a new one for Elasticsearch.

  2. Add a new Output Feed and select Elasticsearch as the output type.

  3. Enter the Endpoint URL of your Elasticsearch instance.

    Example for Cloud hosted Elasticsearch/OpenSearch:

    text
    https://<deployment-id>.<region>.es.amazonaws.com

    Example for self-hosted:

    text
    https://elasticsearch.example.com:9200

  4. OpenSearch Mode (optional) — if using AWS OpenSearch, select the appropriate mode:

    • Managed — for provisioned OpenSearch Service domains
    • Serverless — for OpenSearch Serverless collections

    Note: The API Version field does not need to be set when using OpenSearch Serverless mode.

  5. Select the API Version:

    • v7 — for Elasticsearch 7.x clusters
    • v8 — for Elasticsearch 8.x clusters

  6. Ingest Pipeline (optional) — if you have an Elasticsearch ingest pipeline to process incoming documents, enter the pipeline name in the Pipeline Name field.

  7. Select the Auth type and enter your credentials:

    • Basic — enter the username and password of your Elasticsearch user
    • AWS — enter the AWS Access Key ID and Secret Access Key of your IAM user, as well as the AWS region of your Elasticsearch/Opensearch domain

  8. Select the Mode — see the sections below for details on each option.

Bulk Mode ​

Bulk mode writes documents to named Elasticsearch indices. Configure an Index Mapping to route each Realm source to a specific index.

Each entry in the index mapping is:

  • Key: the name of a Realm Source (e.g., fortigate-prod)
  • Value: the target Elasticsearch index name (e.g., logs-fortigate)

Data Stream Mode ​

Data Stream mode writes documents to Elasticsearch data streams. Configure the following fields:

Type: logs
Dataset: <dataset name> (e.g., fortigate)

Configure a Namespace Mapping to route each Realm source to a specific data stream namespace:

  • Key: the name of a Realm Source (e.g., fortigate-prod)
  • Value: the data stream namespace (e.g., production)

Documents will be written to a data stream following the naming pattern:

text
<type>-<dataset>-<namespace>

For example: logs-fortigate-production

Export Detections for Detection Integrity ​

Detection Integrity checks your log optimization rules against your Elasticsearch detections, so optimizations never drop data a detection depends on. Export your detections as described below, then upload them to Realm.

Elastic Security ​

Export your Elastic Security detection rules with the Kibana API. The export keeps the rules your team created and the Elastic prebuilt rules your team customized. It leaves out the unmodified prebuilt rules Elastic publishes.

Note: This export requires Elastic Security 8.18 or later.

  1. Use a Kibana user that can read detection rules in the space you export from.
  2. Run the script for your platform below. First set these variables:
    • KIBANA_URL: your Kibana endpoint. In Elastic Cloud, this is the Kibana endpoint on your deployment page.
    • KIBANA_SPACE: the ID of the Kibana space that holds your rules. It appears in Kibana URLs as /s/<space-id>/. Leave it empty for the default space.
    • ES_USER: the Kibana username.

The script exports your custom rules and any pre-built rules you have modified. If you see Unexpected response, nothing was exported. Check the URL, space, and credentials, then run the commands again.

If your rules are in more than one space, run the commands once per space and rename the output file between runs.

macOS and Linux ​

Requires curl and jq.

shell
KIBANA_URL="https://kibana.example.com:5601"
KIBANA_SPACE=""
ES_USER="elastic"
printf 'Elastic password: '; read -rs ES_PASS; echo

out="elastic_detection_rules.jsonl"
auth=$(printf '%s:%s' "$ES_USER" "$ES_PASS" | base64 | tr -d '\n')

resp=$(curl -s -X POST \
  "$KIBANA_URL${KIBANA_SPACE:+/s/$KIBANA_SPACE}/api/detection_engine/rules/_export?exclude_export_details=true" \
  -H @<(printf 'Authorization: Basic %s\n' "$auth") \
  -H "kbn-xsrf: true" -H "Content-Type: application/json")

unset ES_PASS auth

if printf '%s\n' "$resp" | jq -e -s 'length > 0 and (.[0] | has("statusCode") | not)' >/dev/null 2>&1; then
  printf '%s\n' "$resp" | jq -c 'select(.rule_source.type == "internal" or (.rule_source.type == "external" and .rule_source.is_customized == true))' > "$out"
  echo "$(wc -l < "$out" | tr -d ' ') detection rules written to $out"
else
  echo "Unexpected response: $resp" >&2
fi

Windows ​

Requires PowerShell 5.1 or later.

powershell
$KibanaUrl = "https://kibana.example.com:5601"
$KibanaSpace = ""
$EsUser = "elastic"
$SecurePass = Read-Host "Elastic password" -AsSecureString
$EsPass = [Net.NetworkCredential]::new("", $SecurePass).Password

[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
$ProgressPreference = "SilentlyContinue"

$Out = Join-Path (Get-Location) "elastic_detection_rules.jsonl"
$SpacePath = if ($KibanaSpace) { "/s/$KibanaSpace" } else { "" }
$Uri = "$KibanaUrl$SpacePath/api/detection_engine/rules/_export?exclude_export_details=true"
$Auth = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes("${EsUser}:$EsPass"))
$Headers = @{ "Authorization" = "Basic $Auth"; "kbn-xsrf" = "true" }

try {
  $Body = (Invoke-WebRequest -Uri $Uri -Method Post -Headers $Headers -ContentType "application/json" -UseBasicParsing).Content
  if ($Body -is [byte[]]) { $Body = [Text.Encoding]::UTF8.GetString($Body) }
} catch {
  $Body = if ($_.ErrorDetails.Message) { $_.ErrorDetails.Message } else { $_.Exception.Message }
}
Remove-Variable EsPass, SecurePass, Auth, Headers

$Lines = [Collections.Generic.List[string]]::new()
$Valid = $false
foreach ($Line in ($Body -split "\r?\n")) {
  if (-not $Line.Trim()) { continue }
  try { $Rule = $Line | ConvertFrom-Json } catch { $Valid = $false; break }
  if ($Rule.PSObject.Properties.Name -contains "statusCode") { $Valid = $false; break }
  $Valid = $true
  $Source = $Rule.rule_source
  if ($Source.type -eq "internal" -or ($Source.type -eq "external" -and $Source.is_customized -eq $true)) {
    $Lines.Add($Line.Trim())
  }
}

if ($Valid) {
  [IO.File]::WriteAllLines($Out, $Lines)
  Write-Host "$($Lines.Count) detection rules written to $Out"
} else {
  Write-Host "Unexpected response: $Body"
}

OpenSearch ​

Export your custom Security Analytics detection rules with the OpenSearch API.

  1. Use an OpenSearch user that can read Security Analytics rules, for example a user mapped to the security_analytics_read_access role. On Amazon OpenSearch Service, the domain must use fine-grained access control so the user can sign in with a username and password.
  2. Run the script for your platform below. First set these variables:
    • OS_URL: your OpenSearch domain endpoint.
    • OS_USER: the OpenSearch username.

The script fetches your custom rules 1,000 at a time and prints a progress line for each page. The API returns at most 10,000 rules, so if you have more, the script prints a warning and the export is incomplete. If you see Unexpected response, the export is incomplete. Check the URL and credentials, then run the commands again.

macOS and Linux ​

Requires curl and jq.

shell
OS_URL="https://search-example.us-east-1.es.amazonaws.com"
OS_USER=""
printf 'OpenSearch password: '; read -rs OS_PASS; echo

out="opensearch_detection_rules.jsonl"
auth=$(printf '%s:%s' "$OS_USER" "$OS_PASS" | base64 | tr -d '\n')
unset OS_PASS
: > "$out"

from=0
page=0

while [ "$from" -lt 10000 ]; do
  resp=$(curl -s -X POST "$OS_URL/_plugins/_security_analytics/rules/_search?pre_packaged=false" \
    -H @<(printf 'Authorization: Basic %s\n' "$auth") \
    -H "Content-Type: application/json" \
    -d "{\"query\":{\"match_all\":{}},\"from\":$from,\"size\":1000}")

  page=$((page + 1))

  if ! printf '%s' "$resp" | jq -e '.hits.hits' >/dev/null 2>&1; then
    echo "Unexpected response on page $page: $resp" >&2
    break
  fi

  count=$(printf '%s' "$resp" | jq '.hits.hits | length')
  printf '%s' "$resp" | jq -c '.hits.hits[]._source' >> "$out"
  echo "page $page fetched ($count rules)" >&2

  [ "$count" -lt 1000 ] && break
  from=$((from + 1000))
done

[ "$from" -ge 10000 ] && echo "Stopped at 10,000 rules. Some rules may be missing." >&2

unset auth
echo "$(wc -l < "$out" | tr -d ' ') detection rules written to $out"

Windows ​

Requires PowerShell 5.1 or later.

powershell
$OsUrl = "https://search-example.us-east-1.es.amazonaws.com"
$OsUser = "rule-export-svc"
$SecurePass = Read-Host "OpenSearch password" -AsSecureString
$OsPass = [Net.NetworkCredential]::new("", $SecurePass).Password

[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12
$ProgressPreference = "SilentlyContinue"

$Out = Join-Path (Get-Location) "opensearch_detection_rules.jsonl"
$Uri = "$OsUrl/_plugins/_security_analytics/rules/_search?pre_packaged=false"
$Auth = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes("${OsUser}:$OsPass"))
$Headers = @{ "Authorization" = "Basic $Auth" }
Remove-Variable OsPass, SecurePass, Auth

$Lines = [Collections.Generic.List[string]]::new()
$From = 0
$Page = 0

while ($From -lt 10000) {
  $Page++
  $Query = @{ query = @{ match_all = @{} }; from = $From; size = 1000 } | ConvertTo-Json -Depth 5 -Compress

  try {
    $Body = (Invoke-WebRequest -Uri $Uri -Method Post -Headers $Headers -ContentType "application/json" -Body $Query -UseBasicParsing).Content
  } catch {
    $Body = if ($_.ErrorDetails.Message) { $_.ErrorDetails.Message } else { $_.Exception.Message }
  }

  $Resp = $null
  try { $Resp = $Body | ConvertFrom-Json } catch { }
  if (-not ($Resp -and $Resp.PSObject.Properties.Name -contains "hits")) {
    Write-Host "Unexpected response on page ${Page}: $Body"
    break
  }

  $Hits = @($Resp.hits.hits)
  foreach ($Hit in $Hits) { $Lines.Add(($Hit._source | ConvertTo-Json -Depth 20 -Compress)) }
  Write-Host "page $Page fetched ($($Hits.Count) rules)"

  if ($Hits.Count -lt 1000) { break }
  $From += 1000
}

if ($From -ge 10000) { Write-Host "Stopped at 10,000 rules. Some rules may be missing." }

[IO.File]::WriteAllLines($Out, $Lines)
Remove-Variable Headers
Write-Host "$($Lines.Count) detection rules written to $Out"

Upload the Export to Realm ​

Upload the exported file as described in Upload Detections to Realm, selecting your Elasticsearch destination.