Executive:
Amazon OpenSearch Service provides robust mechanisms for snapshot backups, which are essential for disaster recovery, migrations, and compliance. While automated snapshots are handled internally by AWS, there are use cases where manual snapshots become necessary to meet custom requirements, especially during cross-cloud migrations or external recovery strategies.
The Challenge:
One of our clients faced a limitation with OpenSearch’s default automated snapshot feature.
Scenario:
The client was preparing to migrate from AWS to Azure, and while OpenSearch automated snapshots were available, they did not meet the granularity or portability required for such a transition.
Problem:
Automated snapshots are stored in an AWS-managed repository. Direct access or export of these snapshots outside AWS is not supported. The client needed full control over the snapshots, including the ability to store and manage them in a customer-owned S3 bucket.
The Solution:
To overcome this limitation, Cogniv implemented a manual snapshot strategy using OpenSearch’s snapshot APIs, with the snapshots stored in a custom-managed Amazon S3 bucket.
- Full ownership and visibility of backup data
- Greater flexibility for cross-cloud migration
- The ability to script, schedule, and manage snapshots as needed
Prerequisites:
-
An S3 bucket must be created in the same AWS region where the OpenSearch domain is provisioned.
-
An EC2 instance should be running in the same VPC as the OpenSearch domain.
-
The EC2 instance must have an IAM role attached with permissions to access S3 and OpenSearch.
-
Access to the Dev Tools section in the OpenSearch Dashboard is required to register and trigger snapshots.
Implementation of manual snapshot:
Step 1 : Creating a required IAM Role
Create an IAM role with below two policies
Policy Name: PutToOpenSearchAndPassRolePolicy & S3FullAccessToSpecificBucketPolicy
Policy 1 – PutToOpenSearchAndPassRolePolicy
Description:
This policy allows a principal to upload data to an Amazon OpenSearch Service domain using HTTP PUT and to pass a designated IAM role. This is typically used in scenarios where a service needs to write logs or data to OpenSearch and assumes a role for execution.
This policy is for interacting with an Amazon OpenSearch domain and passing an IAM role:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: “iam:PassRole”,
“Resource”: “arn:aws:iam::<account-id>:role/<role-name>”
},
{
“Effect”: “Allow”,
“Action”: “es:ESHttpPut”,
“Resource”: [
“arn:aws:es:<region>:<account-id>:domain/<domain-name>/*”
]
}
]
}
Policy-2 : S3FullAccessToSpecificBucketPolicy
Description:
Grants permission to list, read, upload, and delete objects in a specific Amazon S3 bucket. This is commonly used for services or applications that manage file storage within a single bucket.
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“s3:ListBucket”
],
“Resource”: [
“arn:aws:s3:::<bucket-name>”
]
},
{
“Effect”: “Allow”,
“Action”: [
“s3:GetObject”,
“s3:PutObject”,
“s3:DeleteObject”
],
“Resource”: [
“arn:aws:s3:::<bucket-name>/*”
]
}
]
}
Step 2 : Update the trust policy of the created IAM role
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Sid”: “”,
“Effect”: “Allow”,
“Principal”: {
“Service”: [
“es.amazonaws.com”,
“ec2.amazonaws.com”
]
},
“Action”: “sts:AssumeRole”
}
]
}
Step 3 : Mapping IAM Role ARN in OpenSearch Dashboard
To allow OpenSearch to use the IAM role for snapshot operations, map the created IAM Role ARN to the relevant security roles within the OpenSearch Dashboard.
Navigation Path: Menu → Management → Security → Roles
Instructions:
Map to manage_snapshots Role:
- In the Roles view, search for the role named manage_snapshots[predefined role]
2. Select the role and locate the section titled Mapped users
3. Click Edit and add the ARN of the IAM role created for snapshot access.
Map to all_access Role:
- Similarly, search for the role named all_access.
- Select the role and locate both the Mapped users and Backend roles sections.
- Click Edit for each section and add the same IAM Role ARN.
- Save the changes.
This mapping ensures that the IAM role has the necessary permissions to interact with OpenSearch and manage snapshot operations via the REST API or CLI.
Step 4 : Create a s3 bucket on which the manual snapshot should be stored and keep the bucket as public
Step 5 : Register OpenSearch Snapshot Repository via Python
This guide describes how to install the required dependencies and execute a Python script to register a snapshot repository in the OpenSearch Dashboard via its REST API.
Installation Commands :
Run the following commands on your EC2 or server to set up the environment:
sudo yum install -y python3
sudo yum install -y python3-pip
pip3 install boto3
pip3 install requests
pip3 install requests_aws4auth
Script :
This script uses the boto3, requests, and requests_aws4auth libraries to register an S3 snapshot repository in Amazon OpenSearch Service.
import boto3
import requests
from requests_aws4auth import AWS4Auth
host = ‘https://<your-opensearch-domain-endpoint>/‘
region = ‘<aws-region>’
service = ‘es’
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(
credentials.access_key,
credentials.secret_key,
region,
service,
session_token=credentials.token
)
path = ‘/_snapshot/<repository-name>’
url = host + path
payload = {
“type”: “s3”,
“settings”: {
“bucket”: “<your-s3-bucket-name>”,
“region”: “<aws-region>”,
“role_arn”: “arn:aws:iam::<account-id>: role/<role-name>”
}
}
headers = {“Content-Type”: “application/json”}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)
Run this script to register OpenSearch snapshot Repository
Step 6 : Execute a Manual Snapshot Backup
To initiate a manual snapshot backup of your OpenSearch indices, use the following REST API command from the OpenSearch Dashboard (Dev Tools )
Command Syntax:
PUT /_snapshot/<your-snapshot-repo-name>/<snapshot-name>
To check the status, use the following command and run it in the dev tools
Command Syntax:
GET /_snapshot/_status
After running these command manual snapshot will be created and stored in the created s3 bucket
