S3 Resource Replication with Arpio

Arpio replicates the following Amazon Simple Storage Service resource types.

S3 Bucket

Arpio automates the replication of S3 bucket contents and configuration to an alternate bucket in the recovery region and recovery account. This process leverages S3 bucket replication and scales to support massive buckets containing hundreds of millions of objects.

Arpio automates the creation of a new bucket in the recovery environment that becomes the destination of the object replication process. Because S3 bucket names must be globally unique, the recovery bucket will have a different name than the primary bucket. Arpio will name this bucket by appending a random suffix to the name of the primary bucket. Your application may need to become aware of this new bucket name, and be updated to reference objects from this location when running in the recovery environment.

Arpio automates the security configuration of the recovery environment bucket to enable the replication process to copy objects into the bucket. Arpio creates an IAM role in the production environment that S3 bucket replication will use to read objects from the primary bucket and write them to the recovery bucket.

Once the security configuration is in place, Arpio will configure the bucket replication settings on the primary bucket to begin the replication process. If you manage your primary bucket with an infrastructure-as-code solution (such as Terraform or CloudFormation), you may want to configure these settings yourself with your existing automation. In that case, Arpio can generate the Terraform or CloudFormation configuration for you that you can easily add to your existing solution.

After bucket replication has been configured, Arpio will backfill the recovery bucket with objects from the primary bucket. The backfill process utilizes S3 Batch Operations to copy objects from the primary bucket to the recovery bucket. If you do not need all objects copied to the recovery bucket, you can specify a timeframe of objects to include based on object age (excluding older objects).

The replica bucket is only accessible to principles in the recovery account when not under test or not failed over. During the test and failover process, the bucket ACL and bucket policy are updated to match the ACL and policy of the primary bucket at which point other principles may be granted access.

The following attributes are translated during replication:

Attribute Translation
Lifecycle Rules Lifecycle rules are copied from primary bucket. An additional lifecycle rule is added for deleted objects that are older than the configured recovery point retention policy so that objects with delete markers that should no longer be retained will be fully deleted.
Notification Configurations When testing the recovery environment or actively failed over, Arpio translates notification configurations that reference SNS topics, SQS queues, and Lambda functions, function versions, and aliases to target the corresponding topics and queues in the recovery environment. These configurations are not enabled when the recovery environment is not in use.
Encryption Configuration Arpio replicates the default encryption configuration from the primary bucket. If KMS encryption is being utilized with the primary bucket, Arpio creates a new customer managed KMS key in the recovery environment and configures S3 default encryption to utilize it.
Bucket ACL Arpio replicates the bucket ACL from the primary bucket to the recovery bucket. When the recovery environment is not in use, Arpio configures the ACL to grant access only to the recovery environment account. During a failover or a failover test, Arpio reconfigures the ACL to clone the primary bucket's ACL, with account numbers translated as appropriate.
Bucket Policy Arpio replicates the bucket policy from the primary bucket to the recovery bucket, and adds additional policy statements to support the bucket replication process. When the recovery environment is not in use, only the bucket replication statements are included. During a test or a failover, the primary bucket statements are included and translated according to the policy document translation process.

The following resources are automatically selected into recovery points when an S3 bucket is selected:

  • SNS Topics referenced by notification configurations on the primary environment bucket
  • SQS Queues referenced by notification configurations on the primary environment bucket
  • Lambda Functions, Versions, and Aliases referenced by notification configurations on the primary environment bucket.