Convey the relationships between your in-cluster resources and other Arpio-supported AWS resources
Overview
By default, Arpio will dynamically inventory and backup the resources within your Kubernetes clusters. Upon restoration, Arpio also attempts to translate these resources and their dependencies accurately. However, due to the dynamic nature of Kubernetes, Arpio sometimes needs some additional direction to accurately interpret the resource identifiers in your cluster’s configuration. For this reason, Arpio allows customers to create custom ConfigMaps to add directly to their clusters, which enable Arpio to translate resources more precisely.
Background
When inventorying resources in environments, Arpio relies on unique identifiers attached to resources to correctly identify a source resource type and its dependencies and to recreate them faithfully in the recovery environment.
For many identifiers, such as VPC IDs, Subnet IDs, Security Group IDs, etc., the id numbers are regionally unique and thus are easier to find and replace as necessary in order to rebuild the infrastructure elsewhere.
On the other hand, some identifiers, such as IAM Role Names, are not unique to a region within an account. Sometimes these kinds of identifiers are reused with nothing to identify the resource type in the configuration. This can lead Arpio to misinterpret the resource type upon cluster restore.
Because of the potential for overlapping names across resource types, we’ve implemented this hinting system to allow cluster configuration to explicitly convey to Arpio resource types. This allows us to find the relationships between your in-cluster resources and other Arpio-supported AWS resources.
Creating a Custom ConfigMap
For all Amazon EKS clusters, Arpio applies a default set of rules to translate your cluster resources. Those rules are published at https://api.arpio.io/api/info/eks/baseTranslationConfig.json | https://api.arpio.io/api/info/eks/baseTranslationConfig.yaml.
In addition to Arpio’s default rules, customers can add their own customized translation rules by creating their own ConfigMaps. To do this, you’ll follow the format described below to create a new ConfigMap with custom resource definitions.
Once complete, a customized Arpio ConfigMap can be added to the source cluster. It should be named arpio-resource-map.
Each map is placed in a yaml file in the configMap’s data object under the key override.
The map installed into the kube-system namespace is considered the default addition/override set for your entire cluster. Any maps installed into other namespaces will only affect that namespace. This allows delegation of responsibilities to those in control of that particular namespace.
Creating Resource Definitions
Each resource definition in your ConfigMap has 3 major pieces:
- The name
- A source object
- A list of target objects.
For example:
data:
override: |
containerEnvVars:
source:
k8sTypes:
- CronJob
- DaemonSet
- Deployment
- Job
- Pod
- StatefulSet
paths:
- ..spec.containers,initContainers,ephemeralContainers[*].env[*].value
targets:
- awsType: AWS::EC2::VPC
field: VpcId
- field: Id
type: AWS::Region
- awsType: AWS::ECR::Repository
field: repositoryUri
matcher: substring
- awsType: AWS::IAM::Role
field: _arn_
- awsType: '*'
field: _arn_
This definition - named containerEnvVars - picks up and translates: VPC Ids, Region names, Amazon ECR Repository substrings, IAM Role ARNs, and any other full ARN in the value entry of environment maps for container, initContainer, and ephemeralContainer specs and template specs in CronJob, DaemonSet, Deployment, Job, Pod, and StatefulSet kinds. The template specs are implicit here by the jsonpath syntax in source.paths.
Most definitions will only have a single target as you’ll want to limit the ability for it to be picked up as the wrong thing. One might think the AWS::Region match would also catch a part of the AWS::ECR::Repository.repositoryUri, but each of these matches is required to be the full string of that leaf node in the data structure, unless the matcher is set to something like substring or csv.
data:
override: |
overrideSourceConfig:
source:
k8sTypes:
- ConfigMap
fileType: YAML
filePath: data."example.yaml"
paths:
- $.array[*].exampleName.exampleVarKey
targets:
- type: replace
data: exampleVarValue2
This definition - named overrideSourceConfig - picks up and translates a yaml file contained inside of a ConfigMap and allows you to replace a target variable inside the yaml. By defining the fileType YAML and filePath data."example.yaml"under the k8sTypes ConfigMap, Arpio will know to examine all YAML extension files inside ConfigMaps present and apply the targets to the the named file. Then providing the path inside the targetted YAML file (base level defined by $), we can target an array of variables (array[*]) and specific entries underneath it.
Using a target of type replace and providing the data value of exampleVarValue2, we can replace the value of exampleVarKey with exampleVarValue2.
data:
override: |
stripS3LoggerFromIngress:
source:
k8sTypes:
- Ingress
paths:
- metadata.annotations."alb.ingress.kubernetes.io/load-balancer-attributes"
targets:
- matcher: csv
type: static
valueMap:†
access_logs.s3.bucket=adjudication-service-dev-apigw-alb-access-logs: null
idle_timeout.timeout_seconds=135: null
access_logs.s3.enabled=true: null
access_logs.s3.prefix=logs : null
access_logs.s3.bucket=adjudication-service-dev-alb-access-logs: null
This definition - named stripS3LoggerFromIngress - strips the attributes from an ingress that would define a named s3 bucket to hold logs from an ALB.
Doing this allows you to remove references to storage that you wouldn't want to copy over to your recovery account, and prevents your recovery kubernetes ALB from creating unnecessary log storage buckets and logging.
By defining the exact path to the load balancer attributes, and matching the static valueMap values via a CSV matcher, you can set lines to null to effectively remove them from the attribute mapping.
Definition Names
Defining a map of the same name as one of Arpio’s defaults will override that resource map at the top level. This means that you will need to reiterate every part of the default resource map you want to retain in your ConfigMap, and leave out just the portion you do not wish to use. See the full example below for a reference.
Source Objects
The source object has 4 keys: k8sTypes, paths, filePath, and fileType. Basic entries will just require the first two. The file* keys are optional, but if one is specified, the other is required.
k8sTypes: list of k8s kinds to match this definition against
paths: A list of jsonpath-like paths to locate the targets in. See Resources.
filePath: jsonpath location of a file inside this resource
fileType: one of YAML or YAML.b64 with the latter being a base64 encoded YAML file
Target Objects
The target objects have N keys: awsType, type, field, and matcher. Only one of awsType or type is allowed and matcher is optional - defaulting to a full string match.
awsType: These will mostly align with the Resource Type Value in this page. Arpio maintains a list of supported awsConfigTypes in our API.
type: The type field may be used instead of awsType when either static translation is needed, or there is not a corresponding AWS resource type that is referenced.
static- Used for statically replacing values from a given value mapping that is supplied via thevalueMapfield. If the existing value does not exist within the supplied value map, it is left as-is.replace- Used for statically replacing the value from the primary environment with another fixed value that is supplied via thedatafield.AWS::Region- Replaces occurrences of the primary environment AWS region (e.g.us-east-1) with the corresponding recovery environment region value.AWS::AvailabilityZone- Replaces occurrences of the primary environment AWS availability zone (e.g.us-east-1a) with the corresponding recovery AZ value.AWS::Account- Replaces occurrences of the primary environment AWS account ID with the corresponding recovery account ID.AWS::ECR::HostedRepository- Replaces occurrences of AWS-managed ECR repositories in the primary environment with corresponding repository values in the recovery environment. https://docs.aws.amazon.com/eks/latest/userguide/add-ons-images.html
field: If type is used, this should be Id, otherwise, it’s a jsonpath inside the AWS/boto description of that particular resource or the special value _arn_ which is the full ARN of the resource. For example, if you want to translate a security group id, the groupId field name would be the field value.
matcher: csv or substring. Some fields will contain a list of comma separated ARNs, subnet IDs, or security group IDs. In those cases, using csv will allow Arpio to split it up on those commas and rebuild it back into a csv. Substring matches are useful for finding things like AWS::Regions inside other strings, or more commonly, ECR repository URIs as only part of the image definitions in workload specs.
A full working ConfigMap example:
kind: ConfigMap
apiVersion: v1
metadata:
name: arpio-resource-map
namespace: kube-system
data:
override: |
argoRolloutEcr:
source:
k8sTypes:
- Rollout
paths:
- $.spec.template.spec.containers[*].image
targets:
- awsType: AWS::ECR::Repository
field: repositoryUri
matcher: substring
karpenterNodeClassAcct:
source:
k8sTypes:
- Ec2NodeClass-disabled
paths:
- spec.amiSelectorTerms[*].owner-disabled
targets:
- field: Id
type: AWS::Account
This example has two definitions in it. The first one translates the image identifiers for ArgoCD Rollouts, and the second overwrites the default karpenterNodeClassAcct translation but makes it such that it won’t match either the Kind or the path in the Ec2NodeClass resources.
Resources
Jsonpath documentation: https://goessner.net/articles/JsonPath/
JsonPath Evaluator: https://jsonpath.com/