Convey the relationships between your in-cluster resources and other Arpio-supported AWS resources
Overview
By default, Arpio will dynamically inventory and backup the resources within your Kubernetes clusters. Upon restore, Arpio also attempts to translate these resources and their dependencies accurately. However, due to the dynamic nature of Kubernetes, Arpio sometimes needs some additional direction to accurately interpret the resource identifiers in your cluster’s configuration. For this reason, Arpio allows customers to create custom ConfigMaps to add directly to their clusters, which enable Arpio to translate resources more precisely.
Background
When inventorying resources in environments, Arpio relies on unique identifiers attached to resources to correctly identify a source resource type and its dependencies and to recreate them faithfully in the recovery environment.
For many identifiers, such as VPC IDs, Subnet IDs, Security Group IDs, etc., the id numbers are regionally unique and thus are easier to find and replace as necessary in order to rebuild the infrastructure elsewhere.
On the other hand, some identifiers, such as IAM Role Names, are not unique to a region within an account. Sometimes these kinds of identifiers are reused with nothing to identify the resource type in the configuration. This can lead Arpio to misinterpret the resource type upon cluster restore.
Because of the potential for overlapping names across resource types, we’ve implemented this hinting system to allow cluster configuration to explicitly convey to Arpio resource types. This allows us to find the relationships between your in-cluster resources and other, Arpio supported, AWS resources.
Creating a Custom ConfigMap
For all EKS clusters, Arpio applies a default set of rules to translate your cluster resources. Those rules are published at https://api.arpio.io/api/info/eks/baseTranslationConfig.json | https://api.arpio.io/api/info/eks/baseTranslationConfig.yaml.
In addition to Arpio’s default rules, customers can add their own customized translation rules by creating their own ConfigMaps. To do this, you’ll follow the format described below to create a new ConfigMap with custom resource definitions.
Once complete, a customized Arpio ConfigMap can be added to the source cluster. It should be named arpio-resource-map.
Each map is placed in a yaml file in the configMap’s data object under the key override.
The map installed into the kube-system namespace is considered the default addition/override set for your entire cluster. Any maps installed into other namespaces will only affect that namespace. This allows delegation of responsibilities to those in control of that particular namespace.
Creating Resource Definitions
Each resource definition in your ConfigMap has 3 major pieces:
- The name
- A source object
- A list of target objects.
For example:
containerEnvVars:
source:
k8sTypes:
- CronJob
- DaemonSet
- Deployment
- Job
- Pod
- StatefulSet
paths:
-
$..spec.containers,initContainers,ephemeralContainers[*].env[*].value
targets:
- awsType: AWS::EC2::VPC
field: VpcId
- field: Id
type: AWS::Region
- awsType: AWS::ECR::Repository
field: repositoryUri
matcher: substring
- awsType: AWS::IAM::Role
field: _arn_
- awsType: '*'
field: _arn_
This definition - named containerEnvVars - picks up and translates: VPC Ids, Region names, ECR Repository substrings, IAM Role ARNs, and any other full ARN in the value entry of environment maps for container, initContainer, and ephemeralContainer specs and template specs in CronJob, DaemonSet, Deployment, Job, Pod, and StatefulSet kinds. The template specs are implicit here by the jsonpath syntax in source.paths.
Most definitions will only have a single target as you’ll want to limit the ability for it to be picked up as the wrong thing. One might think the AWS::Region match would also catch a part of the AWS::ECR::Repository.repositoryUri, but each of these matches is required to be the full string of that leaf node in the data structure, unless the matcher is set to something like substring or csv.
Definition Names
Defining a map of the same name as one of Arpio’s defaults will override that resource map at the top level. This means that you will need to reiterate every part of the default resource map you want to retain in your ConfigMap, and leave out just the portion you do not wish to use. See the full example below for a reference.
Source Objects
The source object has 4 keys: k8sTypes, paths, filePath, and fileType. Basic entries will just require the first two. The file* keys are optional, but if one is specified, the other is required.
k8sTypes: list of k8s kinds to match this definition against
paths: list of jsonpath-like paths to locate the targets in
filePath: jsonpath location of a file inside this resource
fileType: one of YAML or YAML.b64 with the latter being a base64 encoded YAML file
Target Objects
The target objects have N keys: awsType, type, field, and matcher. Only one of awsType or type is allowed and matcher is optional - defaulting to a full string match.
awsType: These will mostly align with the Resource Type Value in this page. Arpio maintains a list of supported awsConfigTypes in our API.
type: Some base-level AWS identifiers don’t have associated AWS::Config designations. For those, we use similarly formatted entries that are forced outside the namespace. AWS::Region, AWS::ECR::HostedRepository, AWS::AvailabilityZone, and AWS::Account are the currently supported types here.
field: If type is used, this should be Id, otherwise, it’s a jsonpath inside the AWS/boto description of that particular resource or the special value _arn_ which is the full ARN of the resource. For example, if you want to translate a security group id, the groupId field name would be the field value.
matcher: csv or substring. Some fields will contain a list of comma separated ARNs, subnet IDs, or security group IDs. In those cases, using csv will allow Arpio to split it up on those commas and rebuild it back into a csv. Substring matches are useful for finding things like AWS::Regions inside other strings, or more commonly, ECR repository URIs as only part of the image definitions in workload specs.
Custom Type Definitions:
AWS::Account 12 digit AWS Account id numbers
AWS::AvailabilityZone The lowercase availability zone names - useful for EBS volume attachments, etc.
AWS::Region The lowercase instances of the region IDs.
AWS::ECR::HostedRepository AWS hosts some ECR Repos across regions, but they end up being in different accounts, and obviously with different region IDs embedded in them. This allows Arpio to translate those hosted ECR repositories into the one local to your target region.
A full working ConfigMap example:
kind: ConfigMap
apiVersion: v1
metadata:
name: arpio-resource-map
namespace: kube-system
data:
override: |
argoRolloutEcr:
source:
k8sTypes:
- Rollout
paths:
- $.spec.template.spec.containers[*].image
targets:
- awsType: AWS::ECR::Repository
field: repositoryUri
matcher: substring
karpenterNodeClassAcct:
source:
k8sTypes:
- Ec2NodeClass-disabled
paths:
- spec.amiSelectorTerms[*].owner-disabled
targets:
- field: Id
type: AWS::Account
This example has two definitions in it. The first one translates the image identifiers for Argo CD Rollouts, and the second overwrites the default karpenterNodeClassAcct translation but makes it such that it won’t match either the Kind or the path in the Ec2NodeClass resources.