Arpio's service delivery is built on documented design, development, testing, delivery, and operations processes. We do annual audits to ensure we comply with these policies, and because the service is built with a modern technology stack that incorporates advanced test automation, we're able to deliver product updates and fixes quickly and with high quality. Email support@arpio.io for a copy of our latest SOC II audit.
When we need to make a change to our service, to add a new feature or fix a problem, we start by creating a work item. We use it to track the development, testing, and delivery of the change.
Before a change can be deployed into internal test environments, it must be reviewed and approved by senior development staff. All changes to production code require corresponding unit test updates. Additionally, code that adds or changes how Arpio interacts with AWS APIs require coverage in one or more of our integration test suites. Test code is updated along-side production code, tracked in the same ticket, and reviewed together.
When a change is ready for review, our continuous integration system runs the full suite of unit tests automatically, identifying regressions and defects. The system also runs tools that examine the source code for security-related errors, as well as validating our dependency chain for known vulnerabilities.
After all unit tests pass and the changes are approved, they get merged into our main development branch, and then they're automatically deployed to internal test service deployment. This Arpio service is configured to run in the same regions, and with the same capacity and configuration as the Arpio production service, although in a separate security domain.
The Arpio test service deployment continuously backs up and restores live AWS workloads across many accounts and regions, with the goal of surfacing problems due to complex service interactions after new changes are integrated. Developers and testers can use this system interactively to manually test advanced configurations, and this is often done to quickly validate customer configurations.
Each night, the code in our main development branch is validated by over 50 full-scale integration test suites. Each of these suites configures real AWS resources in 1 or 2 AWS accounts dedicated for the purpose, and then adds, removes, and mutates these resources as a customer would over time, all while validating that Arpio's backup, restore, and failback code works as expected at each step. These tests help us identify changes in AWS APIs, logic defects in our code, and (over time) subtle timing sensitivities in how we interact with AWS. Because these tests use real AWS resources, not mocks, they can run for tens of minutes or several hours. Because they're isolated from each other, we can run them all concurrently, which lets us validate cross-cutting changes to the product quickly.
Importantly, we can invoke these integration test suites manually for code in feature branches before they're merged into the main development branch. This lets us test product fixes and their related integration test changes quickly, often while the code is being reviewed.
After changes are reviewed and approved, and its work item updated with its current state, we can prepare a release to the production environment. Production releases require at least 2 code reviews by senior development staff over the set of all changes included in the release. Then a member of management triggers the production environment deployment (which takes about 30 minutes), and the team monitors for changes in system behavior post-release.
Some code releases require customers to update CloudFormation templates to adopt the latest changes. See instructions here.