How we built a custom CI/CD system using GitHub Actions and AWS CodeBuild

The engineering team at Simply Business deploy our many apps to production over 300 times a week. Until recently, we've used Jenkins as the main technology for our continuous integration and continuous deployment (CI/CD) needs. However, we faced some issues with Jenkins, so our DevOps engineers set about finding a better way to support our tech organisation when it came to the reliability, performance and usability of our CI/CD system. With that in mind, we've developed and deployed a home-grown solution to handle our CI/CD needs, which we refer to as Huxley.

Continuous deployment tooling

Our DevOps engineers have been working for many years on building our in-house continuous deployment platform. We've developed a collection of tools and Infrastructure-as-Code (IaC) scripts that provide our engineers with a toolbox for easily scaffolding and deploying applications on AWS infrastructure. A core part of the toolbox is a Docker image containing scripts and the logic to execute those scripts. The Docker image runs through a custom command-line tool called bnw_runner, which enables our engineers to deploy applications in any environment, whether locally or on a build agent. The criticality of our deployment toolbox will be discussed later on.

Challenges with Jenkins

While Jenkins is well-known as the chief deployment tool for many companies, from start-ups to enterprises, it has a number of shortcomings that are acknowledged across the industry and which for our DevOps operations are becoming more prevalent.

Many of our engineers found the Jenkins user interface to be somewhat unintuitive, and when issues diverged from the happy path of deployment, they had issues debugging the actual root cause. Jenkins was also not nearly as stable as many would expect. Engineers would often experience connectivity issues between the Jenkins built-in node and Jenkins agents. In addition, we would run into issues where the Jenkins agents would run out of disk space and were unable to free up any space. All builds would fail until agents were manually terminated and new agents were spun up. This, combined with disk space management issues on the agents themselves, would lead to frustration as engineers attempted to deploy code to customers as quickly as possible. Taking on board our engineers' experiences with Jenkins, the DevOps team started looking into building a custom solution to address these issues.

Our custom-built CI/CD alternative to Jenkins

After a year of design and development, the DevOps team created an in-house CI/CD deployment solution, using GitHub Actions and AWS CodeBuild, which we've named Huxley. Read our separate post to learn how to create GitHub actions in Ruby.

Simply put, Huxley is a solution that uses GitHub Actions as the front-end, while the actual CI/CD actions are performed in AWS CodeBuild. This solution helps to solve the unintuitive nature of Jenkins by having GitHub act as the UI component - a familiar interface for all engineers. Thus our engineers no longer need to leave GitHub to debug deployment issues.

Using GitHub Actions as orchestrator

A key design component of our Huxley solution is the modularisation of the different stages in the pipeline. Each stage is treated as an autonomous event that does not necessarily need any other build stage. This means that our CI/CD system can simply orchestrate each stage into a functioning pipeline, based on the designed workflow.

We use GitHub Actions as the orchestration engine in Huxley. GitHub Actions maintains the order in which the jobs are run and when the jobs are sent to CodeBuild.

Using AWS CodeBuild as the build engine

On the back-end, we're using CodeBuild as the engine to call our custom deployment scripts and determine which build artefacts are created and tested. This could be a Docker image, an AMI, a gem or anything similar. The running of the builds and deploys is handled by our deployment tooling.

CodeBuild also acts as the engine that determines which build artefact is deployed to the larger environment. This could be an internal or external application.

CodeBuild provides us with build agents on an as-needed basis. By having all of our CI/CD tooling in a single Docker image, we can scale to as many build agents as we need during a typical work day. As the build agents are started as-needed, they last only as long as the assigned CI/CD action takes place. This solution allows for increased stability and usability for our engineers. We will explain this solution further down.

Build engine as builder and deployer

Using our toolbox (described later), CodeBuild is able to act as an engine to run the build stage of the pipeline and subsequently as the deployer to deploy the artefacts it creates in the build stage. Each stage is run independently, with CodeBuild used as the runtime engine to run our build and deploy tooling. All of this is managed by Github Action acting as the Orchestrator of the overall pipeline.

Plug-and-play solution

The tooling we use for our deployment framework is contained within a single Docker image or toolbox. We developed this toolbox so that it could be run locally or on a build agent. The toolbox contains all of our custom bash scripts, CLIs written in GoLang, security tools and our secret sauce that enables Simply Business to deploy reliably and securely hundreds of times per day. In terms of our CI/CD solution, this meant that the Docker image could be run in the same way locally, on Jenkins and in CodeBuild. This made the migration from Jenkins to Huxley much easier.

Future customisation of our CI/CD solution

While the adoption of our GitHub Actions and CodeBuild CI/CD solution has improved the quality of life for engineers at Simply Business, it has also allowed for improvements in other parts of the business as well.

By incorporating GitHub Actions, it's opened up workflow possibilities that did not exist under the Jenkins model. We've built a workflow that enables security patching for our internal build systems, resulting in a decrease from two weeks to around one hour. In addition, a complex patching cycle with excessive manual touchpoints is now encompassed by another workflow that, in one click, can trigger the entire patching process.

The future possibilities for customising our CI/CD solution Huxley lies not only in further improvements in engineering but also in adopting new workflows that can transform what we can do to simplify and automate normal everyday DevOps tasks.