From this article, you will learn step by step how to run automated tests with CI/CD managed by GitHub Actions together with a test runner able to deal with your private services deployed in an AWS VPC (virtual private cloud).
You will see how powerful is the combination of AWS Code Build and K6 tools in a case when automated performance testing is a must-have.
Why performance test automation is so important?
How many times have you started with a small, fast service but as the service started growing the performance was degrading? Finally, you ended up with a fancy service but with poor performance.
Probably you were also in a situation where preparing every time simple performance tests took ages and you could have used that time more efficiently. Having stress tests automated allow you to spare your time and focus more on providing business value. Luckily there is a solution for that. With available tools, you can avoid such cases and make performance testing a routine and be a part of the CI/CD process of your microservices.
Additionally, you can have some kind of a gateway that only allows deploying to production if the service meets performance requirements.
Here is our scenario
We are going to test the REST API of a service that is deployed in a private subnet of an AWS VPC. Let’s have a look at the infrastructure diagram.
You’ve noticed probably that in such a case we can’t access the service’s REST API from GitHub Action CI/CD because it’s not publicly available. It’s deployed in the private subnet. That’s exactly the case when AWS CodeBuild comes into play.
“AWS Code Build is a fully managed continuous integration service that compiles source code, runs tests, and produces software packages that are ready to deploy. With CodeBuild, you don’t need to provision, manage, and scale your own build servers.”
We can configure the AWS Code build to „be part” of our VPC and then grant access to our private subnet. Thanks to that we can test our private service’s REST API.
Hint
You can also use AWS CodeBuild when your REST API is publicly available but you need more powerful resources to perform the performance tests. GitHub Actions job runners have 2 x86_64 CPU cores and 7 GB RAM. CodeBuild computes types provide more interesting options including:
- up to 72 x86_64 vCPUs
- up to 145 GB RAM
- up to 8 ARM64 vCPUs
- GPU hardware devices
Implementing the CI
We will use GitHub Actions – a modern CI/CD automation server that allows us to customize and execute our software development workflows right in your GitHub repository.
CodeBuild allows us to access our microservice in the private network and run the tests.
GitHub Action will just trigger the CodeBuild and then wait for a result. For performance testing, we will use the powerful open-source load testing tool K6. The tests will be run by the CodeBuild.
1) First let’s create the AWS CodeBuild project
- navigate to the AWS CodeBuild console and create a new project
- for more details go to: Create a build project (AWS CLI)
a) Next, under Source, configure source repository
You will need to configure GitHub access token although a repository is public:
creating-a-personal-access-token
b) Configure the environment which will be used to run our performance test
c) The most important part – we have to configure the access to the private subnets in the VPC where the tested service is deployed.
The private subnet must have a NAT gateway assigned! Access Resources in a VPC from AWS CodeBuild Builds
d) finally – configure log files to get details about executed tests
2) Now we will create a simple parameterized performance test with K6
Let’s create the test-api.js in the root project folder.
The test is parametrized via environment variables as follow: the URL is the http URL to be called during the test (with GET method), MAX_FAILED_RATE is the maximum accepted percent of fail responses and maximum value of the accepted 95th latency MAX_P95_MS
You can install K6 and run the tests locally but remember that the URL you provide has to be accessible from the local environment ( eg. it can be achieved via bastion host and ssh tunnel).
3) Now we have to instruct AWS CodeBuild how to call the tests
To do that we have to define the pipeline in the buildspec.yml file in the root project folder.
The pipeline installs the K6 tool and runs the tests passing environment variables.
4) As we want to have the performance tests to be part of our pipeline finally we have to configure GitHub action to run the AWS CodeBuild project.
Here is the example of the Github action pipeline containing the step for running performance tests. This is our Github action workflow file: deploy_ci.yml
As you can see there is the step Run CodeBuild performance tests. The step uses the AWS-actions/AWS-code build run action to run our CodeBuild project.
The step Configure AWS credentials
is also mandatory. This is the place where access to the AWS account is configured.
The variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
must be defined as Github secrets https://docs.github.com/en/actions/reference/encrypted-secrets
5) The result we can see after pushing changes to GitHub. It will trigger GitHub Actions. Then AWS Code Build will be triggered and our test will be performed
After they are finished AWS GitHub Actions are notified about the result.
Here you can see the example output.
Hints
You can also export metrics from the tests to the AWS CloudWatch and visualize them on a dashboard. For details please check the link results-visualization/amazon-cloudwatch
Be aware that to do precise performance tests there are much more things which shall be taken into account like:
- warming the environment up to get stable tests results,
- right test data preparation and randomizing to prevent server-side caching from impacting tests.
For stress and spike/performance testing the presented solution is sufficient for most cases but for load testing, it can better create a dedicated environment setup on EC2 to guarantee stability and much more power.
Summary
In this post, you read about how to do automated performance tests for private AWS microservices.
Using GitHub Actions combined with AWS CodeBuild and K6 gives a powerful way to deliver a feature-rich CI/CD pipeline to keep your services in good condition avoiding performance degradation.
You can find code sources on my GitHub: https://github.com/amalioadam/code_build_k6
