Dead Simple Load Testing at Scale on Lambda or Fargate

Checking the performance and resiliency of your web applications is not an optional step in the software development process these days. Users want what they’re looking for fast, and if your site doesn’t load quickly enough for them, they’ll leave and likely never return. If you want to keep them around, you’ll need to make sure your site is speedy and responsive.

My favorite testing tool to check the performance of my web apps is Artillery, an open-source load testing tool with everything you need to check how much load your site can withstand without breaking down. Using Artillery to create and execute load tests that go through common flows on your web APIs is straightforward, helping you spot obvious performance issues that you can fix quickly.

Running these tests from your local development machine works well for the basics, but what if you really want to test the limits of your application and infrastructure? While you can send hundreds or thousands of simultaneous requests as part of your testing, chances are you’ll be restricted by resource limitations since you’ll need to generate all that load on your local system.

Fortunately, you won’t have to worry about that with Artillery. In this article, I’ll show you how to go farther with your load testing by running these tests at scale on the cloud.

Load Tests at Scale With Artillery Through AWS

Artillery has everything you need built-in, so you won’t need to mess with plugins, separate services, or complicated setups. All you’ll need is Artillery and an Amazon Web Services (AWS) account.

With a single command, Artillery can run your load tests on the cloud through AWS Lambda functions or Amazon ECS Fargate tasks, handling all the setup on your AWS account automatically for you. You’ll only pay for what you use, making it a cost-effective solution for large-scale load tests.

Let’s see an example Artillery load test I built for Airport Gap, a web application to help developers and testers improve their API testing skills. If you’re new to Artillery, check out the article Battle-Test Your API With Artillery Load Testing for an introduction.

config:
  target: "https://demo.airportgap.com/api"
  phases:
    - duration: 30
      arrivalRate: 1
      rampTo: 10
    - duration: 300
      arrivalRate: 20
  payload:
    path: "users.csv"
    fields:
      - "username"
      - "password"

scenarios:
  - flow:
      - get:
          url: "/airports?page={{ $randomNumber(1,100) }}"
          capture:
            - json: "$.data[{{ $randomNumber(1,10) }}].id"
              as: "airportId1"
            - json: "$.data[{{ $randomNumber(11,20) }}].id"
              as: "airportId2"
      - post:
          url: "/airports/distance"
          json:
            from: "{{ airportId1 }}"
            to: "{{ airportId2 }}"

  - flow:
      - post:
          url: "/tokens"
          json:
            email: "{{ username }}"
            password: "{{ password }}"
          capture:
            json: "$.token"
            as: "token"
      - get:
          url: "/favorites"
          headers:
            Authorization: "Bearer token={{ token }}"

This load test has two phases. First, it begins with a gentle warm-up phase with one virtual user and ramps up to 10 virtual users in 30 seconds. After that first phase, the next phase runs a steady flow of 20 concurrent virtual users every second for the next five minutes. The test also loads a CSV file containing a list of test account emails and passwords to use during the test.

For the scenarios, we’ll have two flows. The first flow hits an endpoint to fetch a random page of airports and captures two random entries, and then hits a second endpoint to calculate the distance between those two airports. The second flow logs in as one of the users defined in the CSV file to capture their authentication token and fetch their favorite airports.

After installing Artillery, I can run this test locally using the command artillery run load_test.yml to see it in action. This initial test gives a small taste of how the application performs at a base level since we’re only sending a limited number of requests.

I can bump up the number of virtual users to send to the API or increase the length of the test if I want to stretch the limits of this system. However, simulating thousands of virtual users is pretty demanding on your hardware, and you’ll need plenty of bandwidth to sustain throughout the test run if you want consistent results.

Instead of finding more powerful hardware or hoping you have enough stable bandwidth, a better solution is to offload these tests to the cloud. Artillery allows you to run your load tests on AWS Lambda or ECS Fargate right from your terminal, in most cases without having to modify the test scripts at all.

Setting up AWS to Work With Artillery

Before we run load tests on AWS, we’ll need to ensure our system has the correct permissions to let Artillery set up the resources it needs on our AWS account. Artillery uses the AWS SDK to create the resources it needs to run load tests on your AWS account. You’ll need to configure the system running the tests to access AWS resources through the command line.

You can set up your AWS credentials locally in a few ways, like setting environment variables or creating a credentials file. If you’re setting up AWS for the first time, you’ll need to configure your system before running your tests with Artillery.

You’ll also need to make sure the AWS profile you configure on your system has the correct permissions to create the resources Artillery needs to run your tests. The easiest way is to use an IAM user with the AdministratorAccess policy assigned to it, which grants it permission to create just about anything on an AWS account.

However, it’s best to apply the principle of least privilege, meaning that an IAM user should have the fewest permissions needed to accomplish a given task. The Artillery documentation has example policies for running load tests using Lambda or Fargate that you can set up on your AWS account and apply it to the IAM user for your configured profile.

The documentation for each page also has a CloudFormation stack available that can automatically provision a policy and role by clicking the “Launch Stack” button on the documentation, so you don’t need to set it up manually.

Once your system is configured with a profile with access to generate and use the AWS resources Artillery needs, we can run our load tests on the cloud.

Before proceeding, I’ll tweak the load test shared earlier, changing the number of virtual users for the main phase from 20 virtual users per second to 1000. This change will help us begin testing how far we can stretch our API.

config:
  target: "https://demo.airportgap.com/api"
  phases:
    - duration: 30
      arrivalRate: 1
      rampTo: 10
    - duration: 300
      arrivalRate: 1000 # Change the number of virtual users per second.

# ...
# The remainder of the test remains the same.

Running Load Tests at Scale on AWS Lambda

To execute an Artillery load test on AWS Lambda, the command to use is artillery run-lambda. This command is similar to running the test locally (artillery run), but it requires a flag to specify which AWS region you want to set up the resources and run the Lambda function from. The --region flag lets you do this.

For instance, if you want to run an Artillery load test from a file called load_test.yml in the us-west-2 region (US West Oregon), you can use the following command:

artillery run-lambda \
  --region us-west-2 \
  load_test.yml

That command is all you need to have Artillery run your tests on the cloud. You likely won’t need to modify the test script or install any additional plugins or tools for this so it can’t be any more simpler than this.

When you run this command, Artillery will check if your configured AWS account has the resources it needs and create them if needed. It then prepares a Lambda function and uploads a bundle containing the files for your load test, including any payload files specified in the test script.

The first time you run this command, it’ll take a minute or two while Artillery creates the resources on AWS before it starts to run the tests. After a short while, Artillery begins running the load test on Lambda and showing metrics every ten sections, just as if you were running the tests locally.

Artillery running a load test on AWS Lambda

Upon completing the test, you’ll see results similar to those when running the tests locally. It also shows you an estimated cost for running this test on Lambda so you can have an idea of how much one of these test runs will cost. Typically, they’ll just be a few pennies or less, so unless you’re running a very large-scale load test, running these tests on Lambda won’t cost you much.

Additional flags for running Artillery tests on Lambda

The artillery run-lambda command only requires the --region flag. Artillery uses a few default settings that you likely won’t need to change, but they are good to know if you want to tweak how your load tests run on Lambda.

By default, Artillery will run your load tests on a Lambda function powered by AWS’s ARM-based Graviton processors. It’s a good default because Lambda functions running on Graviton can offer better performance at a lower cost. If you need or prefer to use an x86-powered Lambda function, the --architecture flag allows you to change this setting with a value of x86_64:

artillery run-lambda \
  --region us-west-2 \
  --architecture x86_64 \
  load_test.yml

Another helpful setting you might want to tweak is the memory size of the Lambda function. Artillery sets the memory size of the Lambda function to 4096 Megabytes by default, but you can adjust this setting by using the --memory-size flag. For instance, if I want to bump up the memory size of the Lambda function to 8192 megabytes, I can use the following command:

artillery run-lambda \
  --region us-west-2 \
  --memory-size 8192 \
  load_test.yml

You can also adjust how many load generators you want to spin up on Lambda. The run-lambda command runs your test on a single Lambda function. However, if you want to run multiple load tests simultaneously, using the --count flag will tell Artillery to run the load tests on multiple Lambda functions.

The following command will run your load test on three simultaneous Lambda functions, meaning your systems will receive three times the requests. If you want to test the limits of your system, this is a great way to do it:

artillery run-lambda \
  --region us-west-2 \
  --count 3 \
  load_test.yml

You can find more flags to change how Artillery runs your load tests in Lambda in the Artillery documentation.

Limitations when running Artillery load tests on Lambda

While using AWS Lambda to run Artillery load tests is as simple as running them locally, there are a few limitations you need to be aware of:

Because Lambda functions have a maximum function timeout of 15 minutes, you’ll need to make sure your load tests finish within that period. Otherwise, you’ll receive incomplete test results.
Once Artillery launches the load test in Lambda, there’s no way to interrupt the test, so be careful about where you’re targeting your load tests because once it starts, you’ll have to wait until it finishes.
Some Artillery features work differently or aren’t available when running your tests on Lambda, such as before and after hooks running per Lambda function instead of once in a test run.

Running Load Tests at Scale on AWS Fargate

The command to run Artillery tests on Fargate is almost identical to running them on Lambda. Instead of artillery run-lambda, we’ll use artillery run-fargate. We also need to specify the region through the --region flag:

artillery run-fargate \
  --region us-west-2 \
  load_test.yml

Running this command will create an ECS cluster on your AWS account and launch a task to run the Artillery load tests. Once that happens, Artillery begins showing test metrics every ten seconds, including a few additional metrics that don’t appear when running the tests locally or on Lambda.

Artillery running a load test on AWS ECS Fargate

Just as when running tests on AWS Lambda, you’ll be able to see the results of the test run when it finishes. However, unlike running the tests on Lambda, you won’t see a cost estimate at the end. Depending on the use case, the pricing should be similar, so running your load tests on ECS is still a cost-effective solution.

Additional flags for running Artillery tests on Fargate

The artillery run-fargate command also sets a few default configuration settings you may want to adjust for your needs.

As mentioned, Artillery will create an ECS cluster on your account to launch tasks on Fargate. If you have an existing cluster that you would prefer to use, you can set it with the --cluster flag and specify the name of your desired cluster:

artillery run-fargate \
  --region us-west-2 \
  --cluster my-cluster \
  load_test.yml

Artillery configures the resources for the Fargate tasks that run load tests with 4 vCPUs and 8 gigabytes of memory. If you need to increase the capacity for the tasks, the --cpu and --memory flags will adjust them. For example, if you need the Fargate tasks to run with 8 vCPUs and 16 gigabytes of memory, use the following command:

artillery run-fargate \
  --region us-west-2 \
  --cpu 8 \
  --memory 16 \
  load_test.yml

Artillery also launches one Fargate task to run your load tests by default. You can increase the number of concurrent tasks by setting the --count flag so you can generate additional load during your tests, similar to how the same flag works on Lambda:

artillery run-fargate \
  --region us-west-2 \
  --count 3 \
  load_test.yml

The run-fargate command has lots more additional flags to change how Artillery runs your load tests on Amazon ECS, so make sure to check out the documentation to learn more and see a few useful examples of other ways to launch these tests on Fargate.

Which Service Should I Use to Run Artillery Load Tests on AWS?

Running Artillery load tests on Lambda or ECS Fargate work almost identically, with Artillery creating the required AWS resources and running your load tests from the cloud. In many cases, using either Lambda or Fargate will work equally well. However,

Run your tests on Lambda if…

Your tests will run in less than 15 minutes.
Your tests don’t require too many computing resources.
You want the simplest route for running load tests from your AWS account.

Run your tests on ECS Fargate if…

Your tests run for over 15 minutes.
You need additional computing capacity when generating load for testing.
You need more control over the runtime environment.
You want to save cash by running load tests on spot instances.

Wrap Up

If you need to run load tests at scale, there’s no better solution than Artillery. With its built-in support to run load tests on AWS Lambda or ECS Fargate using your own account without any extra work, you’ll be up and running in no time compared to other load testing tools. Lambda and Fargate work equally well, but each has its strengths and limitations, so it’s worth understanding your testing goals and choosing accordingly.

Running load tests regularly as part of your continuous integration process should become a priority on your team if you want to keep your systems running smoothly. Thanks to Artillery, the ability to run these tests at scale on the cloud will help you test the limits of your applications without breaking the bank. Whether you run your tests on Lambda or Fargate, you’ll have all you need to catch performance issues early and help you fix them before they reach your users.

Are You Looking To Get Started With Load Testing?

Getting started with load testing can be confusing. How many virtual users should you begin with? Which application flows should you test? Should you test in a production environment? You’ll likely have these questions and many more.

If you want to kick-start your load-testing processes, I can help get you started on the right foot. Contact me today, and let’s start a conversation about what you’re looking for.

Screencast

If you found this article or video helpful, please consider subscribing to my YouTube channel and follow for more tips on helping developers ship their code confidently, from development to deployment.