Review Apps With Kamal (Part 6) - Docker Cache and PR Comments

Throughout this series of articles, we’ve gone through the entire process of setting up a review application workflow for web apps using Kamal, from setting up our cloud infrastructure to getting our application fully deployed and ready for testing.

The review app process works well and is stable enough to be used on any web project hosted on GitHub. Still, we can tweak a few things to make the workflow more efficient and easier to manage. This article will cover two additional nice-to-have modifications that will make updates to review apps much faster and help others access them with a single click.

Improvement #1: Caching Docker builds

When setting up this process, you might notice that when we perform the deployment with Kamal, it rebuilds the Docker image from scratch every time we update a pull request.

By default, Docker caches the build on disk using its internal cache, which speeds up future builds by skipping any layers that the codebase hasn’t modified until it encounters a layer that needs updating. This kind of caching works great when building Docker images on your local system, but since we’re building the app on a GitHub Actions self-hosted runner, we lose that build cache every time we use Kamal for deployments.

Not having the build cached leads to slow updates to our review apps, especially on GitHub Actions, since their default runners are much weaker than a typical developer’s system. In the example used in this series, it takes over five minutes just to build the TeamYap app when updating the pull request.

Five minutes doesn’t seem like much, but this time adds up, especially when you’re trying to test and iterate on a branch rapidly. Instead of rebuilding the image every time, even if there’s a minor change to the codebase, we should find a way to cache the build externally so we can reuse it and speed up the process.

Enabling caching in Kamal

Fortunately, Kamal has support for setting up Docker’s cache storage backend, which we can use to cache our builds. To get it working, we need to add a few new settings to the Kamal configuration file in our repo. I’ll set up caching for the review destination only, so I’ll open up the config/deploy.review.yml file that contains the configuration for this destination and add the following new section:

builder:
  cache:
    type: gha
    options: mode=max

The top-level key called builder lets us configure how to build the Docker image when deploying with Kamal. You can use this key to control different settings that Kamal passes over to Docker during the build process of your application’s image. The configuration we’re interested in relates to caching, which we can set using the cache key under builder. This setting uses a few parameters to configure how we want caching to work.

First, we need to specify the type of cache we’ll use. This setting tells Kamal which storage backend to use. Docker supports multiple caching mechanisms, but as of Kamal 2.6, it only accepts two kinds: registry, which builds and pushes the image and the cached build to a Docker registry, and gha, which is more convenient for our use case since it uses GitHub Actions. The gha cache type saves the build cache, including Docker image layers, to the integrated caching service provided by GitHub Actions, so we don’t need to worry about setting up a registry or other storage backend.

Besides setting the type, we can also set additional options to use for the caching process as comma-separated key-value pairs, each assigned by the equals sign. The gha cache has a few different options to use, most of which you likely won’t need. However, one option that you should consider setting up is the mode option to specify what to cache.

There are two options for setting the mode. The min cache mode only caches exported layers, which are essentially the layers contained in the final build, and won’t include any layers from intermediate stages. This mode reduces the amount of storage used for caching and speeds up the caching step. However, depending on your Docker builds, you’ll probably have fewer cache hits, which defeats the purpose of caching.

The max cache mode, on the other hand, caches every single layer in the process, including all layers from intermediate build stages. This mode gives you a much better chance of having cache hits during subsequent builds, with the trade-off being that the resulting cache will be much larger.

Depending on how you set up your Dockerfile, one mode can work better than the other when it comes to caching. In my experience, the typical Docker build for a Rails app benefits from having all its layers cached, so I set the mode to max when creating the image with Kamal. If you’re using Kamal to deploy other types of web applications, it’s worth experimenting with both modes to see what works best in your use case.

[Sidenote: Speaking of cache storage, it’s worth noting that the GitHub Actions cache service has a 10GB limit. When exceeding this limit, your old caches will get evicted from the service, which can lead to slower builds, so make sure to keep your Docker images as small as possible to reduce the chance of this happening.]

Configuring GitHub Actions workflow to enable caching

With this builder configuration, Kamal will handle setting the appropriate flags during the Docker build process. However, this configuration alone won’t get caching working on GitHub Actions. We’ll need to add a few more steps to our GitHub Actions workflow to ensure everything works correctly.

First, we’ll need to update the self-hosted runner to use the correct build driver for Docker. Kamal uses the default Docker build driver during the build process, which doesn’t support exporting the cache to the gha storage backend that we set up in the Kamal configuration. Thankfully, switching the driver on a GitHub Actions workflow is straightforward.

The Docker team has an action called docker/setup-buildx-action that handles this automatically for us, so let’s add it to the deploy job we set up earlier in this series. We’ll need to set this up before triggering the Kamal setup or deployment so it’s ready to use at that stage. I’ll place it after checking out the code into the runner:

name: Review app deployment

on:
  pull_request:
    types:
      - opened
      - reopened
      - synchronize

jobs:
  provision:
    # Provision job configuration and steps omitted.

  deploy:
    # Deploy job configuration omitted.

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      # New step.
      - name: Set up correct build driver for Docker
        uses: docker/setup-buildx-action@v3

      # Remaining steps omitted.

This step allows us to configure the Docker builder on the runner. By default, it uses the docker-container build driver, which allows cache exports to the GitHub Actions cache service, so we don’t need to add any additional configuration to this step here.

Using docker/setup-buildx-action configures Docker to export the cache for our builds, but the gha cache storage needs to know the remote URL of the service to retrieve and set the cache. It also needs authentication to allow the workflow to connect to the service. Again, we’re lucky that an action exists to automatically handle these steps for us, called crazy-max/ghaction-github-runtime. Let’s add it after the new step we just added:

name: Review app deployment

on:
  pull_request:
    types:
      - opened
      - reopened
      - synchronize

jobs:
  provision:
    # Provision job configuration and steps omitted.

  deploy:
    # Deploy job configuration omitted.

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up correct build driver for Docker
        uses: docker/setup-buildx-action@v3

      # New step
      - name: Expose GitHub Runtime for cache
        uses: crazy-max/ghaction-github-runtime@v3

      # Remaining steps omitted.

This new action exposes a few environment variables that the gha storage backend can use for authentication and to know where to send the cache. These environment variables automatically become available on the runner without any additional configuration:

• ACTIONS_RUNTIME_TOKEN: The authentication token that gha uses to access the GitHub Actions cache service.
• ACTIONS_CACHE_URL: The URL pointing to the GitHub Actions cache service for the current repository.

With these new steps added, our cache is in place and ready to work.

Workaround for Kamal timeouts

In some cases, adding this additional configuration to set up caching with Kamal can lead to timeouts during the setup or deploy process when the Docker build isn’t cached. Kamal opens an SSH connection to the server, which remains open during the build process. If it takes too long, the connection will drop with a timeout error, and the workflow will fail.

In the example application for this series, the time overhead added by the Docker build caching causes these times to occur when running the workflow for the first time in a new pull request. While I’m not sure exactly why this is happening, I suspect it’s likely due to how SSH works on the GitHub Actions self-hosted runner, which prevents any stuck connections from halting a workflow.

As a workaround, I added an extra step just before running the Kamal setup or deployment to build and cache the image first without logging in to the review app server and push it to the local image store on the runner. This way, the subsequent setup or deploy step can access the cached build and avoid the timeout:

name: Review app deployment

on:
  pull_request:
    types:
      - opened
      - reopened
      - synchronize

jobs:
  provision:
    # Provision job configuration and steps omitted.

  deploy:
    # Deploy job configuration omitted.

    steps:
      # Previous steps omitted.

      # New step before running Kamal setup or deploy.
      - name: Build and push Docker image
        run: bundle exec kamal build push --output=docker -d review

      # Kamal setup/deploy step omitted.

The new step uses the kamal build push command to use Kamal to build and push the Docker image. We’re also specifying the --output=docker flag to push the image to local storage instead of the container registry, which is the default for the kamal build push command. We also need to specify the destination (-d review) to use the new caching configuration we just set up in this branch.

With this additional step, GitHub Actions will build the application’s Docker image before moving on to the Kamal setup or deploy step, which will only take a couple of seconds since the image will be cached beforehand.

Testing the Docker build cache

When opening a new pull request in the repository and going through the provision and deploy workflow we have set up, Kamal performs the full build for the Docker image, as seen earlier. However, if we go to the cache section for our repository, we’ll see a lot of new entries:

Each of these entries is one of the cached layers from the build, along with some build metadata, since we set the mode to max in the Kamal configuration. If we set the mode to the default min setting, we’d have fewer items in the cache.

We have a cached build available for this pull request, so let’s update the branch and push the changes to trigger the workflow again. When we see the time spent during the build, we’ll see that most of the layers were cached and don’t require building, so the overall time to complete this process is drastically reduced.

With just a handful of changes to Kamal and GitHub Actions, we now sped up our Kamal deploys by almost 90% after the initial build. This improvement helps reduce the amount of time spent waiting for the review app to redeploy after an update.

Now that we’ve sped up redeployments to our review apps, one last thing I wanted to address in this process is making it easy for people to know the location of the review app.

Improvement #2: Show the URL of the review app

In part 4 of this series, I modified this workflow to generate a unique URL for each review app. The URL is based on the pull request number to make it easy to access, but we can make this easier by having GitHub Actions submit a comment after a successful deployment, showing where others can find the app.

We can handle this directly in our GitHub Actions workflow by using actions/github-script. This action sets up everything we need to access the GitHub REST API, including authentication, the workflow’s context, and references to different libraries for interacting with the runner.

The GitHub API has an endpoint for creating issue comments, which we can use here since a pull request is technically an issue on GitHub’s system. Let’s begin setting up this action in our review app workflow by adding a new step after we set up or deploy our app since we want this step to take place after the Kamal process wraps up.

name: Review app deployment

on:
  pull_request:
    types:
      - opened
      - reopened
      - synchronize

jobs:
  provision:
    # Provision job configuration and steps omitted.

  deploy:
    # Deploy job configuration omitted.

    steps:
      # Previous steps omitted.

      # New step after running Kamal setup or deploy.
      - name: Leave a comment
        uses: actions/github-script@v7
        with:
          script: |
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: `Your review app can be found at https://${process.env.REVIEW_APP_HOST}.`,
            });

Explaining actions/github-script

This action requires one parameter (set under with) called script. The parameter takes an asynchronous JavaScript function call, which we can use to make REST API requests to GitHub. The action itself uses the Octokit REST JavaScript library under the hood, and it’s pre-authenticated using the default GITHUB_TOKEN environment variable that GitHub Actions creates on every run, so all we need to do is specify our function call to the API using Octokit.

The github-script action sets up the Octokit client object as github in this action, so we’ll use that to call our function. We can set up the request for creating a new issue comment with await github.rest.issues.createComment, using await since the action works with asynchronous JavaScript functions. The createComment function requires four parameters:

• owner: The owner of the repository where we’re making the API request. The workflow’s context contains this information, which we can access here using context.repo.owner.
• repo: The full name of the GitHub repository for the request. Again, this information is in the workflow’s context under context.repo.repo.
• issue_number: The number of the issue or pull request where the comment gets posted. As mentioned, GitHub considers pull requests as an issue, so the name may seem confusing, but setting the pull request number here is valid. We already have this information available in an environment variable, but it’s also accessible in the workflow’s context under context.issue.number.
• body: The content of the comment’s body.

We can use plain text or Markdown, but for simplicity, we’ll use plain text. As shown here, we also have access to the runner’s environment variables using process.env. I previously set up the URL for the review app in the runner as REVIEW_APP_HOST, so I can include it in my comment body with process.env.REVIEW_APP_HOST.

With this, our workflow will create a new comment in the pull request using the GitHub Actions bot user since we’re authenticating the client using the default GITHUB_TOKEN. When updating the pull request I created for this article, I can see a new comment appear showing the URL for this review app.

Preventing duplicate comments

Creating a new comment is working great, but we have an issue. If the pull request is updated after it has the comment pointing to the review app, the GitHub Actions workflow will create another comment with duplicate content every time. We should only have this comment created once, so let’s update the step to do that.

We’ll want to check any existing comments for the pull request to determine whether we should create one or not. There are a couple of ways to do this, such as checking the body of all comments to see if they contain the review app’s URL. For this example, I’ll assume that the GitHub Actions bot user only creates this specific comment, so I’ll update the script to check if any comments are coming from this user. If there isn’t, we’ll create the comment. If there is an existing comment, we’ll skip the API request.

Here’s how the updated script looks like:

name: Review app deployment

on:
  pull_request:
    types:
      - opened
      - reopened
      - synchronize

jobs:
  provision:
    # Provision job configuration and steps omitted.

  deploy:
    # Deploy job configuration omitted.

    steps:
      # Previous steps omitted.

      # Updated step to prevent duplicates.
      - name: Leave a comment
        uses: actions/github-script@v7
        with:
          script: |
            const { data: comments } = await github.rest.issues.listComments({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number
            });

            const existingComment = comments.some(comment => comment.user.login === 'github-actions[bot]')

            if (!existingComment) {
              await github.rest.issues.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: context.issue.number,
                body: `Your review app can be found at https://${process.env.REVIEW_APP_HOST}.`,
              });
            }

At the beginning of the script, I’ll make a request to the GitHub REST API as I did before using the await github.rest.issues.listComments function. The listComments function retrieves all the comments for a given issue or pull request. This function requires three parameters, which are identical to the owner, repo, and issue_number parameters used in the createComment function. The results of the API request are stored in the destructured comments variable, containing an array of all the comments in this specific pull request.

Next, I want to see if any of the objects inside comments matches the GitHub Actions bot username, so I’ll create a new constant called existingComment. For its value, I’ll iterate through the comments array using the some method. If the function specified in this method returns true for any of the elements in the array, then it will return true. If none of the elements returns true, then our result is false.

The function I’ll set for some is simple, so we’ll take the comment as the function parameter and check the user to see if it matches the GitHub Actions bot user. The simplest way I found to do that is to check the comment.user.login property to get the GitHub username of the account that created the comment and check if it matches the name github-actions[bot]. This username may look a bit unusual, but it’s the official username of the GitHub Actions bot user, and we can use it to match any comments.

Now that the script knows whether the GitHub Actions bot has created a comment in this pull request, we can use this as a conditional check to generate the comment with the review app URL if necessary. I added an if statement so that when existingComment is false, we can create a new comment. Otherwise, it’ll skip the API request, and we won’t get any duplicates.

With this update, we can now update this pull request as many times as we want, but it will only create the comment with the review app URL only once.

Wrapping up

The updates made in this article are minor and optional for our review app workflow but have a huge impact on team productivity. Developers and testers will enjoy quicker iterations by speeding up build times for updates and having direct links to the temporary environment directly on the pull request. The result of this is faster QA and shipping quicker than ever.

From here, you can continue to expand this process and make it fit your development lifecycle better. Reach out to me and let me know if this series has helped you and how you’ve added and improved upon the basics that I demonstrated in the past six articles. I hope that it serves you and your team well in building and delivering high-quality apps in record time!