Review Apps With Kamal (Part 5) - Using Hooks to Check Setup

Things are shaping up nicely as we roll along in this series on using Kamal to spin up review applications when opening new pull requests. We have a GitHub Actions workflow that creates new servers on the cloud, sets up a destination for these review apps, and automates the deployment process to get a new web application instance up and running on a unique domain name. This entire sequence already works great for letting teams begin testing pull requests when using Kamal for deployments, but we still have some lingering issues to address.

In Part 3 of this series, we set up our continuous integration environment (GitHub Actions) to have Kamal set up and deploy the application to the new server we spin up when opening a pull request. To handle this, I set up the workflow to check the event type and determine which step to take. We’ll run ’ kamal setup ’ if we trigger the workflow by opening a pull request. If the workflow event is from an update to the topic branch for the PR, we’ll run kamal deploy instead.

The reason for splitting up these steps is because Kamal needs to perform the initial setup and subsequent deployments separately. The kamal setup command installs Docker on the host servers if not present, boots up all defined accessories in the configuration, and performs the initial deployment. The kamal deploy command bypasses the Docker check and booting of accessories, only doing the deployment step.

The Current Issue With Kamal’s Setup and Deploy Commands

The problem with these commands is that they depend on one another. If we run kamal setup on a server that already had the command complete successfully, Kamal will attempt to boot the accessories again and fail because they’re already available on the remote servers. Similarly, if we run kamal deploy on a server that has never completed the setup step, it will also fail since the server won’t have some of the necessary components.

Because there are two separate commands that we need to run with Kamal, we need to conditionally check when we should run either command. If we’re opening or reopening a pull request, we’ll run the setup command since part of the workflow spins up a fresh new server on the cloud. But if we update the branch for a pull request, we assume Kamal is already set up on the server from the initial setup and will run the deploy command instead.

If everything works as expected, this process is fine, but you can’t trust that the CI job will always complete the setup step successfully. There’s always the possibility of something not working. For instance, we might push some code that breaks our Docker builds or Kamal configuration, so the workflow can’t complete the setup step. Another example that can happen is when our provisioned server has a sporadic issue that shuts down the connection in the middle of the setup process.

If a scenario like this were to happen when triggering the review app workflow on a newly opened pull request, we’d get stuck since we can’t run the deploy command unless we’ve set up our server first. If we push a new update to the pull request to fix the problem that caused the setup step to fail, it’ll attempt to deploy without having all the pieces in place, and the command will fail. The only way to resolve these issues with our current workflow is to delete the pull request to tear down the server and open a new pull request to start the process from scratch.

This strategy is far from ideal, so we want to fix this so that GitHub Actions doesn’t rely on the type of event triggering the workflow.

A Different Way to Check Server Deployment Readiness

Unfortunately, Kamal doesn’t have a command or other straightforward way to determine whether it has fully deployed an application on a server. Kamal has commands like kamal accessory details all to check the accessories currently running on a server. We could potentially use this and similar commands to parse any details we need, like checking the availability of an accessory to determine that we’ve completed the setup step previously.

However, using these commands carries their own sets of risks. For instance, if we write a script that uses kamal accessory details all to find a list of known accessories that our application needs, it can work for the existing state of the application. But in the future, we might modify the accessories to add a new one or remove an existing one we no longer need, meaning we’ll need to remember to update our scripts to account for the change. It’s easy to forget this, making this review application process brittle, so I wouldn’t recommend it.

Instead, an easier and more stable solution to verifying whether a Kamal-managed application has been set up on a remote server is to take advantage of Kamal hooks.

Using Kamal Hooks to Set an Indicator for Server State

Kamal hooks are a way to allow us to execute custom commands and scripts at different points of the Kamal lifecycle. When initializing Kamal in a project, it creates a directory under .kamal/hooks containing various sample scripts for each kind of hook that Kamal can fire from your system. These hooks can help us further automate specific actions that need to happen when running a Kamal command. Some helpful uses of Kamal hooks are:

• Updating Docker’s configuration after Kamal installs it on your remote server.
• Performing CI checks before Kamal begins the deployment process.
• Sending notifications after Kamal successfully deploys an application.

Kamal has different hooks that can fire off before building the Docker image for your application, before and after it deploys, and more. Sadly, Kamal does not have a post-setup hook that would have been perfect for this use case. Out of the existing hooks included with Kamal, the pre-deploy hook is the closest one that runs after the initial setup happens. As the name suggests, this hook runs just before the deployment step kicks off, making it a good spot to indicate that the setup process completed on the server previously.

What indicator can we use to mark a server as having its Kamal setup step done? The simplest way I found is to create an artifact on the remote server that we can use on the GitHub Actions workflow to check before deciding whether to run the setup or deploy command. If the artifact—in our case, a file—doesn’t exist on the server, we’ll assume the setup step hasn’t happened yet. Otherwise, we can assume that the presence of the file on the server means everything is in place for deployment.

Granted, this still isn’t a 100% bulletproof solution since the pre-deploy hook still runs just before Kamal boots up the application’s accessories for the first time during the setup process, and there’s a slight possibility of the hook firing off the script before that happens. However, I’ve been using Kamal for over a year, and this has not happened to me, so I think it’s a rare occurrence and an acceptable trade-off for this.

Setting up the pre-deploy Script

The files inside the .kamal/hooks directory are scripts interpreted by the system running the Kamal commands. When setting it up in the project, the samples generated by the kamal init command show various ways to use hooks, from basic shell scripts to fully-fledged Ruby scripts. To fire off these hooks properly, we’ll need to ensure the filename doesn’t have any extension attached to it and return a zero exit code. Any non-zero exit codes in a hook will abort the Kamal command.

Let’s begin setting up the pre-deploy script by using the sample that exists in our project. We’ll rename the file from .kamal/hooks/pre-deploy.sample to .kamal/hooks/pre-deploy, which Kamal expects. This file contains a Ruby script as an example to demonstrate how we can use this kind of hook. Let’s clear the entire contents of this file and create a very simple shell script with the following contents:

#!/bin/sh

if [ "$KAMAL_DESTINATION" = "review" ]; then
  ssh root@$REVIEW_APP_IP "echo '$KAMAL_VERSION' > ~/kamal_version"
fi

This script first checks if the KAMAL_DESTINATION environment variable has the value of “review”, so we can only run this pre-deploy script when deploying using the review destination we have set up in this project. Kamal will set up a few pre-defined environment variables like this one for our hooks (as we see in the following line of this script), so check the documentation for more details on what’s available.

If KAMAL_DESTINATION matches the review destination, it’ll connect to the remote server via SSH, which the system should already have access to since it will deploy the app. We intend to run this script on the GitHub Actions workflow we set up previously, so we’ll have the REVIEW_APP_IP environment variable set up with the IP address of our temporary server on the cloud. We also use Kamal’s default user for deployments (root) to connect to the server.

When the script connects to the server, it’ll use the pre-defined KAMAL_VERSION environment variable containing the commit SHA currently being deployed. It will set this value in a file called kamal_version in the root user’s home directory. This file will signal that the kamal setup command already happened on the remote server.

Next, we’ll need to update our continuous integration workflow to check the existence of this file and determine which command to run.

Updating Github Actions Workflow to Check Server State

I’ll open up the .github/workflows/review-app-deploy.yml file to modify the existing workflow that provisions the server on Hetzner Cloud and conditionally runs the kamal setup or kamal deploy command. The first thing I’ll do is remove the conditional steps set in the deploy job that is checking for the event type:

name: Review app deployment

on:
  pull_request:
    types:
      - opened
      - reopened
      - synchronize

jobs:
  provision:
    # Provision job configuration and steps omitted.

  deploy:
    # Deploy job configuration omitted.

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Ruby
        uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true

      - name: Configure SSH Key
        uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.REVIEW_APP_SSH_PRIVATE_KEY }}

      # Remove these two steps.
      - name: Kamal setup
        if: github.event.action == 'opened' || github.event.action == 'reopened'
        run: bundle exec kamal setup -d review

      - name: Kamal deploy
        if: github.event.action == 'synchronize'
        run: bundle exec kamal deploy -d review

After removing those two steps, I’ll replace them with two new steps:

name: Review app deployment

on:
  pull_request:
    types:
      - opened
      - reopened
      - synchronize

jobs:
  provision:
    # Provision job configuration and steps omitted.

  deploy:
    # Deploy job configuration omitted.

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Ruby
        uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true

      - name: Configure SSH Key
        uses: webfactory/ssh-agent@v0.9.0
        with:
          ssh-private-key: ${{ secrets.REVIEW_APP_SSH_PRIVATE_KEY }}

      # NEW: Set up review app server's host keys to known hosts
      - name: Add host key to known_hosts
        run: |
          mkdir -p -m 700 /home/runner/.ssh
          ssh-keyscan $REVIEW_APP_IP >> /home/runner/.ssh/known_hosts
          chmod 600 /home/runner/.ssh/known_hosts

      # NEW: Add command to check existence of ~/kamal_version on remote server.
      - name: Set up or deploy with Kamal
        run: |
          kamal_version_exists=$(ssh root@$REVIEW_APP_IP "if [ -s ~/kamal_version ]; then echo 1; else echo 0; fi")
          if [ "$kamal_version_exists" -eq 1 ]; then
            echo "Running kamal deploy..."
            bundle exec kamal deploy -d review
          else
            echo "Running kamal setup..."
            bundle exec kamal setup -d review
          fi

The first new step, “Add host key to known_hosts,” is necessary to make an SSH connection through the shell in a subsequent step. This step will create the /home/runner/.ssh directory with full permissions only for the current user, then run ssh-keyscan to collect the review app’s server public key fingerprints and place them in known_hosts, and finally make sure the file has read/write permissions only for the current user. The ubuntu-latest GitHub Actions runner uses /home/runner at the default user’s home directory when executing scripts.

We need to perform this step because when we connect to an SSH server for the first time from a new system, the ssh command will ask to confirm the remote server’s fingerprint. We’ll automate this process in GitHub Actions, so we don’t have the opportunity to respond to this prompt. The commands in this step set up the necessary file and contents to allow us to make SSH connections later without needing to accept any verification prompts.

The second step, “Set up or deploy with Kamal,” does two things. It’ll first connect to the server via SSH to check if the ~/kamal_version file exists and has content. This command is why we needed to set up the remote server’s key fingerprints in the previous step. After checking ~/kamal_version, we’ll assign a value of 0 or 1 to the kamal_version_exists variable. This variable will serve as a boolean value since the shell doesn’t have a dedicated boolean (e.g., true or false) data type.

After checking for our artifact on the remote server, the script will perform either kamal setup or kamal deploy with a quick message indicating which execution occurred, according to the value we stored in the kamal_version_exists variable. Now, we have a single step that will handle either scenario depending on the server’s state.

Testing Out the New Workflow

With these changes, I can commit them to a branch in the application’s repository and open a new pull request. We can see from the GitHub Actions logs that the updated steps in the deploy workflow ran successfully, and we can see the message indicating that the initial Kamal setup command ran.

If I add a new commit to the pull request’s branch and push it to the repository, GitHub Actions will fire off again. This time, the logs will show that a normal Kamal deployment happened this time, which is the expected action since the server has a valid ~/kamal_deploy file.

Our review app workflow now has a single step that lets us execute the correct command instead of being tied by the triggering event, knowing when the Kamal setup has been completed previously on the review application’s server once. We can now rest easy knowing that if some failure occurs before the job completes the initial setup, we can fix it in the same pull request instead of tearing everything down to recreate it.

Wrapping Up

We have made our review app workflow a bit more reliable in case of failure when setting up the new server using Kamal. This process isn’t a perfect solution, as there are still potential edge cases where the setup step doesn’t complete successfully, and attempting to run kamal setup again won’t work as expected. But as mentioned earlier, I’ve been using Kamal for over a year and haven’t had this issue yet. Still, keep in mind something can still go wrong with these updates.

Something else to keep in mind is that this process works great in our example scenario because we’re using a single Hetzner Cloud server to deploy the review app. If we need to spin up more than one server as part of our review app infrastructure, this process will get more complex as we need to verify the state of each server before deciding which Kamal command to run.

If you have suggestions for improving this workflow, contact me and share your tips. I’d love to improve what I have here. But at this point, we can consider our workflow complete and fairly reliable. I would recommend this process for teams looking to implement review application functionality outside of platforms like Heroku.

In my final article covering this topic, I’ll add a few extras to this workflow to make things faster and more convenient for anyone needed to build and access these review apps. Keep an eye out for that one coming soon.

Screencast

If this article or video helped you with automating your deployments on GitHub Actions, consider subscribing to my YouTube channel for similar videos containing tips on helping Rails developers ship their code with more confidence, from development to testing to deployment.