Review Apps With Kamal (Part 2): Configuring Kamal Destinations

In my previous article, I began setting up a web application that uses Kamal for deployments to create a review app. The intent for setting up this system is so that any time you or someone on your team opens a new pull or merge request, you’ll have an instance of your application running for verification purposes.

The first part of this series shows how you can use tools like Terraform and GitHub Actions to spin up new servers when opening a pull request and clean things up once finished. In this article, I’ll continue with that setup and begin setting up Kamal to allow deployments to different environments. In future articles, I’ll use this setup to deploy an application to the newly-provisioned infrastructure automatically and have a usable environment ready for testing.

Deploying to Different Environments Using Kamal

By default, Kamal looks for a configuration file in config/deploy.yml to perform deploys to your servers. This configuration works great when deploying to a single environment like production. However, many teams must also deploy their applications to separate environments, like a staging environment that mimics the production servers for testing purposes.

Kamal allows us to set up separate configuration files, or destinations, to set up deployments with different settings. The naming convention to use for the configuration file when using destinations is config/deploy.[destination].yml, where [destination] is your preferred destination name, like “staging” or “qa”. You can then tell Kamal to use this configuration when running any Kamal commands by using the -d flag (such as kamal deploy -d [destination] or kamal app logs -d [destination]).

In most cases, deploying an application to a test environment only requires a few changes in the configuration, like the IP addresses for the servers, different environment variables for configuration, and so on. When specifying a destination for deployment, Kamal treats the main config/deploy.yml file as the base configuration and will merge the destination’s configuration settings. That means most of the configuration used for our application can remain in one place, with the destination configuration file only containing the necessary overrides for that environment.

Setting up a New Kamal Destination for Review Environments

Let’s see how Kamal destinations work by setting one up in the TeamYap application used in this series. For this example, I’ll call my destination “review” since I’ll use this to spin up review apps. I’ll create a new file in the repo called config/deploy.review.yml.

I’ll add the settings I want to change from the base configuration in this file. When setting up a new destination, I typically like to copy the entirety of the base config/deploy.yml file, paste it into the new destination’s configuration file, and eliminate the settings that won’t change in my separate environment. Settings such as the service, the Docker image, and the container registry won’t change in my review destination, so I can eliminate them.

Other settings will need some partial changes. For instance, the server IP address for the web and job roles set up in the base configuration will need to change. We can override these values by keeping the keys in the review destination configuration file and setting up their new values.

We only need to change the values that need overriding from the base configuration so we can remove any settings within a root-level key like server that won’t change in the destination environment. For example, the job role will need a different host value, but the command to run the job process remains the same. I can remove the cmd setting under this role while keeping the host setting.

After going through the base configuration, the YAML keys in our config/deploy.review.yml file will look like this before setting up the overridden values:

servers:
  web:
    -
  job:
    hosts:
      -

proxy:
  host:

env:
  clear:
    APPLICATION_HOST:

accessories:
  db:
    host:

  redis:
    host:

Dynamically Setting up Kamal Configuration Settings

The configuration for the review destination will override the following:

• The IP address for the different servers used in the application.
• The hostname so Kamal Proxy can automatically generate an SSL certificate and route traffic to the application.
• The value of an environment variable that relies on the application’s hostname.

Since we’ll provision new servers for each review app, we’ll need to dynamically set the values of these settings. The simplest way to handle this is to set up environment variables in the system using the configuration file, which Kamal can pick up and use to merge into the base configuration on deployment.

We’ll set up everything in a single server instance for our example. The IP addresses will be the same for each server that needs these details, so I can set up an environment variable called REVIEW_APP_IP. We’ll also only have one hostname to set up for Kamal Proxy and the TeamYap application, which I’ll call REVIEW_APP_HOST.

Kamal configuration files can use ERB tags, so I’ll include these values in the destination configuration file:

servers:
  web:
    - <%= ENV["REVIEW_APP_IP"] %>
  job:
    hosts:
      - <%= ENV["REVIEW_APP_IP"] %>

proxy:
  host: <%= ENV["REVIEW_APP_HOST"] %>

env:
  clear:
    APPLICATION_HOST: <%= ENV["REVIEW_APP_HOST"] %>

accessories:
  db:
    host: <%= ENV["REVIEW_APP_IP"] %>

  redis:
    host: <%= ENV["REVIEW_APP_IP"] %>

This update covers the configuration needed to deploy the TeamYap application to a review environment. Before using this destination for deployments, we’ll need to handle any Kamal secrets that need overriding.

Setting up Secrets for Kamal Destinations

Recent versions of Kamal read the .kamal/secrets file for storing sensitive data like the Docker registry credentials, API keys, passwords, and any other information needed to deploy your application successfully. If you’re using Kamal to deploy without specifying a destination, it’ll use the values from this file when deploying and setting up your application.

However, Kamal does not grab the values set in .kamal/secrets when deploying to a destination. Instead, it first retrieves secrets from a file called .kamal/secrets.[destination], where [destination] is the name of your desired destination. You’ll need to set your destination secrets here, even if they’re exactly the same as those in .kamal/secrets. Otherwise, Kamal won’t have these details, and your deployments won’t work as expected.

If you use Kamal to deploy to multiple destinations, it can also merge shared secrets placed in the .kamal/secrets-common file. For this article, I’ll only deploy to a single destination called review, so I don’t need to use this file.

I’ll set up the secrets for the review destination by creating a file called .kamal/secrets.review. I’ll copy over all of the variables I have set up in my primary .kamal/secrets file that I use to deploy TeamYap to production. Although most of the values for my secrets will remain the same, like the container registry password, some will need changes depending on the environment. For example, my application’s database and Redis services expect an IP address set through an environment variable.

Once again, environment variables come to the rescue, similar to the destination configuration overrides done earlier. I’ll replace all of these values using environment variables of the same name, which I can configure later on GitHub Actions or other environments for testing.

One thing to keep in mind is that Kamal processes the contents of the secrets file through the shell, and it doesn’t use ERB like we did when setting up the config/deploy.review.yml file. In Bash, we can get the value of an environment variable with ${VARIABLE}, so I’ll set those values to receive them from the deployment environment:

KAMAL_REGISTRY_PASSWORD=${KAMAL_REGISTRY_PASSWORD}
RAILS_MASTER_KEY=${RAILS_MASTER_KEY}
POSTGRES_USER=${POSTGRES_USER}
POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
POSTGRES_DB=${POSTGRES_DB}
DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${REVIEW_APP_IP}:5432/${POSTGRES_DB}"
REDIS_URL="redis://${REVIEW_APP_IP}:6379/0"

Testing the New Kamal Destination

After setting up the configuration file and the secrets for the new review destination, Kamal can use it separately from our main production deployments. Let’s test it out manually before setting it up on GitHub Actions in the near future.

For a local test run, I’ll manually perform a couple of steps that GitHub Actions will handle automatically for us in the future:

• Spin up a new Hetzner Cloud server for this test deployment.
• Set up a DNS record to point to the test server (review.teamyap.app for this example).
• Export all the environment variables in my shell that Kamal needs for the destination deployment to work.

Since this is a brand-new server, I’ll need to perform the initial setup, so I’ll run kamal setup -d review. If I set everything up correctly, the initial setup should work as expected on the new server:

When visiting the URL set up for this test server, it should take me to a running application on the new server:

Wrapping Up

In this article, we laid the groundwork for using Kamal to deploy to different servers dynamically by creating a new destination with its own configuration to handle newly-provisioned environments. We also covered how setting up secrets for Kamal destinations differs from using non-destination deployments to keep this sensitive information separate per environment. This configuration will help us automate deployments and get our application up and running when opening pull requests on GitHub.

The next step I’ll cover in the following article will show how to update our existing GitHub Actions workflow that’s already provisioning infrastructure on the cloud to use this updated Kamal configuration to deploy a review app after automatically spinning up a new server. Stay tuned for part 3!

Screencast

If this article or video helped you understand how Kamal destinations work, consider subscribing to my YouTube channel for similar videos containing tips on helping Rails developers ship their code with more confidence, from development to deployment.