Using NixOS with CI/CD

May 17th, 2020

Nix, NixOS, and the surrounding projects are a great set of tools that can be used to manage environments and machines declaratively.

I wanted to use NixOS to manage all my VMs declaratively. There are two parts: deploying the NixOS virtual machine and keeping it up to date.

Deploying a NixOS virtual machine

The files for this section are in this repository. I use packer to create a virtual machine and then convert it into a template. The general steps are as follows.

Create a NixOS virtual machine

This is done however you want to do it. I use Packer and the vSphere provisioner to create a new virtual machine that boots the NixOS minimal install.
Create the partitions

The general partitioning instructions from the manual can be followed, with some tweaks. Instead of creating the swap at the end of the disk and the root partition sandwiched in between, I the swap near the beginning of the disk. The commands that packer runs for this step listed here.

NixOS has a configuration option called boot.growPartition which will grow the root partition on boot. By placing the root partition last, this option allows the root partition to automatically grow. Combined with the autoResize key set to true in the filesystem configuration, I can increase the size of the root partition in vSphere, and NixOS will take care of everything else.

This is very important for a template, as it allows the resulting VM to have an arbitrary amount of storage.
Configure the machine

I use this config file that I created by hand using nixos-generate-config and other customizations. It uses labels to mount filesystems so that it can apply generally to all the virtual machines (and generally helps clarity).

Additionally, it creates a deploy user and gives it passwordless sudo access and adds it to the group of trusted users. This user essentially has the permissions of the root user, so it is important to keep the corresponding ssh key safe. This user is important when using CI/CD.

Lastly, it adds the virtual machine to a private ZeroTier network, a private mesh network that spans all my virtual machines and my desktop. Once the machine is built with nixos-rebuild switch, sudo rm -r /var/lib/zerotier-one/ must be run to clear the identity of the ZeroTier client so that it is not carried over to the VMs that are created from the template.
Convert it to a template

Packer does this for me.

Creating a virtual machine simply involves deploying the template and authorizing it to connect to ZeroTier in the ZeroTier web UI. I wrote a program that gives each ZeroTier node a url to make my life easier. The code is in a git repository but there's not much documentation at the moment.

Configuring NixOS with CI/CD

I use Github Actions due to its integration with Github and support for self hosted runners, although any other platform can be used with some tweaking.

I created a self-hosted runner using the instructions provided by Github on an ubuntu virtual machine. I installed nix, jq, docker, and node, although the last two can be skipped if you are not using docker or javascript based actions (this action is simply a shell script but I also have other actions running).

I wrote the following script to deploy the configurations to my NixOS virtual machines, however, NixOps can also be used. NixOps is a better solution if this is going to be used in production, but I wanted to learn more about Nix so I wrote my own naive implementation. It is mostly based off the steps provided in this post.

#!/bin/bash
set -e

if [ $# != 1 ]; then
    echo "Incorrect number of arguments."
    echo "Use as ./nix-deploy.sh path-to-hosts"
    exit 1
fi

for host in $(cat $1 | jq -r '.[].name'); do

    mkdir -p build/${host}
    cd build/${host}

    /nix/var/nix/profiles/default/bin/nix-build --attr system -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/885a6658073f6c5e9a4b37f131ce870a41a4ce7d.tar.gz ../../${host}.nix;

    /nix/var/nix/profiles/default/bin/nix-copy-closure --to --gzip "deploy@${host}.zt.example.com" ./result;

    ssh "deploy@${host}.zt.example.com" sudo $(realpath result/bin/switch-to-configuration) switch;
    ssh "deploy@${host}.zt.example.com" sudo nix-env --profile /nix/var/nix/profiles/system --set $(realpath result);

    cd ../../

done;

The hosts are defined in a JSON file that currently defined as follows, but extra keys and configuration can easily be added.

[
    {
        "name": "host1"
    },
    {
        "name": "host2"
    }
]

The files for each virtual machine are similar to the following and are standard nix files.

import <nixpkgs/nixos> {
  system = "x86_64-linux";

  configuration = {
    imports = [
      ./configuration.nix
    ];

    networking.hostName = "host1";

  };
}

How the script works

For each item in the JSON array, the name key is read and assigned to the variable host, and the following steps are run.

Switch to a unique temporary directory.
Build the NixOS config. It searches for a file called ${host}.nix, where ${host} is defined as above.

The -I flag is used to pin nixpkgs, mostly because I don't understand nix enough to have a better way to pin nixpkgs, however, it works well enough.
The built config is copied to the NixOS VM using nix-copy-closure using ssh. I use an automatically generated domain using ZeroTier, but the IP address could potentially be defined in the JSON file. The ssh key that was added to the template lives on the runner.
The NixOS VM is switched to the new configuration. Inside the result symlink created by the build step, there is a switch that can be called. As the real path of the symlink is the same on the CI/CD machine and the NixOS VM due to the magic of Nix, making life quite easy.

Using everything with GitHub Actions

I use the following workflow definition. The workflow is run on a self-hosted runner and only run when anything in the nixos folder is changed. All my nix configuration files and the NixOS deploy script live in this folder.

name: Nixos Deploy

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
  push:
    branches: [ prod ]
    paths:
      - 'nixos/**'

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "deploy"
  deploy:
    # The type of runner that the job will run on
    runs-on: [self-hosted, nix] 

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2

    # Runs a set of commands using the runners shell
    - name: Build Nix Machine
      run: |
        cd $GITHUB_WORKSPACE/nixos
        ./nix-deploy.sh hosts.json

Adit Sachde

Using NixOS with CI/CD

Deploying a NixOS virtual machine

Configuring NixOS with CI/CD

How the script works

Using everything with GitHub Actions