Skip to main content

🤖 CI/CD

Requirements​

warning

This will nest a full Ludus install for every pipeline in an existing Ludus server. Going more than 1 layer deep of nested virtualization is not supported.

To set up a CI/CD runner for Ludus development you must meet the following requirements:

  1. A functional, fast, Ludus server with at least 32GB of free RAM, 250GB of free disk space, and 8 cores available (can over-provision cores if necessary)
  2. The debian-13-x64-server-template must be built
  3. Root access to the Ludus server
  4. A Gitlab account with the ability to create a runner token (gitlab.com or self-hosted)
  5. Network access from the Ludus server to the Gitlab instance/gitlab.com

Setup​

To setup the CI/CD runner and template follow these steps:

  1. Create a Gitlab runner with the tag ludus-proxmox-runner. Do not check Run untagged jobs.

  2. Copy the Gitlab runner token

  3. Review the settings in /opt/ludus/ci/setup.sh to ensure they match your environment (i.e. PROXMOX_VM_STORAGE_POOL)

  4. Run /opt/ludus/ci/setup.sh with appropriate env variables as root on the Ludus server:

PROXMOX_USERNAME=root@pam PROXMOX_PASSWORD=password /opt/ludus/ci/setup.sh

  1. When the playbook finishes running, you will see a debian-12-x64-server-ludus-ci-template template in the Proxmox web UI and ludus templates list (admins only).

  2. Review the settings in /opt/ludus/ci/base.sh, specifically the PROXMOX_NODE setting and modify it as necessary.

Now that CI is setup and configured, any commits that are pushed to the Ludus project will build and test as appropriate.

Tags​

The CI system is set up to run the appropriate tests depending on what part of the code base has been modified. However, sometimes you want to override the defaults. To manually control the CI pipeline, you can add "tags" to the final commit message before a push. To use these, simply include one or more of the "tag" strings in your commit message, including the brackets.

The available tags are listed below:

  • [skip ci] - this tag skips all CI jobs
  • [skip build] - skips the documentation build and the binary build stages
  • [build docs] - force a documentation build
  • [build pages] - force a documentation build and pages deploy
  • [full build] - run every step of the CI pipeline, no matter how small the change to the code base
  • [manual] - only run the documentation build (if docs have changed) and binary build, then push the code to an already running CI VM (typically used with the [VMID-XYZ] tag, defaults to the runner with least uptime)
  • [VMID-XYZ] - run jobs on the specified VM where XYZ is the numeric VMID of the CI/CD VM.
  • [client tests] - test basic client commands that do not deploy templates or ranges
  • [template tests] - run a template build and wait for all templates to complete building
  • [range tests] - run a range deploy and wait for it to succeed. This uses the simple-domain.yml range config.
  • [post-deploy tests] - runs all tests related post-deployment tasks (testing mode, allowing and denying domains and IPs, powering on and off a VM, adding and removing users, etc)
  • [testing-mode tests] - runs tests to determine if testing mode functions properly
  • [ansible tests] - runs tests to determine if ansible features function properly
  • [user tests] - runs tests to determine if the functions related to user management function properly
  • [template tests] - runs tests related to adding, building, and removing custom templates
  • [start-at templates] - run all test starting at the template builds
  • [start-at range-admin] - run all test starting at the deployment of the admin user range
  • [start-at post-deploy-admin] - run all test starting after the admin range has deployed
  • [start-at range-user] - run all test starting at the deployment of the standard user range
  • [start-at post-deploy-user] - run all test starting after the deployment of the standard user range
  • [start-at integration] - just run the final integration test

Releases​

Any time a version tag is created in Gitlab, two additional CI jobs are added to the pipeline: upload and release. These jobs are manually triggered (you must click the play button in the pipeline) and upload the compiled binaries to the package registry as well as create the actual release. If you use conventional commits (perhaps created with koji), then git-cliff will automatically generate a change log for the release.

Manual CI VM Setup​

Run these commands on a Debian 13 VM, then power it off and save it as a template

hostname ludus-ci-debian-13

# Resize the disk by hand if needed (should be ~250GB)
fdisk /dev/vda1
p
d
n
1
2048
[End]
N
w

resize2fs /dev/vda1

# Install Go
apt install curl wget ca-certificates
wget https://go.dev/dl/go1.25.1.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.25.1.linux-amd64.tar.gz
# This is required since gitlab-runner ignores .bashrc
echo 'export PATH=$PATH:/usr/local/go/bin' >> /etc/profile


# Add the gitlab-runner user and allow them to sudo
useradd -m -s /bin/bash -c "Gitlab Runner" gitlab-runner 
# This breaks gitlab-runner, remove it
rm /home/gitlab-runner/.bash_logout
echo 'gitlab-runner ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers

# Install needed components
curl -L 'https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh' | bash
curl -s 'https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh' | bash
apt install gitlab-runner git git-lfs build-essential vim tmux htop jq python3-debian

# Install node/yarn for documentation building
curl -fsSL https://deb.nodesource.com/setup_21.x | bash - && apt-get install -y nodejs
npm install --global yarn

# Helpful to auto-load the key on login for troubleshooting
echo 'if [ -f /opt/ludus/ci/.apikey-admin ]; then export LUDUS_API_KEY=$(cat /opt/ludus/ci/.apikey-admin); fi' >> /home/gitlab-runner/.bashrc
echo 'if [ -f /opt/ludus/ci/.apikey-user ]; then export LUDUS_API_KEY=$(cat /opt/ludus/ci/.apikey-user); fi' >> /home/gitlab-runner/.bashrc

# Warm the caches
su gitlab-runner -
cd /tmp
git clone https://gitlab.com/badsectorlabs/ludus

cd ludus/ludus-server
GOOS=linux GOARCH=amd64 go build -trimpath -ldflags "-s -w"

cd ../ludus-client
git clone https://github.com/zimeg/spinner
cd spinner && git checkout unhide-interrupts && cd .. && go mod edit -replace github.com/briandowns/spinner=./spinner
GOOS=linux GOARCH=amd64 go build -trimpath -ldflags "-s -w"
GOOS=linux GOARCH=arm64 go build -trimpath -ldflags "-s -w"
GOOS=darwin GOARCH=amd64 go build -trimpath -ldflags "-s -w"
GOOS=darwin GOARCH=arm64 go build -trimpath -ldflags "-s -w"
# The forked spinner library doesn't compile for windows, so switch back to the original
go mod edit -dropreplace=github.com/briandowns/spinner
GOOS=windows GOARCH=amd64 go build -trimpath -ldflags "-s -w"
GOOS=windows GOARCH=386 go build -trimpath -ldflags "-s -w"
GOOS=windows GOARCH=arm64 go build -trimpath -ldflags "-s -w"

cd ../docs
yarn install
yarn build