Building images

Get the source

Everything below is in the site’s public repository: the Dockerfile and its pinned versions under lab-image/, and the workflow under .github/workflows/. Clone it and work from its root, because every command here names lab-image as a relative path. You need Docker with BuildKit, which is the default builder in Docker Desktop and in Docker Engine 23 and later; the Dockerfile uses RUN --mount and COPY --chmod, which need it. You also need the buildx plugin for the docker buildx imagetools command below. Docker Desktop includes it; on Linux, Docker’s packages ship it separately as docker-buildx-plugin.

git clone https://github.com/HybridCloudWorks/HCW-HybridCloudWorks.git; cd HCW-HybridCloudWorks

git clone https://github.com/HybridCloudWorks/HCW-HybridCloudWorks.git && cd HCW-HybridCloudWorks

One Dockerfile, two targets

The file has five stages. Three of them, fetch, vendor and mirror, only build: they download, verify and prepare, and nothing from them ships except what a later stage copies out. The other two are the images people use:

• runner carries terraform, kubeconform, helm and ansible-core, the Terraform provider mirror and the vendored modules. It starts from the base image again and copies in only what the build stages produced, so curl, unzip and git, which fetch installs to do its work, are not in it.
• full is FROM runner plus the Azure CLI, kubectl, git, curl and jq, and the packages VS Code in the browser needs. Its default command is bash. This is the image the labs run and the one you pull.

--target picks which stage the build stops at, and BuildKit builds only the stages that stage depends on. This builds full and tags it locally; --target runner builds the smaller image the same way. The image is linux/amd64 only, because every binary the Dockerfile downloads is the linux_amd64 build, so on a Mac with Apple silicon add --platform linux/amd64, as in the third line, and the build runs under emulation.

docker build --target full -t hcw-lab:dev lab-image

docker build --target full -t hcw-lab:dev lab-image

docker build --platform linux/amd64 --target full -t hcw-lab:dev lab-image

Docker’s own guide to the pattern is Multi-stage builds (opens in a new tab).

Base images pinned by digest

Both stages that start from an image, fetch and runner, start from the official python:3.14.7-slim-trixie (CPython 3.14.7 on Debian 13), written as the tag, an @, and the image’s index digest. A tag can be moved to a different image at any time; a digest is the hash of the content, so it cannot. As the Dockerfile’s own header puts it, the tag beside the digest is documentation: Docker resolves the digest and ignores the tag. The other three stages start from an earlier stage, which this lists alongside them:

Select-String -Path lab-image/Dockerfile -Pattern '^FROM'

grep -n '^FROM' lab-image/Dockerfile

A FROM line cannot read a file, so the digest is written twice: in the Dockerfile, and as BASE_IMAGE and BASE_DIGEST in versions.env (opens in a new tab). The publish workflow’s first step reads every FROM and fails the build unless each one is an earlier stage or exactly that image and digest, tag included. A bare tag, a different digest or a --platform flag on the line all fail it.

To see what the tag points at today, ask the registry. The Digest line it prints is the value in the FROM lines unless Docker has rebuilt the tag since the pin was last moved, and if the two differ, the pin is doing its job: the build keeps the image it was tested with until someone moves it on purpose.

docker buildx imagetools inspect python:3.14.7-slim-trixie

docker buildx imagetools inspect python:3.14.7-slim-trixie

Every direct download checked against a pinned sum

Every version and checksum the Dockerfile pins lives in versions.env, which each step that downloads something sources before it starts. Each binary is checked with sha256sum -c straight after its download, and because the shell runs with set -eu, a sum that does not match fails the build rather than leaving an unverified binary in place. ansible-core and each package it depends on are fetched by pip in hash-checking mode with --only-binary=:all:, so nothing is built from source and nothing outside the list can be pulled in, and the runner stage installs them a second time from those same files with no package index at all. pip is then removed, so the image carries no pip command.

Packages installed with apt-get are the exception. Apart from the Azure CLI, whose version is pinned, their versions are not in versions.env, and they are checked by APT against the signed Debian and Microsoft package indexes rather than by sha256sum -c. The Microsoft signing key itself is checked against a pinned sum before APT is told to trust it.

Vendored for offline use

The lab’s own checks run with no network, so terraform init has to work without one. Two things make that possible.

• A provider mirror. The mirror stage runs terraform providers mirror for azurerm (at a 5.x and a 4.x version), azapi, alz, random, modtm and time, checks each provider’s zip against its pinned sum, and then unpacks it. Unpacked matters: from a packed mirror terraform init copies each provider into .terraform (225 MB for azurerm), while from an unpacked one it links to the mirror and copies nothing, which leaves a 40 KB .terraform in the repository’s own measurement. A CLI config file then makes the mirror the only source, with no fallback to the registry, so an unmirrored provider fails at once instead of hanging on a registry it cannot reach.
• The Azure Verified Modules. Terraform has a provider mirror but no module mirror. So the image carries every Azure Verified Module the Landing Zone Builder emits, from each module’s release tarball and checked by SHA256, and every registry module those call. A script rewrites each call inside the copies from its registry source to a relative path, and a second terraform get proves nothing left in the tree reaches the registry.

The mirror is an ordinary directory, so you can look at it. This prints the two provider namespaces it holds, azure and hashicorp:

docker run --rm hcw-lab:dev ls /opt/terraform/mirror/registry.terraform.io

MSYS_NO_PATHCONV=1 docker run --rm hcw-lab:dev ls /opt/terraform/mirror/registry.terraform.io

Inside the container, TF_CLI_CONFIG_FILE=/dev/null terraform init sets the config aside for one command and uses the registry instead, which needs the network.

Running as a non-root user

Both images end with USER 65534:65534, Debian’s nobody, and WORKDIR /workspace. The lab mounts /workspace read-only, so everything the tools write goes under /tmp: HOME is /tmp/home, and TMPDIR, Terraform’s data directory, and Helm’s and Ansible’s working files are all under /tmp/run. In the runner image nobody keeps its no-login shell. The full image switches back to root only to install its packages and to give nobody a bash shell and that home, because the browser workspaces run their terminal through the user’s own shell, and then drops back to 65534.

docker run --rm hcw-lab:dev id

docker run --rm hcw-lab:dev id

The output begins uid=65534(nobody).

Smoke tests

A build that succeeds proves the files arrived, not that the tools work. smoke.sh (opens in a new tab) runs inside the image with lab-image mounted read-only and the network off, and it first refuses to pass if the network is reachable or the mount is writable, because an offline terraform init that had a network would prove nothing. Then it compares each tool’s version with versions.env, checks every mirrored provider and vendored module is on disk, and runs terraform init -backend=false and terraform validate against small configurations with no network. For full it also checks the extra tools and the user’s shell. From the repository root, after the build above:

docker run --rm --network none -v "${PWD}\lab-image:/workspace:ro" hcw-lab:dev bash /workspace/smoke.sh full

MSYS_NO_PATHCONV=1 docker run --rm --network none -v "$(pwd -W 2>/dev/null || pwd)/lab-image:/workspace:ro" hcw-lab:dev bash /workspace/smoke.sh full

A passing run ends with smoke: passed (full) and exits 0. A failing check prints FAIL: with the tool’s output under it, and the script runs every remaining check before it exits 1, so one run shows everything that is wrong. The bash line asks Git Bash for a Windows-style path with pwd -W and falls back to pwd anywhere else.

Provenance attestations

A provenance attestation is a signed record of where an image came from: which repository, which workflow file, which commit. The publish workflow makes one for each image it pushes with actions/attest-build-provenance (opens in a new tab), given the image’s name and digest and push-to-registry: true, so the attestation is stored in the registry beside the image. Only the jobs that publish are granted id-token: write and attestations: write, which the action needs to sign and store it.

The builds themselves set provenance: false, which turns off BuildKit’s own attestation. That keeps each pushed image a plain single manifest, whose digest is the one docker pull reports, while the record of how it was made comes from the action instead. You can check it with the GitHub CLI, signed in:

gh attestation verify oci://ghcr.io/hybridcloudworks/hcw-lab:latest --repo HybridCloudWorks/HCW-HybridCloudWorks

gh attestation verify oci://ghcr.io/hybridcloudworks/hcw-lab:latest --repo HybridCloudWorks/HCW-HybridCloudWorks

A good result prints ✓ Verification succeeded! and names .github/workflows/publish-lab-image.yml@refs/heads/main as the workflow that built it.

Publishing to GitHub Container Registry

publish-lab-image.yml (opens in a new tab) has three jobs. The first two keep validation and publishing apart, so the code a pull request can change never runs with a token that can publish.

• build runs on every pull request that touches the image, with a read-only token. It checks the FROM lines, builds both targets, and smoke-tests each one.
• publish runs only after build passes, and only on main. It builds both targets again, smoke-tests what it built, and pushes those exact images with docker push, tagged with the commit’s SHA and latest. It reads each digest back from the registry, attests it, and writes the digests to the run’s summary. This job alone holds packages: write, and it signs in to the registry with the run’s own token, so no password is stored anywhere.

The result is ghcr.io/hybridcloudworks/hcw-lab, public, so pulling it needs no sign-in. The third job, publish-dockerhub, runs after publish when Docker Hub publishing is switched on, and copies the same images there by digest, which is where the lab pages’ own docker run line pulls from, and the bytes are the same in both.

docker pull ghcr.io/hybridcloudworks/hcw-lab:latest

docker pull ghcr.io/hybridcloudworks/hcw-lab:latest