Jump to content

RidgeRun Yocto Developer Guide - Yocto SState Mirror Server Automation

From RidgeRun Developer Wiki

Follow us on: YouTube Twitter LinkedIn Email Share this page

Share This Page

RidgeRun Yocto Developer Guide logo
RidgeRun Yocto Developer Guide


NVIDIA partner logo NXP partner logo




Yocto Sstate-Cache Builder

In this wiki you'll learn how to setup a Yocto sstate-cache mirror server using GitLab CI/CD Pipelines to build images from scratch and publish the resulting artifacts. This enables downstream projects to reuse pre-built artifacts and dramatically reduce build times.

CI/CD Platform: this project uses GitLab CI/CD Pipelines as its automation engine. All build, publish, and cleanup steps are defined in .gitlab-ci.yml and executed by GitLab Runners.

Overview

The builder performs a full Yocto build for the configured machine targets, then syncs the generated sstate-cache/ directory to a centralized mirror served over HTTPS. It also records the exact Git commit hashes of every layer used, so that clients can reproduce the same build environment.

Architecture

builder (this repo)
   │
   ├── Full Yocto build (fetch + compile)
   │
   ├── rsync sstate-cache/ → /mnt/fat/var/yocto/mirror/<release>/sstate-cache/
   │
   └── Save layer versions → /mnt/fat/var/yocto/mirror/<release>/cached_versions/
                                        │
                                        ▼
                              HTTPS mirror server
                             (yocto.server/mirror)
                                        │
                                        ▼
                              builder_client (consumer)

Prerequisites

  • A GitLab runner tagged yocto-sstate-cache with:
    • Docker executor using the dchvs/yocto:scarthgap image (or matching your release).
    • A persistent volume mounted at /mnt/fat/var/yocto/mirror/ that is also served by an HTTPS web server (e.g., Nginx).
    • Sufficient disk space for a full Yocto build (~500 GB recommended).
    • The workspace directory /workdir/builder/ owned by $USER (UID 1000, GID 1000).
  • Network access to upstream layer repositories (git.yoctoproject.org, github.com, git.openembedded.org).

How to Generate the Sstate-Cache Mirror

1. Configure the GitLab Runner

A GitLab Runner must be registered and configured on the host machine. Refer to the official GitLab Runner configuration guide for setup instructions.

2. Configure the HTTPS Server

The runner host must expose the mirror directory (/mnt/fat/var/yocto/mirror/) via HTTPS so that clients can download cached artifacts. Setting up an HTTPS server (e.g., Nginx, Apache) is outside the scope of this documentation.

3. Set CI/CD Variables

Variable Default Description
YOCTO_RELEASE scarthgap Yocto release branch to build.
WORKSPACE /workdir/builder Working directory for the build.
LAYERS_DIR /workdir/builder/layers Directory where layers are cloned.
CACHED_VERSIONS_DIR /mnt/fat/var/yocto/mirror/scarthgap/cached_versions Where layer version snapshots are stored.
SSTATE_CACHE_DIR /mnt/fat/var/yocto/mirror/scarthgap/sstate-cache Mirror destination for sstate-cache artifacts.
SSTATE_MIRRORS_URL_BASE https://yocto.server/mirror Base URL of the mirror (used for health checks).
SSTATE_MIRRORS_URL https://yocto.server/mirror/scarthgap/sstate-cache Full URL to the sstate-cache directory.

4. Trigger the Pipeline

The pipeline accepts two trigger sources:

  • Downstream trigger (pipeline): Triggered by the builder_client project via the trigger_builder stage.
  • Manual (web): Run manually from the GitLab UI (CI/CD → Pipelines → Run pipeline).

5. What the Pipeline Does

  1. Validates the environment
    • Checks that at least 300 GB of disk space is available (check_disk_space.sh — see Appendix A.2). The threshold can be overridden via the REQUIRED_SPACE_GB variable.
    • Checks that the sstate-cache server is reachable (check_sstate_server.sh — see Appendix A.3).
    • Verifies workspace ownership matches the expected user/UID/GID (verify_workspace_ownership.sh — see Appendix A.5).
  2. Clones all required layers at the configured YOCTO_RELEASE branch:
    • poky, meta-openembedded, meta-tegra, meta-tegra-community, meta-virtualization, meta-selinux.
  3. Configures BitBake
    • Initializes the build environment via oe-init-build-env.
    • Registers all layers with bitbake-layers add-layer.

The following settings must be present in the project's build/conf/local.conf when creating the builder project: 

#
# Sstate-cache policy
#
BB_SIGNATURE_HANDLER = "OEBasicHash"
BB_HASHSERVE = ""

Optionally, you can also add pressure-control settings to prevent OOM and CPU overload: 

#
# Modulate build pressure to avoid OOM and CPU overload
#
BB_PRESSURE_MAX_CPU = "300"
BB_PRESSURE_MAX_IO = "200"
BB_PRESSURE_MAX_MEMORY = "200"
  1. Builds the image
    • Runs bitbake core-image-base --runall=fetch to populate source downloads.
    • Runs bitbake core-image-base to compile everything for jetson-agx-orin-devkit.
  2. Publishes the cache
    • Saves a timestamped file with each layer's Git hash to CACHED_VERSIONS_DIR (create_cached_versions_file.sh — see Appendix A.4).
    • Syncs the local sstate-cache/ to the mirror directory via rsync.
  3. Cleans up the workspace regardless of build outcome.

6. Adding a New Machine Target

Add another bitbake invocation in the build stage: 

- MACHINE=jetson-agx-orin-devkit            bitbake core-image-base --runall=fetch -q
- MACHINE=jetson-agx-orin-devkit            bitbake core-image-base -q
- MACHINE=<new-machine>                     bitbake core-image-base --runall=fetch -q
- MACHINE=<new-machine>                     bitbake core-image-base -q

7. Adding a New Layer

  1. Add the git clone command in the layers cloning section.
  2. Add the corresponding bitbake-layers add-layer command.
  3. Commit and push — the next pipeline run will include the new layer in the sstate-cache.

File Reference

File Purpose
.gitlab-ci.yml Main pipeline definition.
build/conf/local.conf BitBake configuration including sstate-cache policy and pressure settings.
.gitlab-ci/check_disk_space.sh Pre-flight check for available disk space (default: 300 GB).
.gitlab-ci/check_sstate_server.sh Validates the mirror server is reachable.
.gitlab-ci/create_cached_versions_file.sh Records Git hashes of all layers after a successful build.
.gitlab-ci/verify_workspace_ownership.sh Ensures workspace ownership matches expected UID/GID.

Troubleshooting

Symptom Cause Fix
ERROR: the sstate-cache server ... is not reachable Mirror web server is down or URL is wrong. Verify Nginx is running and SSTATE_MIRRORS_URL_BASE is correct.
Error: /workdir/builder is not owned by ridgerun Runner volume has wrong permissions. Run chown 1000:1000 /workdir/builder on the runner host.
ERROR: Not enough disk space Workspace partition has less than 300 GB free. Free up disk space or override with REQUIRED_SPACE_GB.
Build killed by OOM Pressure limits too high for the runner. Lower BB_PRESSURE_MAX_* values in build/conf/local.conf.
Pipeline timeout (>50h) Build is too large or runner is underpowered. Add more CPU/RAM or split into multiple machine builds.

Appendix A: Source Code Reference

A.1 .gitlab-ci.yml

image: dchvs/yocto:scarthgap

stages:
  - build
  - clean

default:
  tags:
    - yocto-sstate-cache

variables:
  USER: "ridgerun"
  UID: "1000"
  GID: "1000"
  YOCTO_RELEASE: "scarthgap"
  WORKSPACE: "/workdir/builder"
  LAYERS_DIR: "/workdir/builder/layers"
  CACHED_VERSIONS_DIR: "/mnt/fat/var/yocto/mirror/scarthgap/cached_versions"
  SSTATE_CACHE_DIR: "/mnt/fat/var/yocto/mirror/scarthgap/sstate-cache"
  SSTATE_MIRRORS_URL_BASE: "https://yocto_server/mirror"
  SSTATE_MIRRORS_URL: "https://yocto_server/mirror/scarthgap/sstate-cache"

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "pipeline"
    - if: $CI_PIPELINE_SOURCE == "web"

build:
  stage: build
  timeout: 50h
  script:
    - bash

    - mkdir -p $WORKSPACE
    - cd $WORKSPACE

    - git clone $CI_REPOSITORY_URL -b $YOCTO_RELEASE .

    - $WORKSPACE/.gitlab-ci/check_disk_space.sh
    - $WORKSPACE/.gitlab-ci/check_sstate_server.sh
    - $WORKSPACE/.gitlab-ci/verify_workspace_ownership.sh

    - mkdir -p $LAYERS_DIR/
    - cd $LAYERS_DIR/
    - git clone https://git.yoctoproject.org/poky -b $YOCTO_RELEASE
    - git clone https://git.openembedded.org/meta-openembedded -b $YOCTO_RELEASE
    - git clone https://github.com/OE4T/meta-tegra -b $YOCTO_RELEASE
    - git clone https://github.com/OE4T/meta-tegra-community -b $YOCTO_RELEASE
    - git clone https://git.yoctoproject.org/meta-virtualization -b $YOCTO_RELEASE
    - git clone https://git.yoctoproject.org/meta-selinux -b $YOCTO_RELEASE

    - cd $WORKSPACE
    - . $LAYERS_DIR/poky/oe-init-build-env $WORKSPACE/build/


    - bitbake-layers add-layer $LAYERS_DIR/poky/meta
    - bitbake-layers add-layer $LAYERS_DIR/poky/meta-poky
    - bitbake-layers add-layer $LAYERS_DIR/poky/meta-yocto-bsp
    - bitbake-layers add-layer $LAYERS_DIR/poky/meta-skeleton
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-oe
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-python
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-networking
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-gnome
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-multimedia
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-xfce
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-initramfs
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-perl
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-filesystems
    - bitbake-layers add-layer $LAYERS_DIR/meta-openembedded/meta-webserver
    - bitbake-layers add-layer $LAYERS_DIR/meta-virtualization

    - bitbake-layers add-layer $LAYERS_DIR/meta-tegra
    - bitbake-layers add-layer $LAYERS_DIR/meta-tegra-community

    - MACHINE=jetson-agx-orin-devkit            bitbake core-image-base --runall=fetch -q
    - MACHINE=jetson-agx-orin-devkit            bitbake core-image-base -q

    - echo "Create the cached versions file"
    - $WORKSPACE/.gitlab-ci/create_cached_versions_file.sh

    - echo "Sync the sstate-cache to the mirror"
    - rsync -La --info=NAME $WORKSPACE/build/sstate-cache/ $SSTATE_CACHE_DIR/

    - echo "Build succeded"

cleanup_task:
  stage: clean
  when: always
  script:
    - |
      if [ -n "$WORKSPACE" ]; then
        rm -rf $WORKSPACE/
      fi

A.2 check_disk_space.sh

#!/bin/bash

set -e

REQUIRED_SPACE_GB=${REQUIRED_SPACE_GB:-300}

AVAILABLE_GB=$(python3 -c "import shutil; print(shutil.disk_usage('$WORKSPACE').free // (1024 ** 3))")

if [ "$AVAILABLE_GB" -lt "$REQUIRED_SPACE_GB" ]; then
  echo "ERROR: Not enough disk space in $WORKSPACE. Available: ${AVAILABLE_GB}G, Required: ${REQUIRED_SPACE_GB}G"
  exit 1
fi

echo "INFO: Disk space OK. Available: ${AVAILABLE_GB}G (required: ${REQUIRED_SPACE_GB}G)"

A.3 check_sstate_server.sh

#!/bin/bash

set -e

if curl -sSf --retry 3 --retry-delay 2 -o /dev/null "$SSTATE_MIRRORS_URL_BASE"; then
  echo "INFO: the sstate-cache server is up"
else
  echo "ERROR: the sstate-cache server $SSTATE_MIRRORS_URL_BASE is not reachable"
  exit 1
fi

A.4 create_cached_versions_file.sh

#!/bin/bash

mkdir -p $CACHED_VERSIONS_DIR/

for layer in "$LAYERS_DIR"/*/; do
  name=$(basename "$layer")
  hash=$(git -C "$layer" rev-parse "$YOCTO_RELEASE" 2>/dev/null) && \
    echo "$name: $hash"
done > "$CACHED_VERSIONS_DIR/cached_versions_${YOCTO_RELEASE}_$(date +"%m-%d-%Y").txt"

A.5 verify_workspace_ownership.sh

#!/bin/bash

OWNER_UID=$(stat -c '%u' "$WORKSPACE")
OWNER_GID=$(stat -c '%g' "$WORKSPACE")
OWNER_NAME=$(stat -c '%U' "$WORKSPACE")

if [ "$OWNER_NAME" != "$USER" ] || [ "$OWNER_UID" != "$UID" ] || [ "$OWNER_GID" != "$GID" ]; then
  echo "Error: $WORKSPACE is not owned by $USER ($UID:$GID). Found $OWNER_NAME ($OWNER_UID:$OWNER_GID)"
  exit 1
fi



RidgeRun Yocto Developer Guide logo
Back to top ↑


Retrieved from "https://developer.ridgerun.com/wiki/index.php?title=RidgeRun_Yocto_Developer_Guide/Yocto_SState_Cache_Mirror_Server_Automation/Yocto_SState_Cache_Server_Builder&oldid=79743"
Cookies help us deliver our services. By using our services, you agree to our use of cookies.