RidgeRun Yocto Developer Guide - Yocto SState Cache Server Builder Client

Table of Contents [Sticky]

Yocto Sstate-Cache Builder Client

In this wiki you'll learn how to consume a Yocto sstate-cache mirror to download pre-built artifacts instead of compiling from scratch, reducing build times from hours to minutes.

CI/CD Platform: this project uses GitLab CI/CD Pipelines as its automation engine. All build, publish, and cleanup steps are defined in .gitlab-ci.yml and executed by GitLab Runners.

Overview

The builder client performs a Yocto build that is configured to fetch cached artifacts from a remote sstate-cache mirror. It also pins layer versions to the exact commits that were used when the cache was generated, ensuring full reproducibility and maximum cache hit rates.

How It Works

builder_client (this repo)
   │
   ├── 1. trigger_builder stage → triggers builder repo to refresh the cache
   │
   ├── 2. build stage
   │     ├── Clone builder repo (contains CI scripts)
   │     ├── Clone layers
   │     ├── Checkout to cached layer versions (pinned commits)
   │     ├── Configure SSTATE_MIRRORS in local.conf
   │     └── bitbake core-image-base (downloads from mirror instead of compiling)
   │
   └── 3. clean stage → removes workspace

Prerequisites

A GitLab runner tagged yocto-sstate-cache with:
- Docker executor using the dchvs/yocto:scarthgap image.
- Read access to the cached versions directory at /mnt/fat/var/yocto/mirror/<release>/cached_versions/.
- The workspace directory /workdir/builder_client/ owned by ridgerun (UID 1000, GID 1000).
A populated sstate-cache mirror (generated by the builder project).
CI/CD variables DEPLOY_USER and DEPLOY_TOKEN configured in GitLab for cloning the builder repository.

Usage

1. Set CI/CD Variables

Configure these in GitLab → Settings → CI/CD → Variables:

Variable	Scope	Description
`DEPLOY_USER`	Protected	GitLab username with access to the builder repo.
`DEPLOY_TOKEN`	Protected/Masked	GitLab deploy token for the builder repo.

The following variables have sensible defaults in .gitlab-ci.yml (see Appendix A.1) but can be overridden:

Variable	Default	Description
`YOCTO_RELEASE`	`scarthgap`	Yocto release branch.
`WORKSPACE`	`/workdir/builder_client`	Working directory.
`LAYERS_DIR`	`/workdir/builder_client/layers`	Where layers are cloned.
`CACHED_VERSIONS_DIR`	`/mnt/fat/var/yocto/mirror/scarthgap/cached_versions`	Path to layer version snapshots from the builder.
`SSTATE_CACHE_DIR`	`/mnt/fat/var/yocto/mirror/scarthgap/sstate-cache`	Local path to the sstate-cache mirror.
`SSTATE_MIRRORS_URL_BASE`	`https://yocto_server/mirror`	Base URL for health checks.
`SSTATE_MIRRORS_URL`	`https://yocto_server/mirror/scarthgap/sstate-cache`	URL BitBake uses to download cached artifacts.

2. Trigger the Pipeline

The pipeline can be triggered in three ways:

Schedule (schedule): Set up a recurring schedule in GitLab → CI/CD → Schedules.
Downstream trigger (pipeline): Triggered by another project.
Manual (web): Run from the GitLab UI.

3. Pipeline Stages

Stage 1: `trigger`

Triggers the builder pipeline to regenerate the sstate-cache mirror with the latest layer sources. Uses strategy: depend, so this stage waits for the builder to finish successfully before proceeding.

Stage 2: `build`

Clones the builder repository (which contains the CI helper scripts: check_sstate_server.sh, verify_workspace_ownership.sh).
Validates the environment:
- check_sstate_server.sh — confirms the mirror server is reachable.
- verify_workspace_ownership.sh — ensures correct file ownership.
Clones all required Yocto layers at the YOCTO_RELEASE branch.
Pins layers to cached versions:
- Reads the most recent cached_versions_<release>_<date>.txt file.
- Checks out each layer to the exact commit hash recorded by the builder.
- This ensures the local layer state matches what was used to generate the cache, maximizing sstate hits.
Initializes the BitBake environment and registers all layers.

Appends the sstate-cache mirror configuration to local.conf:

#
# RidgeRun sstate-cache mirror
#
SSTATE_MIRRORS = "file://.* ${SSTATE_MIRRORS_URL}/PATH;downloadfilename=PATH"

Runs bitbake core-image-base for jetson-agx-orin-devkit and reports the sstate summary (cache hit/miss ratio).

CI helper scripts (check_sstate_server.sh, verify_workspace_ownership.sh) are provided by the builder repository, which is cloned at the start of the build stage.

Stage 3: `clean`

Removes the workspace directory. Runs regardless of build outcome (when: always).

4. Interpreting the Sstate Summary

The build output includes a line like:

Sstate summary: Wanted 1234 Found 1200 Missed 34 Current 0 (97% match, 3% missed)

Found: Artifacts successfully downloaded from the mirror.
Missed: Artifacts that had to be built locally (layer drift, new recipes, etc.).
A high match rate (>90%) indicates the cache is effective.

5. Adding a New Machine Target

Add another bitbake line in the build stage:

- MACHINE=jetson-agx-orin-devkit            bitbake core-image-base | grep "Sstate summary"
- MACHINE=<new-machine>                     bitbake core-image-base | grep "Sstate summary"

The new machine must also be built by the builder project for its artifacts to exist in the mirror.

6. Using the Mirror in a Local Build

You can use the same sstate-cache mirror outside of CI. Add this to your local.conf:

BB_SIGNATURE_HANDLER = "OEBasicHash"
BB_HASHSERVE = ""
SSTATE_MIRRORS = "file://.* https://yocto_server/mirror/scarthgap/sstate-cache/PATH;downloadfilename=PATH"

Then ensure your layers are checked out to the same commits listed in the latest cached_versions_*.txt file for maximum cache hits.

File Reference

File	Purpose
`.gitlab-ci.yml`	Main pipeline definition.