RidgeRun Yocto Developer Guide - Setting Up Mirrors in Yocto

From RidgeRun Developer Wiki

Follow Us On Twitter LinkedIn Email Share this page





In this guide, you’ll learn how to create a base build that can be used to retrieve sources and generate mirrors for both downloads/ and sstate-cache/. These mirrors are especially useful for setting up shared development environments, CI pipelines, or local acceleration of builds.

Steps

1. Setup a Yocto Build

Start by setting up a new build directory with your preferred configuration. Ensure that the build includes the supported BSP layers (e.g., MetaTegra in this case). Adjust your bblayers.conf and local.conf files as needed to define your target machine, distro, and any project-specific options.

2. Fetch The Sources and Build The Image

Once your build configuration is ready, initiate a build process. This step fetches and compiles the sources for all packages across the included layers. By doing so, you ensure that the required source archives and prebuilt artifacts are stored in your downloads/ and sstate-cache/ directories.

3. Prepare the Mirrors

After the build completes successfully:

Copy the downloads/ folder to your file hosting server to serve as your source mirror.

Copy the sstate-cache/ folder to the server to serve as your shared state mirror.

These mirrors can then be reused by other developers or CI jobs to speed up future builds and reduce redundant downloads.


Setup a Yocto Build

Get the Layers

# Navigate to the workspace directory
WORKSPACE=$PWD

YOCTO_RELEASE=scarthgap
LAYERS_DIR=$WORKSPACE/layers/

mkdir -p $LAYERS_DIR/
cd $_
git clone https://git.yoctoproject.org/poky -b $YOCTO_RELEASE
git clone https://git.yoctoproject.org/poky -b $YOCTO_RELEASE
git clone https://git.openembedded.org/meta-openembedded -b $YOCTO_RELEASE
git clone https://github.com/OE4T/meta-tegra -b $YOCTO_RELEASE
git clone https://github.com/OE4T/meta-tegra-community -b $YOCTO_RELEASE
git clone https://git.yoctoproject.org/meta-virtualization -b $YOCTO_RELEASE
git clone https://git.yoctoproject.org/meta-selinux -b $YOCTO_RELEASE
cd -

Pull Docker Image For Building

DOCKER_IMAGE_NAME=scarthgap

docker pull dchvs/yocto:$DOCKER_IMAGE_NAME

docker run \
    --rm \
    -it \
    --privileged \
    --network host \
    -e USER=$USER \
    -e UID=$(id -u) \
    -e GID=$(id -g) \
    -v $PWD:/workdir \
    -v $HOME/.gitconfig:$HOME/.gitconfig \
    -v $HOME/.ssh:$HOME/.ssh \
    $DOCKER_IMAGE_NAME \
    bash 

. layers/poky/oe-init-build-env build/

Add The Layers to Build Environment

bitbake-layers add-layer $OLDPWD/layers/poky/meta
bitbake-layers add-layer $OLDPWD/layers/poky/meta-poky
bitbake-layers add-layer $OLDPWD/layers/poky/meta-yocto-bsp
bitbake-layers add-layer $OLDPWD/layers/poky/meta-skeleton
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-oe
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-python
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-networking
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-gnome
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-multimedia
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-xfce
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-initramfs
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-perl
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-filesystems
bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-webserver
bitbake-layers add-layer $OLDPWD/layers/meta-virtualization

bitbake-layers add-layer $OLDPWD/layers/meta-tegra
bitbake-layers add-layer $OLDPWD/layers/meta-tegra-community
Add the appends. Refer to this document's Appends section.

Make sure you add the following configuration to your project, it's important for proper mirror generation:


BB_SIGNATURE_HANDLER = "OEBasicHash"

BB_HASHSERVE = ""


BB_GENERATE_MIRROR_TARBALLS = "1"


Proceed with the next step to build the image for generating the caches. Otherwise, you can jump to RidgeRun Yocto Developer Guide/Setting Up Mirrors in Yocto#Setup SState-Cache Mirror —User Mode, in case your setting up the mirrors in user mode.

Setup SState-Cache/Downloads Mirror —Developer Mode

Fetch The Sources and Build The Image For The Target Machines

Generate Sources For the Mirror - The downloads/ Directory

The sources mirror provides access to the tarballs or Git bare repositories of the project sources. This setup is safe for production use, and it is the recommended option for fetching dependencies during production builds.

Run Bitbake World with a runall fetch for all the desired machines across the supported BSPs: 

# QEMU machines
MACHINE=qemux86-64 bitbake world --runall=fetch -k
MACHINE=qemuarm64  bitbake world --runall=fetch -k

# MetaTegra BSP machine(s)
MACHINE=jetson-agx-orin-devkit            bitbake world --runall=fetch -k

After running the BitBake world fetch, it’s not uncommon to encounter errors in some recipes. The -k flag instructs BitBake to continue processing as much as possible, even if certain recipes fail. Despite these errors, the shared cache will still achieve substantial coverage, ensuring that most fetched sources and metadata remain available for future builds.

Generate SState Cache For the Mirror - The sstate-cache/ Directory

The SState Cache mirror provides access to prebuilt task artifacts generated by Yocto during previous builds. By reusing these cached results, subsequent builds can skip recompiling unchanged components, dramatically reducing build times.

This setup is the recommended approach when you want to accelerate builds across different machines or supported BSPs, for example for developing.

In order to generate this cache, run the following Bitbake World command across all the supported BSPs: 

# QEMU machines
MACHINE=qemux86-64 bitbake world -k
MACHINE=qemuarm64  bitbake world -k

# MetaTegra BSP machines
MACHINE=jetson-agx-orin-devkit            bitbake world -k

The previous Bitbake World command downloads and caches all the packages provided by the layers, creating a general layer-level cache. Depending on your use case, you may instead run Bitbake with your specific target image(s) rather than world. This approach gives you a more focused and end-to-end build cache tailored to your target images. Just remember to keep your cache up to date, since package configurations may have changed due to customizations introduced by your platform build.


Note: it's plausible that not all recipes in a World build may compile successfully, which means the previous commands could encounter errors. The -k flag ensures the build continues as far as possible, even after failures. For example, if 10–15 recipes fail and are not required for your workflow, i.e., your build never builds them, you can still consider the resulting cache a success and proceed to copy it to your mirror.


If a recipe's configuration changes later, you can simply rebuild the cache and re-sync it with the mirror to keep everything up to date. This approach allows you to maintain a usable cache despite partial build failures while ensuring the mirror remains fresh over time.

At this point, you can now copy that data, downloads/ and sstate-cache/, to the file hosting server.

Copy SState-Cache and Downloads to File Hosting Server
rsync -La <origin> <destiny>

Setup SState-Cache Mirror —User Mode

The user can use these mirrors by replacing the URLs from the configuration below, with your own SState-Cache and Downloads locations that were deployed using the setup guide above. Alternatively, you can use these mirrors that RidgeRun provides to the public: 

BB_SIGNATURE_HANDLER = "OEBasicHash"
BB_HASHSERVE = ""

# RidgeRun's SState-Cache Mirror
SSTATE_MIRRORS ?= "\
        file://.* https://yocto.ridgerun.ai/mirror/scarthgap/sstate-cache/PATH;downloadfilename=PATH \
"

# RidgeRun's Downloads Mirror
PREMIRRORS:prepend = "\
        git://.*/.* https://yocto.ridgerun.ai/mirror/scarthgap/downloads/ \
"

Appends

Recommended Configuration

Add the following configuration to the distro configuration files or local.conf: 

BB_NUMBER_THREADS_TO_USE = "${@oe.utils.cpu_count()*2 - 1}"
BB_NUMBER_THREADS = "${BB_NUMBER_THREADS_TO_USE}"
BB_NUMBER_PARSE_THREADS = "${BB_NUMBER_THREADS_TO_USE}"
PARALLEL_MAKE = "-j${BB_NUMBER_THREADS_TO_USE}"

# Init System
INIT_MANAGER = "systemd"

# SSH provider
IMAGE_FEATURES:append = "\
    ssh-server-openssh \
"
PREFERRED_RPROVIDER_ssh = "openssh"

# Default package management
PACKAGE_CLASSES = "package_deb"

CORE_IMAGE_EXTRA_INSTALL:append = " dpkg"
IMAGE_FEATURES:append = " package-management"
DISTRO_FEATURES:append = "\
    pam \
    virtualization \
"

BB_SIGNATURE_HANDLER = "OEBasicHash"
BB_HASHSERVE = ""

BB_GENERATE_MIRROR_TARBALLS = "1"



Retrieved from "https://developer.ridgerun.com/wiki/index.php?title=RidgeRun_Yocto_Developer_Guide/Setting_Up_Mirrors_in_Yocto&oldid=71927"