RidgeRun Yocto Developer Guide - Setting Up Mirrors in Yocto
In this guide, you’ll learn how to create a base build that can be used to retrieve sources and generate mirrors for both downloads/ and sstate-cache/. These mirrors are especially useful for setting up shared development environments, CI pipelines, or local acceleration of builds.
Steps
1. Setup a Yocto Build
Start by setting up a new build directory with your preferred configuration. Ensure that the build includes the supported BSP layers (e.g., MetaTegra in this case). Adjust your bblayers.conf and local.conf files as needed to define your target machine, distro, and any project-specific options.
2. Fetch The Sources and Build The Image
Once your build configuration is ready, initiate a build process. This step fetches and compiles the sources for all packages across the included layers. By doing so, you ensure that the required source archives and prebuilt artifacts are stored in your downloads/ and sstate-cache/ directories.
3. Prepare the Mirrors
After the build completes successfully:
Copy the downloads/ folder to your file hosting server to serve as your source mirror.
Copy the sstate-cache/ folder to the server to serve as your shared state mirror.
These mirrors can then be reused by other developers or CI jobs to speed up future builds and reduce redundant downloads.
Setup a Yocto Build
Get the Layers
WORKSPACE=<set your workspace directory> mkdir -p $WORKSPACE cd $WORKSPACE YOCTO_RELEASE=scarthgap LAYERS_DIR=$WORKSPACE/layers/ mkdir -p $LAYERS_DIR/ cd $_ git clone https://git.yoctoproject.org/poky -b $YOCTO_RELEASE git clone https://git.yoctoproject.org/poky -b $YOCTO_RELEASE git clone https://git.openembedded.org/meta-openembedded -b $YOCTO_RELEASE git clone https://github.com/OE4T/meta-tegra -b $YOCTO_RELEASE git clone https://github.com/OE4T/meta-tegra-community -b $YOCTO_RELEASE git clone https://git.yoctoproject.org/meta-virtualization -b $YOCTO_RELEASE git clone https://git.yoctoproject.org/meta-selinux -b $YOCTO_RELEASE cd -
Pull Docker Image For Building
DOCKER_IMAGE_NAME=scarthgap
docker pull danielchvs/yocto:$DOCKER_IMAGE_NAME
docker run \
--rm \
-it \
--privileged \
--network host \
-e USER=$USER \
-e UID=$(id -u) \
-e GID=$(id -g) \
-v $PWD:/workdir \
-v $HOME/.gitconfig:$HOME/.gitconfig \
-v $HOME/.ssh:$HOME/.ssh \
$DOCKER_IMAGE_NAME \
bash
. layers/poky/oe-init-build-env build/
Add The Layers to Build Environment
bitbake-layers add-layer $OLDPWD/layers/poky/meta bitbake-layers add-layer $OLDPWD/layers/poky/meta-poky bitbake-layers add-layer $OLDPWD/layers/poky/meta-yocto-bsp bitbake-layers add-layer $OLDPWD/layers/poky/meta-skeleton bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-oe bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-python bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-networking bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-gnome bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-multimedia bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-xfce bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-initramfs bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-perl bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-filesystems bitbake-layers add-layer $OLDPWD/layers/meta-openembedded/meta-webserver bitbake-layers add-layer $OLDPWD/layers/meta-virtualization bitbake-layers add-layer $OLDPWD/layers/meta-tegra bitbake-layers add-layer $OLDPWD/layers/meta-tegra-community
[Add the appends. Refer to this document's Appends section]
Fetch The Sources and Build The Image For The Target Machines
Generate Sources For the Mirror - The downloads/ Directory
The sources mirror provides access to the tarballs or Git bare repositories of the project sources. This setup is safe for production use, and it is the recommended option for fetching dependencies during production builds.
Run Bitbake World with a runall fetch for all the desired machines across the supported BSPs:
# QEMU machines MACHINE=qemux86-64 bitbake world --runall=fetch -k MACHINE=qemuarm64 bitbake world --runall=fetch -k # MetaTegra BSP machines MACHINE=jetson-agx-orin-devkit bitbake world --runall=fetch -k MACHINE=jetson-agx-orin-devkit-industrial bitbake world --runall=fetch -k MACHINE=jetson-orin-nano-devkit bitbake world --runall=fetch -k MACHINE=jetson-orin-nano-devkit-nvme bitbake world --runall=fetch -k MACHINE=p3509-a02-p3767-0000 bitbake world --runall=fetch -k MACHINE=p3737-0000-p3701-0004 bitbake world --runall=fetch -k MACHINE=p3737-0000-p3701-0005 bitbake world --runall=fetch -k MACHINE=p3768-0000-p3767-0000 bitbake world --runall=fetch -k MACHINE=p3768-0000-p3767-0001 bitbake world --runall=fetch -k MACHINE=p3768-0000-p3767-0004 bitbake world --runall=fetch -k
If the BitBake World command does not pass on the first attempt, see the “Fixups to BitBake World Command” section in Appends for more details.
Generate SState Cache For the Mirror - The sstate-cache/ Directory
The SState Cache mirror provides access to prebuilt task artifacts generated by Yocto during previous builds. By reusing these cached results, subsequent builds can skip recompiling unchanged components, dramatically reducing build times.
This setup is the recommended approach when you want to accelerate builds across different machines or supported BSPs, for example for developing.
In order to generate this cache, run the following Bitbake World command across all the supported BSPs:
# QEMU machines MACHINE=qemux86-64 bitbake world -k MACHINE=qemuarm64 bitbake world -k # MetaTegra BSP machines MACHINE=jetson-agx-orin-devkit bitbake world -k MACHINE=jetson-agx-orin-devkit-industrial bitbake world -k MACHINE=jetson-orin-nano-devkit bitbake world -k MACHINE=jetson-orin-nano-devkit-nvme bitbake world -k MACHINE=p3509-a02-p3767-0000 bitbake world -k MACHINE=p3737-0000-p3701-0004 bitbake world -k MACHINE=p3737-0000-p3701-0005 bitbake world -k MACHINE=p3768-0000-p3767-0000 bitbake world -k MACHINE=p3768-0000-p3767-0001 bitbake world -k MACHINE=p3768-0000-p3767-0004 bitbake world -k
The previous Bitbake World command downloads and caches all the packages provided by the layers, creating a general layer-level cache. Depending on your use case, you may instead run Bitbake with your specific target image(s) rather than world. This approach gives you a more focused and end-to-end build cache tailored to your target images. Just remember to keep your cache up to date, since package configurations may have changed due to customizations introduced by your platform build.
Note: it's plausible that not all recipes in a World build may compile successfully, which means the previous commands could encounter errors. The -k flag ensures the build continues as far as possible, even after failures. For example, if 10–15 recipes fail and are not required for your workflow, i.e., your build never builds them, you can still consider the resulting cache a success and proceed to copy it to your mirror.
If a recipe's configuration changes later, you can simply rebuild the cache and re-sync it with the mirror to keep everything up to date. This approach allows you to maintain a usable cache despite partial build failures while ensuring the mirror remains fresh over time.
At this point, you can now copy that data, downloads/ and sstate-cache/, to the file hosting server.
Copy Sstate-cache and Downloads to File Hosting Server
rsync -La <origin> <destiny>
Appends
Local.conf Recommended Configuration
Add the following configuration to the distro configuration files or local.conf:
BB_NUMBER_THREADS_TO_USE = "${@oe.utils.cpu_count()*2 - 1}"
BB_NUMBER_THREADS = "${BB_NUMBER_THREADS_TO_USE}"
BB_NUMBER_PARSE_THREADS = "${BB_NUMBER_THREADS_TO_USE}"
PARALLEL_MAKE = "-j${BB_NUMBER_THREADS_TO_USE}"
PARALLEL_MAKEINST = "-j${BB_NUMBER_THREADS_TO_USE}"
# Init System
INIT_MANAGER = "systemd"
# SSH provider
IMAGE_FEATURES:append = "\
ssh-server-openssh \
"
PREFERRED_RPROVIDER_ssh = "openssh"
# Default package management
PACKAGE_CLASSES = "package_deb"
CORE_IMAGE_EXTRA_INSTALL:append = " dpkg"
IMAGE_FEATURES:append = " package-management"
DISTRO_FEATURES:append = "\
pam \
virtualization \
"
BB_GENERATE_MIRROR_TARBALLS = "1"
Fixups to Bitbake World Command
After running the Bitbake World fetch, it's not rare to find some errors in recipes. You'll have to fix the recipes to let the build pass. Add the fixes directly in local.conf for this case. Appended is an example of how those fixes would look like:
PREFERRED_VERSION_tpm2-tss-engine = "1.1.0" MAINSUM:pn-tegra-libraries-omx = "55c001ffe6b17b886e008bd54ce46fb1ab6fea70c123548acf7f30a67f23f986" MAINSUM:pn-tegra-libraries-winsys = "12440f0eff87ff22906726fdad61d87a2681bf8ace9d824b8b208e0ee8dfdbcc" SRC_URI:pn-hunspell-dictionaries = "git://github.com/wooorm/dictionaries.git;branch=main;protocol=https" SRCREV:pn-xvisor = "58592ef18c71526a0045935d1e8eed5e8553b7d6" COMPATIBLE_MACHINE:pn-nvidia-docker = "(tegra|tegra234)"