Mender.io - Software updates for embedded Linux devices
|
|
Mender is a project that allows the OTA (over-the-air) updates on embedded Linux devices in a robust a secure way. Mender has support for multiple platforms; this tutorial will be a focus on the use of Mender in NVIDIA Jetson platforms.
Overview
Mender architecture
Mender implements a client-server architecture, wherein on the server-side the different updates will be stored to be deployed to the corresponding devices, and the client is installed in all those devices to be able to install the updates.
In general, for every update, mender implements the following logic:
- The update is stored in a .menderfile called artifact.
- These artifacts are stored in the Mender server.
- The devices ask for new updates to the server.
- The artifact is downloaded on different devices.
- The artifact is installed on each device.
- The board is rebooted if the artifact requires it, and perform the commit in case of making the update persistent.
The server could be the official Mender server implementation or a custom local server. In case of using the official Mender server, the client installed on the boards will install the updates in a managed mode, and in case of installing the updates using a custom local server or from a USB device, the client will install the update in a standalone mode.
Mender allows its integration in a custom build system, like in the case of Yocto. This wiki page will be explained how to test and then integrate Mender into a Yocto environment.
Basic concepts
The following are important concepts needed to have an adequate understanding of the Mender update system:
A/B redundancy systems
In an A/B redundant system, the partition layout is configured to have an active filesystem A and a redundant filesystem B. During a RootFS update, the update is written to the inactive filesystem B, and when the update is completed, the bootloader is configured to boot at the redundant partition. In case the update has some problem and the board is not able to boot from the redundant filesystem partition, Mender has a recovery mechanism to roll back the upgrade [1].
Artifacts
The artifacts are the files that Mender uses to package the update. These files have the suffix .mender and basically consist of a tar file with all the files needed to perform the update on the board, like some configuration files with information like the type of devices that are compatible with the update, and a payload with the updates files that need to be applied [2] [3].
Update modules
By default, Mender implements the mechanism for the dual rootfs update but also is flexible enough to allow the implementation of custom logic that allows the use of other types of updates [4] [5]. To do this, Mender has the update modules that follow a state machine design pattern as shown in figure 2.
State scripts
The state scripts are used by Mender to be executed in state transitions to overcome a specific use case. The mender client executes this type of script either before or after some state of the update [6]. The state script has the following naming convention:
<STATE_NAME>_<ACTION>_<ORDERING_NUMBER>_<OPTIONAL_DESCRIPTION>
For example, ArtifactInstall_Enter_00_remount_to_rw_filesystem will be executed before the ArtifactInstall state. These state scripts can be included into the Mender installation on the board, or added directly to some update artifact.
Typical partition layout
The expected partition layout to have the correct functioning of Mender is one with A/B partitions for the filesystem updates and a /data partition used by mender to store configuration files that will keep persistent among different updates (See figure 3).
Is also possible to add extra partition, making use of Mender variables for Yocto like MENDER_EXTRA_PARTS, MENDER_EXTRA_PARTS_FSTAB or MENDER_EXTRA_PARTS_SIZES_MB.
Types of updates
RootFS updates
This is the default type of update implemented by Mender to perform a dual rootfs update. A typical flow for this type of update is the following:
- Create the Mender artifact: In a Yocto integration, when the build process completes, a .mender file is created automatically with the rootfs update.
- Upload the artifact to the corresponding server.
- On the board, use the Mender client to download and install the artifact in the inactive filesystem partition.
- For this type of update a reboot is required.
- After performing the update, to make the update persistent is needed to commit the changes using the Mender client.
- In case of revert the update changes, the Mender client can perform a rollback.
Application updates
For uses cases different from the filesystem update, Mender implemented the applications updates. These updates have custom implementations of update-modules to perform actions like single file updates, directory updates, Deb packages updates, Rpm packages updates, docker updates, among others [7] [8] [9] [10] [11] [12].
By default, the update modules come disabled. To enable the mender update-modules, create mender-client_%.bbappend file in your custom meta layer and add the following:
PACKAGECONFIG_append = " modules"
Then, to include some custom update module, add the following to the same .bbappend file:
FILESEXTRAPATHS_prepend := "${THISDIR}/<DIRECTORY-WITH-UPDATE-MODULE>:"
SRC_URI_append = " file://custom-update-module"
do_install_append() {
install -d ${D}/${datadir}/mender/modules/v3
install -m 755 ${WORKDIR}/custom-update-module ${D}/${datadir}/mender/modules/v3/custom-update-module
}
Updates deployments
Managed deployments
The managed deployment method is the one that makes use of the official Mender server. The server can be the one hosted by Mender, with a paid stipend, or hosted in your own infrastructure. From the client-side, when configured as managed mode, the client will run a daemon that constantly will be asking for updates to the server. Mender recommends the managed deployment mode.
Standalone deployments
With the standalone mode the update will be performed manually on the board by executing the following command:
mender -install <URL>
Where the URL can be the address of the artifact in a local web server or in a USB device connected to the board. In some cases, is required to perform a commit of the update changes to make them persistent by running the command:
mender -commit
Security features
You can use the mender-artifact application in your host machine to sign the different artifacts files.
First is needed to generate a private key, used to sign the artifact, and a public key to distribute into the devices:
openssl genpkey -algorithm RSA -out private.key -pkeyopt rsa_keygen_bits:3072 openssl rsa -in private.key -out private.key
Then, you need to integrate the public key to the Yocto image by adding the following to the mender-client_%.bbappend file:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " file://artifact-verify-key.pem"
Please note that you need to rename your public key to artifact-verify-key.pem. After that, with the following command you can sign an existing artifact:
mender-artifact validate artifact-signed.mender -k public.key
Now the mender client installed in the boards won't allow the installation of not signed artifacts.
Recovery features
Mender provides the functionality to recover the system from a failed filesystem update or to perform a rollback to some particular update recently installed. For the case of the filesystem update, after the update is installed the system must be rebooted. If during boot time the system is not able the start from the redundant partition, Mender implements a mechanism to apply a rollback from the bootloader. But even if the update works correctly after the reboot was done, the user can perform a rollback manually, running the following command on the board:
mender -rollback
Some application updates also implement the rollback feature.
Paid features
In general, Mender is an open-source project, but there are some features and services that are not free. The following are some paid features:
- Production server hosted by mender.
- Robust delta updates.
- Secure hosted server.
For the complete list of features, consult [13].
Mender demo using NVIDIA Jetson platforms
The Mender community provides support for different NVIDIA Jetson platforms. To test the Mender functionalities, there is also a demo project for the following platforms:
Prerequisites
- 1. Clone the demo repository:
git clone https://github.com/OE4T/tegra-demo-distro.git
- 2. Switch to the appropriate branch [18].
- 3. Initialize the git submodules:
git submodule update --init
- 4. Source the environment:
source ./setup-env --distro tegrademo-mender --machine <MACH>
Replace MACH with the corresponding machine as shown in the following table:
| Device | <MACH> | |||
|---|---|---|---|---|
| TX2 | jetson-tx2* | jetson-tx2i* | jetson-tx2-4gb* | |
| XAVIER | jetson-xavier* | jetson-xavier-8gb* | ||
| NANO | jetson-nano-devkit* | jetson-nano-emmc | ||
| XAVIER NX | jetson-xavier-nx-devkit* | jetson-xavier-nx-devkit-emmc* | ||
| Note: the machines with * have NOT been tested by RidgeRun yet | ||||
Setup build environment
Initialize build environment:
source ./setup-env --distro tegrademo-mender --machine <MACH>
Build process
For this Yocto environment, you have available different images options to build:
| Image | Description | ||
|---|---|---|---|
| demo-image-base | Basic image with no graphics | ||
| demo-image-egl | Base image with DRM/EGL graphics, no window manager | ||
| demo-image-sato | X11 image with Sato UI | ||
| demo-image-weston | Wayland with Weston compositor | ||
| demo-image-full | Sato image plus nvidia-docker, openCV, multimedia API samples | ||
To start the build run:
bitbake demo-image-base
Testing
When the build completes, in <YOCTO_DEVDIR>/build/tmp/deploy/images/<MACH> you will find a .mender file that corresponds to the artifact to do a dual rootfs update. You can store that artifact in a USB device or a local web server, and using the Mender client on the board, manually install the update (see Standalone deployment section).
Yocto integration
Mender allows its integration to Yocto. For this guide, we will be using the dunfell version of Yocto.
Dependencies
1. Go to your Yocto devdir:
YOCTO_DIR=/path/to/yocto-devdir cd $YOCTO_DIR
2. Select the branch:
export BRANCH="dunfell"
3. 
Download the mender layers:
git clone -b ${BRANCH} https://github.com/mendersoftware/meta-mender.git
git clone -b ${BRANCH} https://github.com/mendersoftware/meta-mender-community.git
git clone -b ${BRANCH} https://github.com/OE4T/meta-tegra-community.git
4. Add the Mender meta layers to the Yocto bblayers configuration files located in YOCTO_DIR/build/conf:
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"
BBPATH = "${TOPDIR}"
BBFILES ?= ""
BBLAYERS ?= " \
...
YOCTO_DIR/meta-mender/meta-mender-core \
YOCTO_DIR/meta-mender-community/meta-mender-tegra \
YOCTO_DIR/meta-tegra-community \
"
Distro setup
Add the following settings values to your custom Yocto distro:
### Additions for mender ###
INHERIT += "mender-full"
INHERIT_append_tegra = " tegra-mender-setup"
# Disabling unneeded mender features
MENDER_FEATURES_DISABLE_append = " mender-systemd"
# Setup mender artifacts
MENDER_ARTIFACT_NAME = "my-release"
ARTIFACTIMG_NAME = "${IMAGE_BASENAME}-boot"
IMAGE_FSTYPES_tegra = "tegraflash mender dataimg"
IMAGE_FSTYPES_pn-tegra-minimal-initramfs_tegra = "${INITRAMFS_FSTYPES}"
# Mender partition settings
MENDER_STORAGE_DEVICE_BASE = "/dev/mmcblk0p"
MENDER_DATA_PART = "${MENDER_STORAGE_DEVICE_BASE}24"
MENDER_ROOTFS_PART_A = "${MENDER_STORAGE_DEVICE_BASE}1"
MENDER_ROOTFS_PART_B = "${MENDER_STORAGE_DEVICE_BASE}18"
# Mender machines need to run checks after an update
SYSTEMD_DEFAULT_TARGET = "finished-booting.target"
In this setup is intended the use of the standalone mode, which means the Mender SystemD service is not needed. That is why the mender-systemd is disabled. Also, verify your partition layout very carefully to set the correct block devices for the rootfs A, rootfs B, and the data partition.
Now you can trigger the build process of your Yocto image again, and this time you should get a .mender file with the rootfs update generated in YOCTO_DIR/build/tmp/deploy/images/<MACH>. Finally, is recommended to follow the Mender checklist to verify this integration work [19].
Artifact creation
To create the artifacts, you need to download and install the mender-artifact tool in your host machine [20].
RootFS artifacts
As mentioned in previous sections, when Mender is integrated with Yocto, each time you complete a build is generated a .mender file with the rootfs artifact. But is also possible to create it manually running the following command:
mender-artifact write rootfs-image \ -t jetson-nano-emmc \ -n release-1 \ --software-version rootfs-v1 \ -f rootfs.ext4 \ -o artifact.mender
Where rootfs.ext4 corresponds to the filesystem image, and:
- -t: specifies the compatible device. In this case, we are creating the artifact for the jetson-nano-emmc.
- -n: this is the name of the artifact.
- --software-version: corresponds to the string software version.
- -o: path of the output file.
Application artifacts
There are different types of application artifacts. In this case, we are going to present an example to create an artifact to perform a single file update.
To create the single file artifact, first download the single-file-artifact-gen tool:
wget https://raw.githubusercontent.com/mendersoftware/mender/master/support/modules-artifact-gen/single-file-artifact-gen chmod +x single-file-artifact-gen
Then, create the sample file to deploy:
echo "File created by Mender single-file Update Module!" > my_update_file.txt
Then, create the artifact with the following command:
ARTIFACT_NAME="my-update-1.0"
DEVICE_TYPE="jetson-nano-emmc"
OUTPUT_PATH="my-update-1.0.mender"
DEST_DIR="/opt/installed-by-single-file/"
FILE="my_update_file.txt"
./single-file-artifact-gen -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -d ${DEST_DIR} -o ${OUTPUT_PATH} ${FILE}
Where:
- ARTIFACT_NAME - The name of the Mender Artifact
- DEVICE_TYPE - The compatible device type of this Mender Artifact
- OUTPUT_PATH - The path where to place the output Mender Artifact. This should always have a .mender suffix
- DEST_DIR - The path on target device where FILE will be installed.
- FILE - The path to the file to be sent to the device in the update.
References
- ↑ https://docs.mender.io/overview/introduction#robust-system-updates
- ↑ https://docs.mender.io/overview/artifact
- ↑ https://github.com/mendersoftware/mender-artifact/blob/master/Documentation/artifact-format-v3.md
- ↑ https://docs.mender.io/artifact-creation/create-a-custom-update-module
- ↑ https://github.com/mendersoftware/mender/blob/2.5.0/Documentation/update-modules-v3-file-api.md
- ↑ https://docs.mender.io/overview/state-script
- ↑ https://hub.mender.io/t/single-file/486
- ↑ https://hub.mender.io/t/deb-packages/326
- ↑ https://hub.mender.io/t/rpm-packages/494
- ↑ https://hub.mender.io/t/docker/324
- ↑ https://hub.mender.io/t/script/328
- ↑ https://hub.mender.io/t/directory/325
- ↑ https://mender.io/plans/features
- ↑ https://hub.mender.io/t/nvidia-tegra-jetson-nano/1360
- ↑ https://hub.mender.io/t/nvidia-tegra-jetson-tx2/123
- ↑ https://hub.mender.io/t/nvidia-tegra-jetson-xavier-nx/2615
- ↑ https://hub.mender.io/t/nvidia-tegra-agx-xavier/2616
- ↑ https://github.com/OE4T/tegra-demo-distro/wiki/Which-branch-should-I-use%3F
- ↑ https://docs.mender.io/client-installation/integration-checklist
- ↑ https://docs.mender.io/downloads#mender-artifact
For direct inquiries, please refer to the contact information available on our Contact page. Alternatively, you may complete and submit the form provided at the same link. We will respond to your request at our earliest opportunity.
Links to RidgeRun Resources and RidgeRun Artificial Intelligence Solutions can be found in the footer below.
ARTIFICIAL INTELLIGENCE
