View source for Mender.io - Software updates for embedded Linux devices

<seo title="Software updates for embedded Linux devices | Mender.io | RidgeRun" titlemode="replace" keywords=" Mender, Mender architecture, NVIDIA Jetson, Jetson, Mender server, Mender update, Mender deployment, Yocto, software updates, software update for embedded linux, embedded Linux, deployment, software deployment, embedded software deployment, managed deployment, standalone deployment"  description="This wiki page has info about the Mender.io a software update tool and mechanism for Software updates for embedded Linux devices"></seo>


[https://mender.io/ Mender] is a project that allows the OTA (over-the-air) updates on embedded Linux devices in a robust a secure way. Mender has support for multiple platforms; this tutorial will be a focus on the use of Mender in NVIDIA Jetson platforms.

== Overview ==

=== <u>Mender architecture</u> ===

[[File:Updater-components.png|600px|thumb|right|<span style="color:#0000FF">Figure 1. Mender architecture</span>]]

Mender implements a client-server architecture, wherein on the server-side the different updates will be stored to be deployed to the corresponding devices, and the client is installed in all those devices to be able to install the updates.

In general, for every update, mender implements the following logic: 

# The update is stored in a ''.mender''file called artifact.
# These artifacts are stored in the Mender server.
# The devices ask for new updates to the server.
# The artifact is downloaded on different devices.
# The artifact is installed on each device.
# The board is rebooted if the artifact requires it, and perform the commit in case of making the update persistent.

The server could be the official Mender server implementation or a custom local server. In case of using the official Mender server, the client installed on the boards will install the updates in a managed mode, and in case of installing the updates using a custom local server or from a USB device, the client will install the update in a standalone mode.

Mender allows its integration in a custom build system, like in the case of Yocto. This wiki page will be explained how to test and then integrate Mender into a Yocto environment.

=== <u>Basic concepts</u> ===

[[File:Update-modules-state-machine.png|500px|thumb|right|<span style="color:#0000FF">Figure 2. State machine pattern implemented to Mender's update module.</span>]]

The following are important concepts needed to have an adequate understanding of the Mender update system:

==== A/B redundancy systems ====

In an A/B redundant system, the partition layout is configured to have an active filesystem A and a redundant filesystem B. During a RootFS update, the update is written to the inactive filesystem B, and when the update is completed, the bootloader is configured to boot at the redundant partition. In case the update has some problem and the board is not able to boot from the redundant filesystem partition, Mender has a recovery mechanism to roll back the upgrade <ref>https://docs.mender.io/overview/introduction#robust-system-updates</ref>.

==== Artifacts ====

The artifacts are the files that Mender uses to package the update. These files have the suffix ''.mender'' and basically consist of a tar file with all the needed files needed to perform the update on the board, like some configuration files with information like the type of devices that are compatible with the update, and a payload with the updates files that need to be applied <ref>https://docs.mender.io/overview/artifact</ref> <ref>https://github.com/mendersoftware/mender-artifact/blob/master/Documentation/artifact-format-v3.md</ref>.

==== Update modules ====

By default, Mender implements the mechanism for the dual rootfs update but also is flexible enough to allow the implementation of custom logic that allows the use of other types of updates <ref>https://docs.mender.io/artifact-creation/create-a-custom-update-module</ref> <ref>https://github.com/mendersoftware/mender/blob/2.5.0/Documentation/update-modules-v3-file-api.md</ref>. To do this, Mender has the update modules that follow a state machine design pattern as shown in figure 2.

==== State scripts ====

The state scripts are used by Mender to be executed in state transitions to overcome a specific use case. The mender client executes this type of script either before or after some state of the update <ref>https://docs.mender.io/overview/state-script</ref>. The state script has the following naming convention:

<source lang="bash">
<STATE_NAME>_<ACTION>_<ORDERING_NUMBER>_<OPTIONAL_DESCRIPTION>
</source>

For example, ''ArtifactInstall_Enter_00_remount_to_rw_filesystem'' will be executed before the '''ArtifactInstall''' state. These state scripts can be included into the Mender installation on the board, or added directly to some update artifact.

=== <u>Typical partition layout</u> ===

The expected partition layout to have the correct functioning of Mender is one with A/B partitions for the filesystem updates and a /data partition used by mender to store configuration files that will keep persistent among different updates (See figure 3).

[[File:Mender client partition layout.png|400px|thumb|center|<span style="color:#0000FF">Figure 3. Mender typical partition layout.</span>]]
<br>
Is also possible to add extra partition, making use of Mender variables for Yocto like MENDER_EXTRA_PARTS, MENDER_EXTRA_PARTS_FSTAB or MENDER_EXTRA_PARTS_SIZES_MB.

=== <u>Types of updates</u> ===

==== RootFS updates ====

This is the default type of update implemented by Mender to perform a dual rootfs update. A typical flow for this type of update is the following:

# ''Create the Mender artifact'': In a Yocto integration, when the build process completes, a .mender file is created automatically with the rootfs update.
# Upload the artifact to the corresponding server.
# On the board, use the Mender client to download and install the artifact in the inactive filesystem partition.
# For this type of update a reboot is required.
# After performing the update, to make the update persistent is needed to commit the changes using the Mender client.
# In case of revert the update changes, the Mender client can perform a rollback.

==== Application updates ====

For uses cases different from the filesystem update, Mender implemented the applications updates. These updates have custom implementations of update-modules to perform actions like single file updates, directory updates, Deb packages updates, Rpm packages updates, docker updates, among others <ref>https://hub.mender.io/t/single-file/486</ref> <ref>https://hub.mender.io/t/deb-packages/326</ref> <ref>https://hub.mender.io/t/rpm-packages/494</ref> <ref>https://hub.mender.io/t/docker/324</ref> <ref>https://hub.mender.io/t/script/328</ref> <ref>https://hub.mender.io/t/directory/325</ref>.

By default, the update modules come disabled. To enable the mender update-modules, create '''mender-client_%.bbappend''' file in your custom meta layer and add the following:

<source lang="bash">
PACKAGECONFIG_append = " modules"
</source>

Then, to include some custom update module, add the following to the same ''.bbappend'' file:

<source lang="bash">
FILESEXTRAPATHS_prepend := "${THISDIR}/<DIRECTORY-WITH-UPDATE-MODULE>:"
SRC_URI_append = " file://custom-update-module"

do_install_append() {
    install -d ${D}/${datadir}/mender/modules/v3
    install -m 755 ${WORKDIR}/custom-update-module ${D}/${datadir}/mender/modules/v3/custom-update-module
}
</source>

=== <u>Updates deployments</u> ===

==== Managed deployments ====

The managed deployment method is the one that makes use of the official Mender server. The server can be the one hosted by Mender, with a paid stipend, or hosted in your own infrastructure. From the client-side, when configured as managed mode, the client will run a daemon that constantly will be asking for updates to the server. Mender recommends the managed deployment mode.

==== Standalone deployments ====

With the standalone mode the update will be performed manually on the board by executing the following command:

<source lang="bash">
mender -install <URL>
</source>

Where the URL can be the address of the artifact in a local web server or in a USB device connected to the board. In some cases, is required to perform a commit of the update changes to make them persistent by running the command:

<source lang="bash">
mender -commit
</source>

=== <u>Security features</u> ===

You can use the mender-artifact application in your host machine to sign the different artifacts files.

First is needed to generate a private key, used to sign the artifact, and a public key to distribute into the devices:

<source lang="bash">
openssl genpkey -algorithm RSA -out private.key -pkeyopt rsa_keygen_bits:3072
openssl rsa -in private.key -out private.key
</source>

Then, you need to integrate the public key to the Yocto image by adding the following to the mender-client_%.bbappend file:

<source lang="bash">
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " file://artifact-verify-key.pem"
</source>

Please note that you need to rename your public key to '''artifact-verify-key.pem'''. After that, with the following command you can sign an existing artifact:

<source lang="bash">
mender-artifact validate artifact-signed.mender -k public.key
</source>

Now the mender client installed in the boards won't allow the installation of not signed artifacts.

=== <u>Recovery features</u> ===

Mender provides the functionality to recover the system from a failed filesystem update or to perform a rollback to some particular update recently installed. For the case of the filesystem update, after the update is installed the system must be rebooted. If during boot time the system is not able the start from the redundant partition, Mender implements a mechanism to apply a rollback from the bootloader. But even if the update works correctly after the reboot was done, the user can perform a rollback manually, running the following command on the board:

<source lang=bash>
mender -rollback
</source>

Some application updates also implement the rollback feature.

=== <u>Paid features</u> ===

In general, Mender is an open-source project, but there are some features and services that are not free. The following are some paid features:

* Production server hosted by mender.
* Robust delta updates.
* Secure hosted server.

For the complete list of features, consult <ref>https://mender.io/plans/features</ref>.

== Mender demo using NVIDIA Jetson platforms ==

The Mender community provides support for different NVIDIA Jetson platforms. To test the Mender functionalities, there is also a demo project for the following platforms:

* Jetson Nano <ref>https://hub.mender.io/t/nvidia-tegra-jetson-nano/1360</ref>.
* Jetson TX2 <ref>https://hub.mender.io/t/nvidia-tegra-jetson-tx2/123</ref>.
* Jetson Xavier NX <ref>https://hub.mender.io/t/nvidia-tegra-jetson-xavier-nx/2615</ref>.
* Tegra AGX Xavier <ref>https://hub.mender.io/t/nvidia-tegra-agx-xavier/2616</ref>.

=== <u>Prerequisites</u> ===

* 1. Clone the demo repository:

<source lang="bash">
git clone https://github.com/OE4T/tegra-demo-distro.git
</source>

* 2. Switch to the appropriate branch <ref>https://github.com/OE4T/tegra-demo-distro/wiki/Which-branch-should-I-use%3F</ref>.

* 3. Initialize the git submodules:

<source lang="bash">
git submodule update --init
</source>

* 4. Source the environment:

<source lang="bash">
source ./setup-env --distro tegrademo-mender --machine <MACH>
</source>

Replace MACH with the corresponding machine as shown in the following table:

<br>
<br>
{| class="wikitable" style="margin: auto;"
!colspan="1"|Device
!colspan="3"|<MACH>
|-
|TX2
|jetson-tx2'''*''' 
|jetson-tx2i'''*'''
|jetson-tx2-4gb'''*'''
|-
|XAVIER
|jetson-xavier'''*''' 
|jetson-xavier-8gb'''*'''
|
|-
|NANO
|jetson-nano-devkit'''*'''
|jetson-nano-emmc
|
|-
|XAVIER NX
|jetson-xavier-nx-devkit'''*'''
|jetson-xavier-nx-devkit-emmc'''*'''
|
|- 
!colspan="5"|Note: the machines with * have NOT been tested by RidgeRun yet
|}

=== <u>Setup build environment</u> ===

Initialize build environment:

<source lang="bash">
source ./setup-env --distro tegrademo-mender --machine <MACH>
</source>

=== <u>Build process</u> ===

For this Yocto environment, you have available different images options to build:

<br>
<br>
{| class="wikitable" style="margin: auto;"
!colspan="1"|Image
!colspan="3"|Description
|-
|demo-image-base
|Basic image with no graphics
|-
|demo-image-egl
|Base image with DRM/EGL graphics, no window manager
|-
|demo-image-sato
|X11 image with Sato UI
|-
|demo-image-weston
|Wayland with Weston compositor
|-
|demo-image-full
|Sato image plus nvidia-docker, openCV, multimedia API samples
|-
|}

To start the build run:

<source lang="bash">
bitbake demo-image-base
</source>

=== <u>Testing</u> ===

When the build completes, in '''<YOCTO_DEVDIR>/build/tmp/deploy/images/<MACH>''' you will find a ''.mender'' file that corresponds to the artifact to do a dual rootfs update. You can store that artifact in a USB device or a local web server, and using the Mender client on the board, manually install the update (see [[Mender.io_-_Software_updates_for_embedded_Linux_devices#Standalone_deployments|Standalone deployment]] section).

== Yocto integration ==

Mender allows its integration to Yocto. For this guide, we will be using the dunfell version of Yocto.

=== <u>Dependencies</u> ===

1. Go to your Yocto devdir:

<source lang="bash">
YOCTO_DIR=/path/to/yocto-devdir
cd $YOCTO_DIR
</source>

2. Select the branch:

<source lang="bash">
export BRANCH="dunfell"
</source>

3. {{Download}} Download the mender layers:

<source lang="bash">
git clone -b ${BRANCH} https://github.com/mendersoftware/meta-mender.git
git clone -b ${BRANCH} https://github.com/mendersoftware/meta-mender-community.git
git clone -b ${BRANCH} https://github.com/OE4T/meta-tegra-community.git
</source>

4. Add the Mender meta layers to the Yocto '''bblayers''' configuration files located in ''YOCTO_DIR/build/conf'':

<source lang="bash">
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  ...
  YOCTO_DIR/meta-mender/meta-mender-core \
  YOCTO_DIR/meta-mender-community/meta-mender-tegra \
  YOCTO_DIR/meta-tegra-community \
  "
</source>

=== <u>Distro setup</u> ===

Add the following settings values to your custom Yocto distro:

<source lang="bash">
### Additions for mender ###

INHERIT += "mender-full"
INHERIT_append_tegra = " tegra-mender-setup"

# Disabling unneeded mender features
MENDER_FEATURES_DISABLE_append = " mender-systemd"

# Setup mender artifacts
MENDER_ARTIFACT_NAME = "my-release"
ARTIFACTIMG_NAME = "${IMAGE_BASENAME}-boot"

IMAGE_FSTYPES_tegra = "tegraflash mender dataimg"
IMAGE_FSTYPES_pn-tegra-minimal-initramfs_tegra = "${INITRAMFS_FSTYPES}"

# Mender partition settings
MENDER_STORAGE_DEVICE_BASE = "/dev/mmcblk0p"
MENDER_DATA_PART = "${MENDER_STORAGE_DEVICE_BASE}24"
MENDER_ROOTFS_PART_A = "${MENDER_STORAGE_DEVICE_BASE}1"
MENDER_ROOTFS_PART_B = "${MENDER_STORAGE_DEVICE_BASE}18"

# Mender machines need to run checks after an update
SYSTEMD_DEFAULT_TARGET = "finished-booting.target"
</source>

In this setup is intended the use of the standalone mode, which means the Mender SystemD service is not needed. That is why the ''mender-systemd'' is disabled. Also, verify your partition layout very carefully to set the correct block devices for the rootfs A, rootfs B, and the data partition.

Now you can trigger the build process of your Yocto image again, and this time you should get a .mender file with the rootfs update generated in ''YOCTO_DIR/build/tmp/deploy/images/<MACH>''. Finally, is recommended to follow the [https://docs.mender.io/client-installation/integration-checklist Mender checklist] to verify this integration work <ref>https://docs.mender.io/client-installation/integration-checklist</ref>.

== Artifact creation ==

To create the artifacts, you need to {{Download}} download and install the [https://docs.mender.io/downloads#mender-artifact mender-artifact tool] in your host machine <ref>https://docs.mender.io/downloads#mender-artifact</ref>.

=== <u>RootFS artifacts</u> ===

As mentioned in previous sections, when Mender is integrated with Yocto, each time you complete a build is generated a ''.mender'' file with the rootfs artifact. But is also possible to create it manually running the following command:

<source lang="bash">
mender-artifact write rootfs-image \
   -t jetson-nano-emmc \
   -n release-1 \
   --software-version rootfs-v1 \
   -f rootfs.ext4 \
   -o artifact.mender
</source>

Where rootfs.ext4 corresponds to the filesystem image, and:

* '''-t''': specifies the compatible device. In this case, we are creating the artifact for the jetson-nano-emmc.
* '''-n''': this is the name of the artifact.
* '''--software-version''': corresponds to the string software version.
* '''-o''': path of the output file.

=== <u>Application artifacts</u> ===

There are different types of application artifacts. In this case, we are going to present an example to create an artifact to perform a single file update.

To create the single file artifact, first {{Download}} download the ''single-file-artifact-gen'' tool:

<source lang="bash">
wget https://raw.githubusercontent.com/mendersoftware/mender/master/support/modules-artifact-gen/single-file-artifact-gen
chmod +x single-file-artifact-gen
</source>

Then, create the sample file to deploy:

<source lang="bash">
echo "File created by Mender single-file Update Module!" > my_update_file.txt
</source>

Then, create the artifact with the following command:

<source lang="bash">
ARTIFACT_NAME="my-update-1.0"
DEVICE_TYPE="jetson-nano-emmc"
OUTPUT_PATH="my-update-1.0.mender"
DEST_DIR="/opt/installed-by-single-file/"
FILE="my_update_file.txt"
./single-file-artifact-gen -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -d ${DEST_DIR} -o ${OUTPUT_PATH} ${FILE}
</source>

Where:

* ARTIFACT_NAME - The name of the Mender Artifact
* DEVICE_TYPE - The compatible device type of this Mender Artifact
* OUTPUT_PATH - The path where to place the output Mender Artifact. This should always have a .mender suffix
* DEST_DIR - The path on target device where FILE will be installed.
* FILE - The path to the file to be sent to the device in the update.

==References==
<references />

{{ContactUs}}

[[Category:HowTo]][[Category:Jetson]][[Category:GStreamer]][[Category:JetsonNano]][[Category:JetsonTX2‏‎]][[Category:JetsonXavierNX]][[Category:NVIDIA Xavier‏‎]]