Over-the-Air Updates in NVIDIA Jetson platforms
 | RidgeRun Platform Security Manual RidgeRun documentation is currently under development. |
This wiki shows how to implement Mender.io for NVIDIA Jetson Platforms, allowing over-the-air updates that can be triggered manually on the board or deployed through a remote server. This tutorial was tested with a Jetson Orin Nano module running JetPack 6 and implementing Mender.io with a Yocto build.
Mender also has support for other platforms such as NXP and Raspberry Pi trough the meta mender community Yocto layer, so the implementation for other platforms is similar to what is depicted in this guide,
Yocto Support
As a first step you need to set up Yocto and configure the build directory, to do so you can follow RidgeRun's Yocto Guide for NVIDIA® Jetson™ with JetPack 6 Integration. After obtaining a base configuration, you need to add layers specific to mender to add the integration, you can obtain them with the following commands:
cd $YOCTO_DIR
git clone https://github.com/openembedded/meta-openembedded.git -b scarthgap
git clone https://github.com/mendersoftware/meta-mender.git -b scarthgap
git clone https://github.com/mendersoftware/meta-mender-community.git -b scarthgap
git clone https://github.com/OE4T/meta-tegra-community.git -b scarthgap
After this, you need to add the layers in file $YOCTO_DIR/build/conf/bblayers.conf. You can modify the file so that it looks similar to the following:
BBLAYERS ?= " \
/home/${USER}/yocto-tegra/meta-mender/meta-mender-core \
/home/${USER}/yocto-tegra/meta-mender-community/meta-mender-tegra \
/home/${USER}/yocto-tegra/meta-openembedded/meta-filesystems \
/home/${USER}/yocto-tegra/meta-openembedded/meta-networking \
/home/${USER}/yocto-tegra/meta-openembedded/meta-oe \
/home/${USER}/yocto-tegra/meta-openembedded/meta-python \
/home/${USER}/yocto-tegra/meta-tegra \
/home/${USER}/yocto-tegra/meta-tegra-community \
/home/${USER}/yocto-tegra/poky/meta \
/home/${USER}/yocto-tegra/poky/meta-poky \
/home/${USER}/yocto-tegra/poky/meta-yocto-bsp \
"
Then you need to update file $YOCTO_DIR/build/conf/local.conf so that your build can correctly support mender, as an example:
# AB-upgrades
UBOOT_EXTLINUX = "1"
USE_REDUNDANT_FLASH_LAYOUT_DEFAULT = "1"
# base
CONF_VERSION = "2"
PACKAGE_CLASSES = "package_ipk"
INIT_MANAGER = "systemd"
EXTRA_IMAGE_FEATURES ?= "debug-tweaks"
# eMMC
EMMC_SIZE = "0"
# mender-artifact
MENDER_ARTIFACT_NAME = "gha_autobuild"
# mender-full
MENDER_EFI_LOADER = ""
MENDER_STORAGE_TOTAL_SIZE_MB = "16384"
INHERIT += "mender-full"
# tegra
# these two classes only work as intended when being inherited in the
# OE4t setup-env.sh style environment, as they modify bblayers.conf
# and expect additional information on the host.
INHERIT:remove = "tegra-support-sanity distro_layer_buildinfo"
INHERIT += "tegra-mender-setup"
MENDER_FEATURES_DISABLE:append = " mender-uboot"
MENDER_STORAGE_DEVICE_BASE="/dev/mmcblk0p"
MENDER_BOOT_PART = "${MENDER_STORAGE_DEVICE_BASE}11"
MENDER_DATA_PART = "${MENDER_STORAGE_DEVICE_BASE}15"
MENDER_ROOTFS_PART_A = "${MENDER_STORAGE_DEVICE_BASE}1"
MENDER_ROOTFS_PART_B = "${MENDER_STORAGE_DEVICE_BASE}2"
MENDER_DATA_PART_SIZE_MB = "400"
IMAGE_FSTYPES:tegra = "tegraflash mender dataimg"
IMAGE_FSTYPES:pn-tegra-minimal-initramfs:tegra = "${INITRAMFS_FSTYPES}"
IMAGE_FSTYPES:pn-tegra-initrd-flash-initramfs:tegra = "${TEGRA_INITRD_FLASH_INITRAMFS_FSTYPES}"
MACHINE ??= "jetson-orin-nano-devkit"
DISTRO ??= "tegrademo"
BBMULTICONFIG ?= ""
You need two partitions for the Mender configuration in order to correctly implement the AB system for updates; in the example, this is done with the following lines:
UBOOT_EXTLINUX = "1"
USE_REDUNDANT_FLASH_LAYOUT_DEFAULT = "1"
And then, the lines specific to the mender configuration are the following:
# mender-artifact
MENDER_ARTIFACT_NAME = "gha_autobuild"
# mender-full
MENDER_EFI_LOADER = ""
MENDER_STORAGE_TOTAL_SIZE_MB = "16384"
INHERIT += "mender-full"
# tegra
# these two classes only work as intended when being inherited in the
# OE4t setup-env.sh style environment, as they modify bblayers.conf
# and expect additional information on the host.
INHERIT:remove = "tegra-support-sanity distro_layer_buildinfo"
INHERIT += "tegra-mender-setup"
MENDER_FEATURES_DISABLE:append = " mender-uboot"
MENDER_STORAGE_DEVICE_BASE="/dev/mmcblk0p"
MENDER_BOOT_PART = "${MENDER_STORAGE_DEVICE_BASE}11"
MENDER_DATA_PART = "${MENDER_STORAGE_DEVICE_BASE}15"
MENDER_ROOTFS_PART_A = "${MENDER_STORAGE_DEVICE_BASE}1"
MENDER_ROOTFS_PART_B = "${MENDER_STORAGE_DEVICE_BASE}2"
MENDER_DATA_PART_SIZE_MB = "400"
Of the above variables, the MENDER_STORAGE_TOTAL_SIZE_MB is the total size of this physical storage medium that is going to be used for the image build, it does not necessarily have to be the total capacity of the storage device. Other variables such as MENDER_BOOT_PART and MENDER_ROOTFS_PART_A are used to indicate the partition number corresponding to each use. As an example, the default Jetson partition layout contains partitions for the Kernel Image and Device Tree, the numbers corresponding to these partitions should not be used for one the previously mentioned variables.
Be careful when setting the Mender Storage Device and the partition numbers to match your device and partition layout, otherwise your device might not be able to boot correctly. After applying the changes to $YOCTO_DIR/build/conf/bblayers.conf and $YOCTO_DIR/build/conf/local.conf you can trigger the build process with the desired image, for example:
bitbake -k core-image-sato
After the building process is completed successfully you will have a .tegraflash.tar.gz and .mender files in $YOCTO_DIR/build/tmp/deploy/images/$MACHINE. The .tegraflash.tar.gz file will be used to flash your board for the first time and the .mender will be used to perform the updates. Each time the build is modified a .mender is automatically generated, this file contains the changes applied to build and can be used to update the board with the respective changes.
To flash the board for the first time after completing the build, it is recommended to use a temporary directory to decompress the .tegraflash.tar.gz file, you can use the following commands to do so:
tmpdir=`mktemp`
rm -rf $tmpdir
mkdir -p $tmpdir
pushd $tmpdir
cp $YOCTO_DIR/build/tmp/deploy/images/$MACHINE/$IMAGE .
tar -xvf $IMAGE
With the Jetson Orin Nano and a core-image-sato image the copy and tar commands look like the following:
cp $YOCTO_DIR/build/tmp/deploy/images/jetson-orin-nano-devkit/core-image-sato-jetson-orin-nano-devkit.tegraflash.tar.gz .
tar -xvf core-image-sato-jetson-orin-nano-devkit.tegraflash.tar.gz
Before starting the flashing process, you need to put the board in recovery mode. In the case of the Jetson Orin Nano, you need to short pins 9 and 10 in J14 header, the pins are labeled as FC_REC and GND respectively. After shorting the pins, you can plug the power adapter to the board and connect the board to your host machine by using the USB C port in the board. You can verify that the board is in recovery mode with the lsusb command output, for example:
lsusb #Example output:
Bus 003 Device 029: ID 0955:7523 NVIDIA Corp. APX
After that, you can start the flashing process by executing the following command:
sudo ./doflash.sh
When the flashing process is completed, you can disconnect the board from the host machine. Make sure of removing the power cord from the board and remove the jumper you used to short the recovery mode pins. Then you can connect your board to a monitor and connect a mouse and keyboard to it. You can verify that the mender implementation is correct by running the following commands:
systemctl is-active mender-authd #Example output:
active
systemctl is-enabled mender-authd #Example output:
enabled
systemctl is-active mender-updated #Example output:
active
systemctl is-enabled mender-updated #Example output:
enabled
mender-update -v #Example output:
4.0.4
Configuring the update server
With the current configuration Mender will run as a service. This means that updates can be triggered manually but the system will also periodically check for updates, and then will apply the update if available. By default, Mender uses the path https://docker.mender.io/ as the update server, checks for updates every 30 minutes and sends inventory data to the update server every 8 hours and each time the device boots or an update is applied.
To configure the update server and the polling intervals you can add the following variables to your local.conf file:
MENDER_SERVER_URL = "https://mender.example.com" #Update server
MENDER_UPDATE_POLL_INTERVAL_SECONDS = "1800" #How frequently the system checks for updates
MENDER_INVENTORY_POLL_INTERVAL_SECONDS = "28800" #How often inventory data is send to the server
For the polling intervals, the corresponding variables are set in seconds. As an example, if you set MENDER_UPDATE_POLL_INTERVAL_SECONDS to 3600 the device will check for updates each hour. On the other hand, is also possible to disable Mender as a service so that updates can only be triggered manually and the device will not constantly check for updates and send data to a server. If you want to do this add the next variable to your local.conf file:
MENDER_FEATURES_DISABLE:append = " mender-systemd"
After that your device will use Mender in the standalone mode.
Full RootFS updates
After the Yocto build has been configured to support mender, a .mender will be automatically generated and this file will be used to perform a full update of the system. This section will show how to trigger an update in your board manually. As a first step, put the .mender file in a USB drive and plug the drive into your board. In order to read the content of the drive in your board, the drive needs to be mounted; in case it is not automatically mounted, you can do it with the following commands:
#Verify the name of the drive:
lsblk #Example output:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 14.7G 0 disk
`-sda1 8:1 1 14.7G 0 part
#Mount the drive:
sudo mkdir /media/usb
sudo mount /dev/sda1 /media/usb
cd /media/usb
Applying the update
After mounting the device you can apply the update by running mender-update install $ARTIFACT and then mender-update commit to confirm that you want to install the update or mender-update rollback to cancel the update. As an example:
#Verify the current partition before applying the update:
mount | grep 'on / ' #Example output:
/dev/mmcblk0p1 on / type ext4 (rw,relatime)
#Apply the update:
mender-update install core-image-sato-jetson-orin-nano-devkit-20250207004007.mender
mender-update commit
reboot
After the reboot, the update will be applied, and the progress will be shown as in the following image, and you can verify that after the update was applied and the system boots, the second partition is being used by the system:
#Verify the second partition is used:
mount | grep 'on / ' #Example output:
/dev/mmcblk0p2 on / type ext4 (rw,relatime)
If the update process fails while the update is being applied, for example, if the board loses the power supply after rebooting and the update progress bar is being shown, the Mender Rollback process will be applied. This ensures that the system will boot normally after the failed update and is not left in an unusable state. Suppose the system was using partition B and an update was being applied. In that case, this means that the update is applied to partition A, so if the update fails, the board will keep using partition B after rebooting, as before the update was started.
Rolling back the update
If you wish to rolled back the update, Mender will handle all the job for you. You can roll back and update before committing or after having committed the update and rebooted the board so that the update is fully applied. If you wish to roll back simply execute the following command:
mender-update rollback
This is going to roll back the last update that was applied by Mender. Keep in mind that if you had already rebooted the board, you will need to reboot the board again after rolling back the update so that the board switches to the original partition. In this case simply execute the reboot command after rolling back the update.
Artifact updates
It is also possible to manually create artifacts to update the Operating System or single application files. To do so, you need to use the Mender artifact tool in the host machine that you will use to create the artifact. You can obtain the tool here.
RootFS artifacts
To create an artifact for a full RootFS update you can use the following command after installing the tool:
mender-artifact write rootfs-image \ -t jetson-orin-nano-devkit \ -n release-1 \ --software-version rootfs-v1 \ -f rootfs.ext4 \ -o artifact.mender
Where rootfs.ext4 corresponds to the filesystem image, and:
- -t: specifies the compatible device. In this case, we are creating the artifact for the jetson-orin-nano-devkit.
- -n: this is the name of the artifact.
- --software-version: corresponds to the string software version.
- -o: path of the output file.
Application updates
It is also possible to update a single file or a directory. For the first case you need to use the single artifact generation tool, you can obtain the script with the following commands:
curl -O https://raw.githubusercontent.com/mendersoftware/mender/4.0.5/support/modules-artifact-gen/single-file-artifact-gen
chmod +x single-file-artifact-gen
Create a file to deploy:
echo "File created by Mender single-file Update Module!" > my_update_file.txt
Then, create the artifact with the following command:
./single-file-artifact-gen \ --device-type jetson-orin-nano-devkit \ -o file-update.mender \ -n test-update-1.0 \ --dest-dir /root \ my_update_file.txt
Where:
- -n: The name of the Mender Artifact
- --device-type: The compatible device type of this Mender Artifact
- -o: The path where to place the output Mender Artifact. This should always have a .mender suffix
- --dest-dir The path on target device where FILE will be installed.
If the file does not exists it will be created, and if the file already existed it will be overwritten with the file you used to create the artifact. In case you want to perform an update for a directory you need to use directory artifact generation tool, use the following commands to obtain it:
curl -O https://raw.githubusercontent.com/mendersoftware/mender/4.0.5/support/modules-artifact-gen/directory-artifact-gen
chmod +x directory-artifact-gen
After that, you can create the artifact to perform a directory update with the next command:
./directory-artifact-gen \ --device-type jetson-orin-nano-devkit \ -o directory-update.mender \ -n test-update-1.0 \ --dest-dir /root/update-directory \ update-directory
DTB artifacts
You can use Mender to update the DTB in your Jetson system, to do so you can use the single artifact generation tool to update your .dtb file and the extlinux.conf file to use your new DTB. It is important to keep in mind that the mender-artifact tool does not allow the use of special characters in file names, so the name of your DTB file can not contain characters such as "&+@". After you compile your new DTB you can create the artifact in the same way as it is done for single files, for example:
./single-file-artifact-gen \ --device-type jetson-orin-nano-devkit \ -o dtb-update.mender \ -n test-update-1.0 \ --dest-dir /boot \ update.dtb
Remember that the artifact creation script must be located in the same path as your .dtb file. Then modify the extlinux.conf, change the FDT line to use the updated file:
FDT /boot/update.dtb
Then create the artifact for the extlinux.conf file:
./single-file-artifact-gen \ --device-type jetson-orin-nano-devkit \ -o extlinux-update.mender \ -n test-update-1.0 \ --dest-dir /boot/extlinux \ extlinux.conf
After this you can apply both artifacts, reboot the board and after rebooting the board will be using the new device tree.
Bootloader artifacts
Another type of update you can perform is changing the bootloader configuration parameters or the bootloader itself if you are using the UEFI bootloader. Bootloader configuration can be changed by modifying the extlinux.conf file, to do this you can create a new file with the parameters you want to change, and then create the artifact and apply it as was done in the previous section when updating the device tree used by the system.
If you want to update the bootloader itself, in the case of the Jetson Orin Nano with the current configuration, the file used is located in /boot/efi/EFI/BOOT/. You can create an artifact for the bootaa64.efi file with the following command using the modified bootloader file:
./single-file-artifact-gen \ --device-type jetson-orin-nano-devkit \ -o bootloader-update.mender \ -n test-update-1.0 \ --dest-dir /boot/efi/EFI/BOOT \ bootaa64.efi
After applying the artifact you can reboot the board so that it boots using the new bootloader.
Kernel Image and DTB Partition artifacts
It is also possible to use Mender to update the Kernel Image and DTB in the default JetPack partitions if you desire to use them instead of the ones in the boot partition. To do this, you need to create a custom Mender update module that allows you to write the desired partition with the new content.
The update module is divided in two parts, one will go to the target machine and it tells the machine how to apply the received update if it is of the partition type, and the other part goes in the host machine and it is used to create the update artifact. For the target machine part, create a file called partition with the following content:
#!/bin/sh
set -e
STATE="$1"
FILES="$2"
dest_part_file="$FILES"/files/dest_part
filename_file="$FILES"/files/filename
case "$STATE" in
NeedsArtifactReboot)
echo "Yes"
;;
SupportsRollback)
echo "Yes"
;;
ArtifactInstall)
dest_part=$(cat "${dest_part_file}")
filename=$(cat "${filename_file}")
test -z "$dest_part" -o -z "$filename" && \
echo "Fatal error: dest_part or filename are undefined." >&2 && exit 1
#Create backup for original partition content
mkdir -p /root/backup_"$dest_part"
sudo dd if=/dev/"$dest_part" of=/root/backup_"$dest_part"/"$dest_part".backup bs=100k
#Update partition
sudo dd if="$FILES"/files/"${filename}" of=/dev/"$dest_part" bs=100k
;;
ArtifactRollback)
dest_part=$(cat "${dest_part_file}")
filename=$(cat "${filename_file}")
test -f /root/backup_"$dest_part"/"$dest_part".backup || exit 0
#Restore partition
sudo dd if=/root/backup_"$dest_part"/"$dest_part".backup of=/dev/"$dest_part" bs=100k
;;
esac
exit 0
This module uses the dd utility to create a back up of the target partition in case you decide to rollback the update and then it writes the partition with the selected update file. The module needs to be placed in the path /usr/share/mender/modules/v3 of the target machine, you create a single-file update as previously describe to deploy the module in your machine. Then, for the partition artifact generation tool create a file in your host machine called partition-artifact-gen with the following content:
#!/bin/sh
set -e
show_help() {
cat << EOF
Simple tool to generate Mender Artifact suitable for partition Update Module
Usage: $0 [options] file [-- [options-for-mender-artifact] ]
Options: [ -n|artifact-name -t|--device-type -d|--dest-part --software-name --software-version --software-filesystem -o|--output_path -h|--help ]
--artifact-name - Artifact name
--device-type - Target device type identification (can be given more than once)
--dest-part - Target destination partition where to deploy the update
--software-name - Name of the key to store the software version: rootfs-image.NAME.version,
instead of rootfs-image.single-file.version
--software-version - Value for the software version, defaults to the name of the artifact
--software-filesystem - If specified, is used instead of rootfs-image
--output-path - Path to output file. Default: file-install-artifact.mender
--help - Show help and exit
file - Single file to bundle in the update
Anything after a '--' gets passed directly to the mender-artifact tool.
EOF
}
show_help_and_exit_error() {
show_help
exit 1
}
check_dependency() {
if ! which "$1" > /dev/null; then
echo "The $1 utility is not found but required to generate Artifacts." 1>&2
return 1
fi
}
if ! check_dependency mender-artifact; then
echo "Please follow the instructions here to install mender-artifact and then try again: https://docs.mender.io/downloads#mender-artifact" 1>&2
exit 1
fi
device_types=""
artifact_name=""
dest_part=""
output_path="single-file-artifact.mender"
file=""
passthrough_args=""
while [ -n "$1" ]; do
case "$1" in
--device-type | -t)
if [ -z "$2" ]; then
show_help_and_exit_error
fi
device_types="$device_types $1 $2"
shift 2
;;
--artifact-name | -n)
if [ -z "$2" ]; then
show_help_and_exit_error
fi
artifact_name=$2
shift 2
;;
--dest-part | -d)
if [ -z "$2" ]; then
show_help_and_exit_error
fi
dest_part=$2
shift 2
;;
--software-name | --software-version | --software-filesystem)
if [ -z "$2" ]; then
show_help_and_exit_error
fi
passthrough_args="$passthrough_args $1 $2"
shift 2
;;
--output-path | -o)
if [ -z "$2" ]; then
show_help_and_exit_error
fi
output_path=$2
shift 2
;;
-h | --help)
show_help
exit 0
;;
--)
shift
passthrough_args="$passthrough_args $@"
break
;;
-*)
echo "Error: unsupported option $1"
show_help_and_exit_error
;;
*)
if [ -n "$file" ]; then
echo "File already specified. Unrecognized argument \"$1\""
show_help_and_exit_error
fi
file="$1"
shift
;;
esac
done
if [ -z "${artifact_name}" ]; then
echo "Artifact name not specified. Aborting."
show_help_and_exit_error
fi
if [ -z "${device_types}" ]; then
echo "Device type not specified. Aborting."
show_help_and_exit_error
fi
if [ -z "${dest_part}" ]; then
echo "Destination dir not specified. Aborting."
show_help_and_exit_error
fi
if [ -z "${file}" ]; then
echo "File not specified. Aborting."
show_help_and_exit_error
fi
# Create tarball, accepts single file
filename=""
if [ -e "${file}" ]; then
if [ -f "${file}" ]; then
filename=$(basename $file)
else
echo "Error: \"${file}\" is not a regular file. Aborting."
exit 1
fi
else
echo "Error: File \"${file}\" does not exist. Aborting."
exit 1
fi
# Create required files for the Update Module
tmpdir=$(mktemp -d)
dest_part_file="$tmpdir/dest_part"
filename_file="$tmpdir/filename"
permissions_file="$tmpdir/permissions"
# Create dest_part file in plain text
echo "$dest_part" > $dest_part_file
# Create single_file file in plain text
echo "$filename" > $filename_file
STAT_HAS_f=1;
stat -f %A "${file}" >/dev/null 2>&1 || STAT_HAS_f=0;
# Create permissions file in plain text
if [ $STAT_HAS_f -eq 1 ]; then
stat -f %A "${file}" > $permissions_file
else
stat -c %a "${file}" > $permissions_file
fi
mender-artifact write module-image \
-T partition \
$device_types \
-o "$output_path" \
-n "$artifact_name" \
-f "$dest_part_file" \
-f "$filename_file" \
-f "$permissions_file" \
-f "$file" \
$passthrough_args
rm -rf $tmpdir
echo "Artifact $output_path generated successfully:"
mender-artifact read $output_path
exit 0
Then you can create an update artifact for the Kernel Image or DTB with the following command, make sure to use the correct update file and target partition:
./partition-artifact-gen --device-type jetson-orin-nano-devkit -o dtb-update.mender -n test-update-1.0 --dest-part mmcblk0p4 update.dtb
Managing updates with the Mender server
If you do not wish to create a custom update server for your devices, you can use the official server provided by Mender. This server will allow you to manage and deploy updates for specific device types or for an specific device, and there you can remotely control the installation and roll back updates. The Mender provided server can be used with no cost for up to 10 devices and 12 months. You can learn more about it in Mender's official documentation.
