View source for RidgeRun Yocto Developer Guide/How to Create Yocto Projects/Yocto Utils

<noinclude>
{{RidgeRun Yocto Developer Guide/Head
|previous=https://developer.ridgerun.com/wiki/index.php/RidgeRun_Yocto_Developer_Guide/How_to_Create_Yocto_Projects/Compile_RidgeRun_Products
|title=Utils|description=Here you will find useful tools such as repo, devtool, devshell or Yocto SDK that will help you and make things easier during the Yocto development process.}}
</noinclude>

{{DISPLAYTITLE:How to Create Yocto Projects - Yocto Utils|noerror}}

== Yocto Utils == 
This section will guide you through a set of tools that will make your development in Yocto easier, faster and more comfortable.

=== Repo ===

Repo is a tool built on top of Git. Repo helps manage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow. Repo is not meant to replace Git, only to make it easier to work with Git. The repo command is an executable Python script that you can put anywhere in your path.

==== Install repo ====

You can install it using apt: 

<syntaxhighlight lang="bash">
sudo apt-get install repo
</syntaxhighlight>

Or downloading it directly

1. Add repo script to bin directory in home.
<syntaxhighlight lang="bash">
mkdir -p ~/bin
curl http://commondatastorage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
chmod a+x ~/bin/repo
</syntaxhighlight>
2. Append the directory if it's not already in PATH
<syntaxhighlight lang="bash">
PATH=$PATH:~/bin
</syntaxhighlight>

==== How to use repo with Yocto ====

Repo uses an XML manifest in order to enumerate multiple remotes, repositories, branches and revisions, then it fetches them. For example, using a Boundary Devices repository with a manifest we can use:

<syntaxhighlight lang="bash">
cd <ANY_DIR>
repo init -u https://github.com/boundarydevices/boundary-bsp-platform -b dunfell
</syntaxhighlight>

With these options being the most important:

<syntaxhighlight lang="bash">
-u URL, --manifest-url=URL : manifest repository location
-b REVISION, --manifest-branch=REVISION : manifest branch or revision (use HEAD for default)
-m NAME.xml, --manifest-name=NAME.xml: initial manifest file (defaults to default.xml), you can also use this to specify a completely different manifest in your system and it will be used. 
</syntaxhighlight>

After doing repo init, you can sync it using:

<syntaxhighlight lang="bash">
repo sync
</syntaxhighlight>

This will generate a directory structure with the given repositories defined in the XML manifest, normally in Yocto it will create a '''sources''' directory with each layer, and a '''setup script''' to create a build. 

==== Analyzing the repo manifest ====

Take as an example the manifest located in: https://github.com/boundarydevices/boundary-bsp-platform/blob/gatesgarth/default.xml

===== Defaults =====

The default settings are set as:

<syntaxhighlight lang="bash">
<default sync-j="4" revision="gatesgarth"/>
</syntaxhighlight>

===== Remotes =====

Remotes are defined as below, they are named to be used later for each project: 
<syntaxhighlight lang="bash">
<remote fetch="https://git.yoctoproject.org/git" name="yocto"/>
<remote fetch="https://github.com/Freescale" name="freescale"/>
<remote fetch="https://github.com/openembedded" name="oe"/>
<remote fetch="https://github.com/boundarydevices" name="boundary"/>
</syntaxhighlight>

===== Projects =====

Project are basically the repositories to fetch, they will use the remote defined above, the name on the hosting server, and then clone them into a path. You can also use the revision which is the commit to checkout, and upstream which is the branch to use, this is not necessarily the same for all projects obviously. 
<syntaxhighlight lang="bash">
<project remote="yocto"     revision="6bd890d9e011014cf323e61267f8b256949d44aa" upstream="pyro" name="poky" path="sources/poky"/>
<project remote="yocto"     revision="83b779a7ef69006564fb8d27e65c31917bc3b6f0" upstream="pyro" name="meta-freescale" path="sources/meta-freescale"/>
<project remote="oe"        revision="5e82995148a2844c6f483ae5ddd1438d87ea9fb7" upstream="pyro" name="meta-openembedded" path="sources/meta-openembedded"/>
<project remote="freescale" revision="a73a14247afbbcf840bab7b76f953004fe109253" upstream="pyro" name="meta-freescale-3rdparty" path="sources/meta-freescale-3rdparty"/>
<project remote="freescale" revision="cd5c7a2539f40004f74126e9fdf08254fd9a6390" upstream="pyro" name="meta-freescale-distro" path="sources/meta-freescale-distro"/>
</syntaxhighlight>


You can also perform operations like copy files from the project directly after syncing:

<syntaxhighlight lang="bash">
<project remote="boundary" name="boundary-bsp-base" path="sources/base">
  <copyfile dest="README" src="README"/>
  <copyfile dest="setup-environment" src="setup-environment"/>
</project>
</syntaxhighlight>

==== Creating a manifest ====

You can create a manifest of your current build by using: 

<syntaxhighlight lang="bash">
repo manifest -r -o my_manifest.xml
</syntaxhighlight>

=== Devtool ===
Let's assume we are building for qemuarm64 and you have set the `$DEVDIR` variable.

First use devtool to set a new workspace for the recipe:

<syntaxhighlight lang="bash">devtool modify gst-perf</syntaxhighlight>

Source code now will be located in the new created folder:
<syntaxhighlight lang="bash">
ls $BUILDDIR/workspace/sources/gst-perf
</syntaxhighlight>

When you compile and install this recipe, these sources will be used, you can check it by running:

<syntaxhighlight lang="bash">
bitbake -c install gst-perf # Compilation and installation
tree $BUILDDIR/tmp/work/aarch64-poky-linux/gst-perf/1.0-r0/image # Files installed
</syntaxhighlight>

==== Devtool deploy ====
deploy method is agnostic to the platform and any paths, it takes into account everything and just needs the name of the recipe to update it in the target by using SSH

===== Setting up =====
<syntaxhighlight lang="bash">
cd $BUILDDIR
runqemu qemuarm64 core-image-training nographic
</syntaxhighlight>

Let's get the IP from the QEMU instance, inside the instance:
<syntaxhighlight lang="bash">
ifconfig eth0 | grep "inet " | awk -F'[: ]+' '{ print $4 }'
</syntaxhighlight>

Save this IP, we will use it later.

===== Testing =====
Modify the source code, for example the Description of the plugin, and then build again:
<syntaxhighlight lang="bash">
bitbake -c install gst-perf
</syntaxhighlight>

Then you can deploy the changes using devtool:

<syntaxhighlight lang="bash">
devtool deploy-target gst-perf root@$QEMU_IP
</syntaxhighlight>

You can see the new version by running in the QEMU:

<syntaxhighlight lang="bash">
ldconfig
gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1
</syntaxhighlight>

You can also undeploy the changes
<syntaxhighlight lang="bash">
undeploy-target gst-perf root@$QEMU_IP
</syntaxhighlight>

===== Saving changes =====

To save the changes

<syntaxhighlight lang="bash">
devtool update-recipe --append=$DEVDIR/sources/meta-training/ gst-perf
devtool reset gst-perf
</syntaxhighlight>

Equivalent to:

<syntaxhighlight lang="bash">
devtool finish gst-pref $DEVDIR/sources/meta-training/
</syntaxhighlight>

Both commands will just save the changes made in the new commits made for the branch inside the <code>$BUILDDIR/workspace/sources/</code> folder

==== NFS ====
NFS method let's you mount the filesystem in your host and QEMU will load it, allowing changes to it in real time. Reference: https://docs.yoctoproject.org/dev-manual/qemu.html#running-under-a-network-file-system-nfs-server

===== Setting up =====
<syntaxhighlight lang="bash">

# Set variables
cd $BUILDDIR
export NFS_ROOT=`pwd`/test-nfs

# Install dependencies for NFS bitbake and QEMU support
sudo apt-get install rpcbind
bitbake meta-ide-support

# Set up NFS and start QEMU instance
runqemu-extract-sdk ./tmp/deploy/images/qemuarm64/core-image-training-qemuarm64.tar.bz2 $NFS_ROOT
runqemu qemuarm64 $NFS_ROOT nographic
</syntaxhighlight>

Now, every change to the filesystem inside `$NFS_ROOT` will be seen in real time on the QEMU instance running

===== Testing =====
Modify the source code, and then:
<syntaxhighlight lang="bash">
cd $BUILDDIR
bitbake -c install gst-perf
cp -r ./tmp/work/aarch64-poky-linux/gst-perf/1.0-r0/image/* $NFS_ROOT/
</syntaxhighlight>

===== Stopping =====
To stop the NFS sharing use:
<syntaxhighlight lang="bash">
runqemu-export-rootfs stop $NFS_ROOT
</syntaxhighlight>

Then remove the folders
<syntaxhighlight lang="bash">
rm -rf $NFS_ROOT
rm -rf $NFS_ROOT.pseudo_state
</syntaxhighlight>

=== Devtool for Kernel Development ===

==== Modify ====
Setup the kernel sources:
<syntaxhighlight lang="bash">
devtool modify virtual/linux
</syntaxhighlight>

Modify <code>$BUILDDIR/workspace/sources/linux-yocto/init/calibrate.c</code> by adding these printk to the <code>calibrate_delay</code> function.
<syntaxhighlight lang="bash">
     void calibrate_delay(void)
     {
         unsigned long lpj;
         static bool printed;
         int this_cpu = smp_processor_id();

         printk("*************************************\n");
         printk("*                                   *\n");
         printk("*        HELLO YOCTO KERNEL         *\n");
         printk("*                                   *\n");
         printk("*************************************\n");

     	if (per_cpu(cpu_loops_per_jiffy, this_cpu)) {
</syntaxhighlight>

{{message
|text=
For Boundary's i.MX use <code>devtool modify linux-boundary</code>, the kernel source code will be located at <code>workspace/sources/linux-boundary/</code>, and <code>calibrate.c</code> is located at: <code>$BUILDDIR/workspace/sources/linux-boundary/init/calibrate.c</code>.
}}

===== Deploy =====
We need to copy the image <code>/boot/</code>
<syntaxhighlight lang="bash">
scp tmp/work/qemuarm64-poky-linux/linux-yocto/5.4.153+git999-r0/image/boot/Image root@<QEMU_IP>:/boot/
</syntaxhighlight>

Then exit Qemu, and enter again. You should see the message from above while running:
<syntaxhighlight lang="bash">
dmesg | less
</syntaxhighlight>

{{message
|text=
For Boundary's board it's pretty more difficult when booting from an SD card. Follow these instructions to update the kernel:
<syntaxhighlight lang="bash">
# Search and set the IP of your board:
export IMX_IP=192.168.0.22

ssh root@$IMX_IP 'umount /mnt/mmcblk1p1'
ssh root@$IMX_IP 'mkdir -p /mnt/mmcblk1p1'
ssh root@$IMX_IP 'mount /dev/mmcblk1p1 /mnt/mmcblk1p1'

scp tmp/work/nitrogen6x-poky-linux-gnueabi/linux-boundary/5.4.80+git999-r0/image/boot/zImage* root@$IMX_IP:/mnt/mmcblk1p1

# Reboot
# dmesg | less
</syntaxhighlight>
}}

=== Devshell ===

==== Setting up ====

Remember the $DEVDIR folder? Get into there and get into the build directory if you haven't done so.

<syntaxhighlight lang="bash">
cd $DEVDIR
source ./sources/poky/oe-init-build-env training-build/
</syntaxhighlight>

We are going to modify the perf element. So get into the board and check the current Description for this element.

<syntaxhighlight lang="bash">
gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1
</syntaxhighlight>

==== Usage ====

If you added gst-perf to the workspace using devtool, first remove it: <code>devtool reset gst-perf</code>.

Let's start using devshell, execute:

<syntaxhighlight lang="bash">
bitbake gst-perf -c devshell
</syntaxhighlight>

This will open up the sources on the $S variable, normally $WORKDIR/git <ref>https://intranet.ridgerun.com/wiki/index.php?title=Yocto_Tips_%26_Tricks#Normal_Recipes </ref> if using git. 

===== Updating the code =====

0. Check if the source code is using git or not, if not and you are planning to make changes that you are planning to capture you will need a new git local repo or to use with quilt, in any case you need to either start the repo or create a new quilt patch before modifying the files. More information in the [[Yocto_Tips_&_Tricks_2#Saving_changes_2|Saving changes]] section below.  

1. Modify the code, for example update the Description for the element:

Use vim or your editor, and search for gst_element_class_set_static_metadata function. Modify it.

===== Compiling =====

1. Run by hand:
<syntaxhighlight lang="bash">
./autogen.sh
./configure --prefix `pwd`/../image/usr/ --libdir `pwd`/../image/usr/lib --build=x86_64-linux --host=aarch64-poky-linux
make && make install
</syntaxhighlight>

Then all the will be installed at first level inside

<syntaxhighlight lang="bash">
tree ../image/
../image/
└── usr
    └── lib
        └── gstreamer-1.0
            ├── libgstperf.la
            └── libgstperf.so
</syntaxhighlight>

==== Testing ====

You can test by updating the binaries and files in the board or in the QEMU instance or

<syntaxhighlight lang="bash">
scp -r ../image/* root@192.168.7.2:/
</syntaxhighlight>

Look that the copies need to be done by hand, different to devtool. 

Then run on the board:

<syntaxhighlight lang="bash">
ldconfig
gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1
</syntaxhighlight>

==== Saving changes ====
====== Create the patch ======

======= git =======
Create patch, if using git for example:

<syntaxhighlight lang="bash">
git add plugins/gstperf.c
git commit -m "update"
git format-patch HEAD~1
</syntaxhighlight>

====== quilt ======

<syntaxhighlight lang="bash">
quilt new 0001-update.patch
quilt add plugins/gstperf.c
edit plugins/gstperf.c
</syntaxhighlight>

Modify file and then

<syntaxhighlight lang="bash">
quilt refresh
</syntaxhighlight>

Now you can copy the quilt generated file. 

===== Update recipe with patch =====

Save the path of the patch file, using pwd and the name of it:

<syntaxhighlight lang="bash">
echo `pwd`/0001-update.patch
</syntaxhighlight>

Update recipe with patch:

<syntaxhighlight lang="bash">
cd $DEVDIR/sources/meta-training/recipes-multimedia/gstreamer
mkdir files
cp $PATCH_PATH files/
</syntaxhighlight>

And append it to the SRC_URI for the recipe manually. 

<syntaxhighlight lang="bash">
cd $DEVDIR/sources/meta-training/recipes-multimedia/gstreamer/
</syntaxhighlight>

And then add to the gst-perf_%.bbapped:

<syntaxhighlight lang="bash">
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += "file://0001-update.patch \
           "
</syntaxhighlight>

Now to test you may copy the values from the image folder in the workdir, first get this value doing a grep: 
<syntaxhighlight lang="bash">
bitbake -e gst-perf | grep ^D=
</syntaxhighlight>

Then just the same copy and test commands. 

NOTE: Take a look at how many manual steps are needed in order to achieve the same that the devtool is doing for us.

==== run.do_* scripts ====

==== Setting up ====

Remember the $DEVDIR folder again? Get into there and get into the build directory if you haven't done so.

<syntaxhighlight lang="bash">
cd $DEVDIR
source ./sources/poky/oe-init-build-env training-build/
</syntaxhighlight>

We are going to modify the perf element again. So get into the board and check the current Description for this element.

<syntaxhighlight lang="bash">
gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1
</syntaxhighlight>

==== Usage ====

To start, we need to get into the code of the package, for this check where it is located by using:

<syntaxhighlight lang="bash">
bitbake -e gst-perf | grep ^S=
</syntaxhighlight>

And move to the directory it is pointing.

===== Updating the code =====

0. Check if the source code is using git or not, if not and you are planning to make changes that you are planning to capture you will need a new git local repo or to use with quilt, in any case you need to either start the repo or create a new quilt patch before modifying the files. More information in the [[Yocto_Tips_&_Tricks_2#Saving_changes_2|Saving changes]] section of devshell, same applies here.  

1. Modify the code, for example update the Description for the element:

Use vim or your editor, and search for gst_element_class_set_static_metadata function. Modify it.

===== Compiling =====
This is the easy part, you can just run the compilation and install scripts autogenerated by bitbake:

<syntaxhighlight lang="bash">
../temp/run.do_compile
../temp/run.do_install
</syntaxhighlight>

If these scripts don't exist, execute the install task for the desired recipe.

==== Testing ====

You can test by updating the binaries and files in the board or in the QEMU instance or

<syntaxhighlight lang="bash">
scp -r ../image/* root@192.168.7.2:/
</syntaxhighlight>

Look that the copies need to be done by hand, different to devtool. 

Then run on the board:

<syntaxhighlight lang="bash">
ldconfig
gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1
</syntaxhighlight>

=== Yocto Extensible SDK ===

Building Yocto can take time. There's an alternative for bigger teams working on a Yocto image, so that only one person builds Yocto, and the rest of the team uses the resulting build to develop their applications. This is the so-called Yocto SDK.

This way, Yocto SDK provides a simpler environment to application engineers, to avoid all the complexity and time spent to build the whole Yocto image.

Yocto provides two SDK types: Standard and Extensible. We will build the extensible SDK, as it contains more useful features as can be seen in the following table:

{| class="wikitable"
|-
! Feature !! Standard SDK !! Extensible SDK
|-
| Toolchain || Yes || Yes
|-
| Debugger || Yes || Yes
|-
| Size || 100+MB || 1+GB
|-
| devtool || No || Yes
|-
| Build Images || No || Yes
|-
| Updateable || No || Yes
|-
| Managed Sysroot || No || Yes
|-
| Installed Packages || No || Yes
|-
| Construction || Packages || Shared State
|}

==== Build the SDK ====
To build the <code>Extensible SDK</code>:
<syntaxhighlight lang="bash">
bitbake core-image-training -c populate_sdk_ext
</syntaxhighlight>

==== Install the SDK ====
The SDK can be found inside the <code>$BUILDDIR/tmp/deploy/sdk</code> directory.

To install the SDK:
<syntaxhighlight lang="bash">
cd $BUILDDIR
./tmp/deploy/sdk/poky-glibc-x86_64-core-image-training-aarch64-qemuarm64-toolchain-ext-3.1.11.sh
</syntaxhighlight>

You will be asked for the directory where the SDK will be installed. From now on, we are going to call this directory <code>$SDK_DIR</code>

==== Initialize the SDK ====
Every time you want to use the SDK, you have to source the environment script:

<syntaxhighlight lang="bash">
cd $SDK_DIR
source environment-setup-aarch64-poky-linux
</syntaxhighlight>

Check that you get the following text after the source command:
<syntaxhighlight lang="bash">
SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.
</syntaxhighlight>

==== Extended SDK with Devtool ====

First we need to create the image with
<syntaxhighlight lang="bash">
devtool build-image core-image-training
</syntaxhighlight>

To add the <code>gst-perf</code> sources to the workspace, run:
<syntaxhighlight lang="bash">
devtool modify gst-perf
</syntaxhighlight>

Now you can modify it at:
<syntaxhighlight lang="bash">
workspace/sources/gst-perf/
</syntaxhighlight>

Then, to build the changes:
<syntaxhighlight lang="bash">
devtool deploy-target gst-perf root@$QEMU_IP
</syntaxhighlight>

{{message
|text=You must have Qemu (or the board) up and running, and change $QEMU_IP with the proper IP.
}}

<code>devtool</code> knows which files need to be updated and where they need to be installed, so you don't have to worry.

<noinclude>
{{RidgeRun Yocto Developer Guide/Foot
|previous=https://developer.ridgerun.com/wiki/index.php/RidgeRun_Yocto_Developer_Guide/How_to_Create_Yocto_Projects/Compile_RidgeRun_Products
}}
</noinclude>