View source for CUDA ISP for NVIDIA Jetson/Getting Started/Building custom v4l2src element

<noinclude>
{{CUDA ISP for NVIDIA Jetson/Head|previous=Getting Started/Building CUDA ISP for NVIDIA Jetson|next=User Manual|description=Learn how to build a custom v4l2src for CUDA ISP. Follow the patching process, compile the element, and configure it for use.}}
</noinclude>

{{DISPLAYTITLE:Building custom v4l2src|noerror}}

== Introduction ==

In NVIDIA Jetson, it is common to have MIPI cameras that provide raw (or bayer) images with bits per pixel (bpp) greater than 8, and <code>nvarguscamerasrc</code> (provided in the Linux for Tegra) is one of the GStreamer elements utilised to capture, bypassing the image to the Image Signal Processor embedded into the platform. However, to capture the raw image, it is required to use <code>v4l2src</code> (provided by GStreamer).

Nonetheless, <code>v4l2src</code> only supports raw with bpp = 8, which is unsuitable for capturing most cameras in the Jetson context. We have fixed this issue by applying a patch to the element. You can contact us ([mailto:support@ridgerun.com support@ridgerun.com]) in order to get the binary if you don't want to go through the entire compilation process.

There are two ways to get v4l2src 10-bit bayer support:

# Compiling our experimental v4l2src element (recommended for testing)
# Patch the v4l2src on GStreamer (valid for GStreamer 1.14)

== 1. Building the experimental RidgeRun v4l2src ==

We have a port of the Video For Linux 2 (v4l2) GStreamer plug-ins with support from GStreamer 1.14 up to 1.18 and 10-bit bayer for MIPI cameras. It is possible to evaluate a non-production version of the video source using a patched version that is available in [https://gitlab.ridgerun.com/open/gstreamer/gst-v4l2src this repo].

{{Colored box|background-content-color=#EDF1F7|background-title-color=#6586B9|title-color=#FFFFFF|title='''Note'''|icon=notice-icon-white.png
  |style=overflow:hidden;
  |content=
This element is a reference implementation. Please adapt it to your GStreamer version based on the changes from this repo before deploying to production.
}}

Please consider the following instructions:

1. Clone the repo:

<syntaxhighlight lang=bash>
git clone https://gitlab.ridgerun.com/open/gstreamer/gst-v4l2src.git -b bpp-10
cd gst-v4l2src
</syntaxhighlight>

2. Configure the build

<syntaxhighlight lang=bash>
# Create the folder
mkdir -p ${HOME}/custom-gst-v4l2src

# Configure the path in build, compile and install
meson builddir --prefix=${HOME}/custom-gst-v4l2src --libdir=${HOME}/custom-gst-v4l2src/lib
ninja -C builddir install
</syntaxhighlight>

3. Export the environment variables

<syntaxhighlight lang=bash>
export LD_LIBRARY_PATH=${HOME}/custom-gst-v4l2src/lib:${LD_LIBRARY_PATH}
export GST_PLUGIN_PATH=${HOME}/custom-gst-v4l2src/lib/gstreamer-1.0:${GST_PLUGIN_PATH}
</syntaxhighlight>

''These exports can be used in the .bashrc to automate them''.

4. Test the installation

<syntaxhighlight lang=bash>
gst-inspect-1.0 rrv4l2src
</syntaxhighlight>

It shall show a valid inspect.

{{Colored box|background-content-color=#EDF1F7|background-title-color=#6586B9|title-color=#FFFFFF|title='''Note'''|icon=notice-icon-white.png
  |style=overflow:hidden;
  |content=Notice that the name of the element is rrv4l2src and not v4l2src
}}

== 2. Patching process ==

<pre style="background-color:cyan">
Please, follow these steps only if you want to capture from GStreamer. Otherwise, these steps are optional
</pre>

These steps are intended for the <code>v4l2src</code> element included in '''GStreamer 1.14'''. For later versions, the patch might not work.

To provide <code>v4l2src</code> with support for raw 10 (often required by MIPI-CSI cameras), please, follow these steps.

* Clone the GStreamer Plug-ins Good according to your GStreamer version

<pre>
VERSION=1.14
git clone https://gitlab.freedesktop.org/gstreamer/gst-plugins-good -b $VERSION
</pre>

{{Ambox
|type=notice
|small=left
|issue='''Note''': Change <code>VERSION</code> according to your version. If the GStreamer version is later than 1.19, please, use the [https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-good GStreamer Monorepo].
|style=width:unset;
}}


* Go to the root of the GStreamer Plug-ins Good:

<pre>
cd gst-plugins-good
</pre>

* Create a file named <code>0001-v4l2src-add-bayer10-support.patch</code> with the following contents:

<pre>
From 62bebf0455b636ea4147b9564b936f47fdebbb78 Mon Sep 17 00:00:00 2001
From: Edison Fernandez <edison.fernandez@ridgerun.com>
Date: Mon, 31 Aug 2020 17:53:54 -0600
Subject: [PATCH] v4l2src add bayer10 support

---
 sys/v4l2/gstv4l2object.c | 44 ++++++++++++++++++++++++++++++++++++--------
 1 file changed, 36 insertions(+), 8 deletions(-)

diff --git a/sys/v4l2/gstv4l2object.c b/sys/v4l2/gstv4l2object.c
index c074196..9f8dab6 100755
--- a/sys/v4l2/gstv4l2object.c
+++ b/sys/v4l2/gstv4l2object.c
@@ -162,6 +162,10 @@ static const GstV4L2FormatDesc gst_v4l2_formats[] = {
   {V4L2_PIX_FMT_SGBRG8, TRUE, GST_V4L2_RAW},
   {V4L2_PIX_FMT_SGRBG8, TRUE, GST_V4L2_RAW},
   {V4L2_PIX_FMT_SRGGB8, TRUE, GST_V4L2_RAW},
+  {V4L2_PIX_FMT_SBGGR10, TRUE, GST_V4L2_RAW},
+  {V4L2_PIX_FMT_SGBRG10, TRUE, GST_V4L2_RAW},
+  {V4L2_PIX_FMT_SGRBG10, TRUE, GST_V4L2_RAW},
+  {V4L2_PIX_FMT_SRGGB10, TRUE, GST_V4L2_RAW},
 
   /* compressed formats */
   {V4L2_PIX_FMT_MJPEG, FALSE, GST_V4L2_CODEC},
@@ -1124,6 +1128,10 @@ gst_v4l2_object_format_get_rank (const struct v4l2_fmtdesc *fmt)
     case V4L2_PIX_FMT_SGBRG8:
     case V4L2_PIX_FMT_SGRBG8:
     case V4L2_PIX_FMT_SRGGB8:
+    case V4L2_PIX_FMT_SBGGR10:
+    case V4L2_PIX_FMT_SGBRG10:
+    case V4L2_PIX_FMT_SGRBG10:
+    case V4L2_PIX_FMT_SRGGB10:
       rank = BAYER_BASE_RANK;
       break;
 
@@ -1423,6 +1431,7 @@ static GstStructure *
 gst_v4l2_object_v4l2fourcc_to_bare_struct (guint32 fourcc)
 {
   GstStructure *structure = NULL;
+  guint bayer_bpp = 8;
 
   switch (fourcc) {
     case V4L2_PIX_FMT_MJPEG:   /* Motion-JPEG */
@@ -1546,15 +1555,23 @@ gst_v4l2_object_v4l2fourcc_to_bare_struct (guint32 fourcc)
       break;
     case V4L2_PIX_FMT_WNVA:    /* Winnov hw compres */
       break;
+    case V4L2_PIX_FMT_SBGGR10:
+    case V4L2_PIX_FMT_SGBRG10:
+    case V4L2_PIX_FMT_SGRBG10:
+    case V4L2_PIX_FMT_SRGGB10:
+      bayer_bpp = 10;
     case V4L2_PIX_FMT_SBGGR8:
     case V4L2_PIX_FMT_SGBRG8:
     case V4L2_PIX_FMT_SGRBG8:
     case V4L2_PIX_FMT_SRGGB8:
       structure = gst_structure_new ("video/x-bayer", "format", G_TYPE_STRING,
-          fourcc == V4L2_PIX_FMT_SBGGR8 ? "bggr" :
-          fourcc == V4L2_PIX_FMT_SGBRG8 ? "gbrg" :
-          fourcc == V4L2_PIX_FMT_SGRBG8 ? "grbg" :
-          /* fourcc == V4L2_PIX_FMT_SRGGB8 ? */ "rggb", NULL);
+          fourcc == V4L2_PIX_FMT_SBGGR8 || fourcc == V4L2_PIX_FMT_SBGGR10 ? "bggr" :
+          fourcc == V4L2_PIX_FMT_SGBRG8 || fourcc == V4L2_PIX_FMT_SGBRG10 ? "gbrg" :
+          fourcc == V4L2_PIX_FMT_SGRBG8 || fourcc == V4L2_PIX_FMT_SGRBG10 ? "grbg" :
+          /* fourcc == V4L2_PIX_FMT_SRGGB8 || fourcc == V4L2_PIX_FMT_SRGGB10 ? */ "rggb", NULL);
+      if (bayer_bpp == 10) {
+        gst_structure_set (structure, "bpp", G_TYPE_INT, 10, NULL);
+      }
       break;
     case V4L2_PIX_FMT_SN9C10X:
       structure = gst_structure_new_empty ("video/x-sonix");
@@ -1650,6 +1667,13 @@ gst_v4l2_object_get_caps_helper (GstV4L2FormatFlags flags)
       }
 
       switch (gst_v4l2_formats[i].format) {
+        case V4L2_PIX_FMT_SBGGR10:
+        case V4L2_PIX_FMT_SGBRG10:
+        case V4L2_PIX_FMT_SGRBG10:
+        case V4L2_PIX_FMT_SRGGB10:
+          alt_s = gst_structure_copy (structure);
+          gst_structure_set (alt_s, "bpp", G_TYPE_INT, 10, NULL);
+          break;
         case V4L2_PIX_FMT_RGB32:
           alt_s = gst_structure_copy (structure);
           gst_structure_set (alt_s, "format", G_TYPE_STRING, "ARGB", NULL);
@@ -1886,15 +1910,19 @@ gst_v4l2_object_get_caps_info (GstV4l2Object * v4l2object, GstCaps * caps,
       fourcc = V4L2_PIX_FMT_VP9;
     } else if (g_str_equal (mimetype, "video/x-bayer")) {
       const gchar *format = gst_structure_get_string (structure, "format");
+      gint bpp;
+      if (!gst_structure_get_int (structure, "bpp", &bpp)) {
+        bpp = 8;
+      }
       if (format) {
         if (!g_ascii_strcasecmp (format, "bggr"))
-          fourcc = V4L2_PIX_FMT_SBGGR8;
+          fourcc = (bpp == 10)? V4L2_PIX_FMT_SBGGR10 : V4L2_PIX_FMT_SBGGR8;
         else if (!g_ascii_strcasecmp (format, "gbrg"))
-          fourcc = V4L2_PIX_FMT_SGBRG8;
+          fourcc = (bpp == 10)? V4L2_PIX_FMT_SGBRG10 : V4L2_PIX_FMT_SGBRG8;
         else if (!g_ascii_strcasecmp (format, "grbg"))
-          fourcc = V4L2_PIX_FMT_SGRBG8;
+          fourcc = (bpp == 10)? V4L2_PIX_FMT_SGRBG10 : V4L2_PIX_FMT_SGRBG8;
         else if (!g_ascii_strcasecmp (format, "rggb"))
-          fourcc = V4L2_PIX_FMT_SRGGB8;
+          fourcc = (bpp == 10)? V4L2_PIX_FMT_SRGGB10 : V4L2_PIX_FMT_SRGGB8;
       }
     } else if (g_str_equal (mimetype, "video/x-sonix")) {
       fourcc = V4L2_PIX_FMT_SN9C10X;
-- 
2.7.4
</pre>

* Apply the patch

<pre>
patch < 0001-v4l2src-add-bayer10-support.patch
</pre>

* Recompile according to the instructions (of the official repo)

In summary, the steps are:

<pre>
meson builddir --prefix=/usr --libdir=/usr/lib/$(uname -m)-linux-gnu/
ninja -C builddir
sudo ninja -C builddir install
</pre>

== Usage ==

After the installation, you can check that the element is installed by using:

<pre>
gst-launch-1.0 v4l2src ! "video/x-bayer,bpp=10" ! fakesink
</pre>

<noinclude>
{{CUDA ISP for NVIDIA Jetson/Foot|Getting Started/Building CUDA ISP for NVIDIA Jetson|User Manual}}
</noinclude>