View source for CUDA ISP for NVIDIA Jetson/Performance/GStreamer

<noinclude>
{{CUDA ISP for NVIDIA Jetson/Head|previous=Performance/Library|next=Contact Us|description=Evaluate CUDA ISP API performance using GstShark & GstPerf. Learn how to measure processing times & compare results on Jetson.}}
</noinclude>

{{DISPLAYTITLE:Measuring CUDA ISP API performance using GstShark and GstPerf|noerror}}

== GStreamer elements performance ==

To measure the performance, we have used two of our GStreamer tools: [[GstShark]] and [https://github.com/RidgeRun/gst-perf GstPerf]. 

For testing purposes, take into account the following points: 
* Maximum performance mode enabled: all cores and Jetson clocks enabled. 
* Jetpack 4.6
* A patch was applied to <code>v4l2src</code> to enable bayer10 captures. You can see how to apply the patch in this link: [[CUDA_ISP_for_NVIDIA_Jetson/Getting_Started/Building_custom_v4l2src_element|Apply patch to v4l2src]]

In summary:

<center>
{| class="wikitable" style="text-align:center;"
|-
! style="text-align:left;" | 
! colspan="6" style="font-weight:bold; background-color:#ffd6a5;" | Jetson Orin (2472x2064)
! colspan="4" style="font-weight:bold; background-color:#ffd7d7;" | Jetson Xavier AGX (+1080p)
! colspan="6" style="vertical-align:middle; font-weight:bold; background-color:#c8ffc6;" | Jetson Xavier NX (4K)
! colspan="6" style="vertical-align:middle; font-weight:bold; background-color:#cce7ff;" | Jetson Nano (4K)
|- style="font-weight:bold;"
| style="text-align:left;" | Element
| colspan="2" style="vertical-align:middle; background-color:#ffd6a5;" | cudashift
| colspan="2" style="vertical-align:middle; background-color:#ffd6a5;" | cudadebayer
| colspan="2" style="vertical-align:middle; background-color:#ffd6a5;" | cudaawb
| colspan="2" style="background-color:#ffd7d7; vertical-align:middle; " | cudadebayer
| colspan="2" style="background-color:#ffd7d7; vertical-align:middle; " | cudaawb
| colspan="2" style="vertical-align:middle; background-color:#c8ffc6;" | cudashift
| colspan="2" style="vertical-align:middle; background-color:#c8ffc6;" | cudadebayer
| colspan="2" style="vertical-align:middle; background-color:#c8ffc6;" | cudaawb
| colspan="2" style="vertical-align:middle; background-color:#cce7ff;" | cudashift
| colspan="2" style="vertical-align:middle; background-color:#cce7ff;" | cudadebayer
| colspan="2" style="vertical-align:middle; background-color:#cce7ff;" | cudaawb
|-
| style="font-weight:bold; text-align:left;" | Output
| colspan="2" style="vertical-align:middle; font-weight:bold; background-color:#ffd6a5;" | bayer 8
| style="vertical-align:middle; font-weight:bold; background-color:#ffd6a5;" | RGB
| style="vertical-align:middle; font-weight:bold; background-color:#ffd6a5;" | I420
| style="vertical-align:middle; font-weight:bold; background-color:#ffd6a5;" | RGB
| style="vertical-align:middle; font-weight:bold; background-color:#ffd6a5;" | I420
| style="font-weight:bold; background-color:#ffd7d7; text-align:left;" | RGB
| style="font-weight:bold; background-color:#ffd7d7; text-align:left;" | I420
| style="font-weight:bold; background-color:#ffd7d7; text-align:left;" | RGB
| style="font-weight:bold; background-color:#ffd7d7; text-align:left;" | I420
| colspan="2" style="vertical-align:middle; font-weight:bold; background-color:#c8ffc6;" | bayer 8
| style="vertical-align:middle; font-weight:bold; background-color:#c8ffc6;" | RGB
| style="vertical-align:middle; font-weight:bold; background-color:#c8ffc6;" | I420
| style="vertical-align:middle; font-weight:bold; background-color:#c8ffc6;" | RGB
| style="vertical-align:middle; font-weight:bold; background-color:#c8ffc6;" | I420
| colspan="2" style="vertical-align:middle; font-weight:bold; background-color:#cce7ff;" | bayer 8
| style="vertical-align:middle; font-weight:bold; background-color:#cce7ff;" | RGB
| style="vertical-align:middle; font-weight:bold; background-color:#cce7ff;" | I420
| style="vertical-align:middle; font-weight:bold; background-color:#cce7ff;" | RGB
| style="vertical-align:middle; font-weight:bold; background-color:#cce7ff;" | I420
|- style="text-align:left;"
| style="font-weight:bold;" | FPS
| colspan="2" style="vertical-align:middle; background-color:#ffd6a5;" | 1382
| style="vertical-align:middle; background-color:#ffd6a5;" | 1063
| style="vertical-align:middle; background-color:#ffd6a5;" | 861
| style="vertical-align:middle; background-color:#ffd6a5;" | 1800
| style="vertical-align:middle; background-color:#ffd6a5;" | 1022
| style="background-color:#ffd7d7;" | 539
| style="background-color:#ffd7d7;" | 458
| style="background-color:#ffd7d7;" | 752
| style="background-color:#ffd7d7;" | 473
| colspan="2" style="vertical-align:middle; background-color:#c8ffc6;" | 396
| style="vertical-align:middle; background-color:#c8ffc6;" | 228
| style="vertical-align:middle; background-color:#c8ffc6;" | 187
| style="vertical-align:middle; background-color:#c8ffc6;" | 370
| style="vertical-align:middle; background-color:#c8ffc6;" | 202
| colspan="2" style="vertical-align:middle; background-color:#cce7ff;" | 92
| style="vertical-align:middle; background-color:#cce7ff;" | 51
| style="vertical-align:middle; background-color:#cce7ff;" | 36
| style="vertical-align:middle; background-color:#cce7ff;" | 91
| style="vertical-align:middle; background-color:#cce7ff;" | 38
|- style="text-align:left;"
| style="font-weight:bold;" | Processing time (milliseconds)

| colspan="2" style="vertical-align:middle; background-color:#ffd6a5;" | 0.7237
| style="vertical-align:middle; background-color:#ffd6a5;" | 0.9411
| style="vertical-align:middle; background-color:#ffd6a5;" | 1.161
| style="vertical-align:middle; background-color:#ffd6a5;" | 0.5557
| style="vertical-align:middle; background-color:#ffd6a5;" | 0.9781
| style="background-color:#ffd7d7;" | 1.854
| style="background-color:#ffd7d7;" | 2.183
| style="background-color:#ffd7d7;" | 1.329
| style="background-color:#ffd7d7;" | 2.111
| colspan="2" style="vertical-align:middle; background-color:#c8ffc6;" | 2.522
| style="vertical-align:middle; background-color:#c8ffc6;" | 4.389
| style="vertical-align:middle; background-color:#c8ffc6;" | 5.353
| style="vertical-align:middle; background-color:#c8ffc6;" | 2.698
| style="vertical-align:middle; background-color:#c8ffc6;" | 4.952
| colspan="2" style="vertical-align:middle; background-color:#cce7ff;" | 10.886
| style="vertical-align:middle; background-color:#cce7ff;" | 19.484
| style="vertical-align:middle; background-color:#cce7ff;" | 27.696
| style="vertical-align:middle; background-color:#cce7ff;" | 10.961
| style="vertical-align:middle; background-color:#cce7ff;" | 26.022
|}
</center>

===Jetson Orin===
In this case, the patch for <code>v4l2src</code> was modified to enable captures in bayer 12.
The processing time and FPS were measured with an input image with 2472x2064 resolution from a camera sensor for all the elements. 

The following pipeline measured the processing time and FPS for the cudashift element. 
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=12, width=2472, height=2064, format=rggb' ! cudashift shift=5 ! fakesink
</syntaxhighlight>

The following pipeline was used to test the cudadebayer and cudaawb elements with an RGB image as output.
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=12, width=2472, height=2064, format=rggb' ! cudadebayer ! cudaawb ! 'video/x-raw, format=RGB' ! fakesink
</syntaxhighlight>
The following pipeline was used to test the cudadebayer and cudaawb elements with an I420 image as output.
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=12, width=2472, height=2064, format=rggb' ! cudadebayer ! cudaawb ! 'video/x-raw, format=I420' ! fakesink
</syntaxhighlight>
The following image shows a plot with the processing time for all elements:

<br>
<br><br>
[[File:ProctimeOrinms.png|1000px|frameless|center|CUDA ISP library ]]
<br>
<br><br>

=== Jetson Xavier AGX ===
The processing time and FPS were measured with an input image with +1080p resolution from a camera sensor for all the elements. 

The following pipeline was used to test the cudadebayer and cudaawb elements with an RGB image as output. 
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=1920, height=1200, format=grbg' ! cudadebayer ! cudaawb ! 'video/x-raw, format=RGB' ! fakesink
</syntaxhighlight>
The following pipeline was used for the cudadebayer and cudaawb elements with an I420 image as output. 
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=1920, height=1200, format=grbg' ! cudadebayer ! cudaawb ! 'video/x-raw, format=I420' ! fakesink
</syntaxhighlight>
The following image shows a plot with the processing time for all elements:

</center>
<br>
<br><br>
[[File:ProctimeAGXms.png|1000px|frameless|center|CUDA ISP library ]]
<br>
<br><br>

===Jetson Xavier NX===
The processing time and FPS were measured with an input image with 4K resolution from a camera sensor for all the elements. 

The following pipeline measured the processing time and FPS for the cudashift element. 
<<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, format=rggb' ! cudashift shift=5 ! fakesink
</syntaxhighlight>

The following pipeline was used to test the cudadebayer and cudaawb elements with an RGB image as output.
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudadebayer ! cudaawb ! fakesink
</syntaxhighlight>
The following pipeline was used to test the cudadebayer and cudaawb elements with an I420 image as output.
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudadebayer ! cudaawb ! 'video/x-raw, format=I420' ! fakesink
</syntaxhighlight>


The following image shows a plot with the processing time for all elements:

<br>
<br><br>
[[File:ProctimeNXms.png|1000px|frameless|center|CUDA ISP library ]]
<br>
<br><br>

===Jetson Nano===
The processing time and FPS were measured with an input image with 4K resolution from a camera sensor for all the elements.

The following pipeline measured the processing time and FPS for the cudashift element.
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, format=rggb' ! cudashift shift=0 ! fakesink
</syntaxhighlight>
he following pipeline was used to testing the cudadebayer and cudaawb elements with an RGB image as output.
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudadebayer ! cudaawb ! fakesink
</syntaxhighlight>
The following pipeline was used to test the cudadebayer and cudaawb elements with an I420 image as output.
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudadebayer ! cudaawb ! 'video/x-raw, format=I420' ! fakesink
</syntaxhighlight>
The following image shows a plot with the processing time for all elements:

<br>
<br><br>
[[File:ProctimeNanoms.png|1000px|frameless|center|CUDA ISP library ]]
<br>
<br><br>

===More cameras===
This section shows the performance results for the elements running simultaneously on multiple cameras on a Jetson Xavier AGX. For all the tests done with an RGB output image, the following pipeline was used to measure the processing time and FPS for the cudaawb and the cudadebayer element with an input image with 1920x1200 resolution coming from multiple camera sensors.
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src device=/dev/video0 io-mode=userptr ! 'video/x-bayer, bpp=10, width=1920, height=1200, format=grbg' ! cudadebayer ! cudaawb ! 'video/x-raw, format=RGB' ! fakesink
</syntaxhighlight>

In the same way, all the tests are done with an I420 output image. the following pipeline was used to measure the processing time and FPS for the cudaawb and the cudadebayer element with an input image with 1920x1200 resolution coming from multiple camera sensors
<syntaxhighlight lang=bash>
GST_DEBUG="GST_TRACER:7" GST_TRACERS="proctime" gst-launch-1.0 -ve v4l2src device=/dev/video1 io-mode=userptr ! 'video/x-bayer, bpp=10, width=1920, height=1200, format=grbg' ! cudadebayer ! cudaawb ! 'video/x-raw, format=I420' ! fakesink
</syntaxhighlight>

The results obtained: 
<center>
{| class="wikitable" 
|-
! colspan=9 | cudadebayer
|-
! Output !! colspan=4|RGB !! colspan=4|I420 
|-
! Number of cameras !! Two !! Three !! Four !! Five !! Two !! Three !! Four !! Five
|-
| FPS || 412 || 429 || 385 || 494 || 464 || 402 || 320 || 332
|-
| Processing time (milliseconds) || 2.426 || 2.354 || 2.597 || 2.025 || 2.154 || 2.486 || 3.128 || 3.011 
|}
</center>

<center>
{| class="wikitable" 
|-
! colspan=9 | cudaawb
|-
! Output !! colspan=4|RGB !! colspan=4|I420 
|-
! Number of cameras !! Two !! Three !! Four !! Five !! Two !! Three !! Four !! Five
|-
| FPS || 397 || 374 || 689 || 347 || 429 || 450 || 289 || 296
|-
| Processing time (milliseconds) || 2.521 || 2.672 || 1.450 || 2.883 || 2.330 || 2.20 || 3.459 || 3.375 
|}
</center>




<noinclude>
{{CUDA ISP for NVIDIA Jetson/Foot|Performance/Library|Contact Us}}
</noinclude>