View source for GStreamer1.0 and V4L2 UserPtr

{{#seo:
|title=GStreamer1.0 and V4L2 UserPtr
|title_mode=replace
|description={{{description|Some people have encountered the issue of V4L2 UserPtr not working in the newer GStreamer1.0. Learn how to work around this V4L2 user pointer problem now.}}}
}}

<table>
<tr>
<td><div class="clear; float:right">__TOC__</div></td>
<td>
{{GStreamer debug}} 
<td> 
<center>
{{ContactUs Button}}
</center>
</tr>
</table>

== V4L2 User Pointer == 

V4L2 default behavior is to allocate its own buffer pool, capture an image in each buffer and provide that buffer to the user to consume. This, although convenient, is not always the desired use case, such as when the buffer needs to be allocated from a special memory pool.  In such occasions, it is required that V4L2 fills user allocated buffers. This mode is known as User Pointer (USERPTR) streaming mode.

Consider the following scenario (pipeline simplified for reading simplicity):

  gst-launch-1.0 v4l2src ! caps filter caps=video/x-raw,width=1920,height=1080,framerate=30/1 ! gloverlay ! glimagesink

# V4L2 source element will capture images from the camera
# GLOverlay will render an PNG on top of the camera frame (a logo for example)
# The overlayed frame will be rendered to a display by GLImageSink

It is important to study the memory requirements of the above pipeline. GLOverlay requires special GPU buffers. If provided otherwise, GLOverlay will allocate its own GPU buffers and data will be copied from the alien buffers into the GPU ones. 
                                                                              memcpy
                      .---- V4L2 buffer --------------------------------------> X ---> GPU buffer -.
  gst-launch-1.0 v4l2src ! caps filter caps=video/x-raw,width=1920,height=1080,framerate=30/1 ! gloverlay ! glimagesink  

There are many reasons to avoid this memory copy: performance, power consumption, latency, etc... This is where UserPtr comes into play. 

                .-------------------------------- GPU Buffer -----------------------------------------------------.                                                                                              
                V   .------------------------------------------------------------> GPU buffer -----------------.   |
  gst-launch-1.0 v4l2src io-mode=userptr ! caps filter caps=video/x-raw,width=1920,height=1080,framerate=30/1 ! gloverlay ! glimagesink  

As it can be seen, now GLOverlay will allocate a GPU buffer, provide it to V4L2src who will fill it with an image and subsequently push it back to the GLOverlay. The memory copy has been avoided.

== User Pointer not working in newer GStreamer-1.0 ==

Gstreamer exposes V4L2 camera capture through its well known GstV4L2Src element. The user pointer mode can be enabled via the ''io-mode'' property.
<pre>
 io-mode             : I/O mode
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
</pre>

However, this seems to be broken in GStreamer-1.0. Specifically, the following message prints when calling the REQBUFS IOCTL in USERPTR mode:

  0:00:02.005602011  2361    0xe1890 ERROR          v4l2allocator gstv4l2allocator.c:771:gst_v4l2_allocator_start:<v4l2src0:pool:src:allocator> error requesting 2 buffers: Invalid argument

Strangely, testing a simpler, lower level application like [https://github.com/fastr/yavta YAVTA] (Yet Another V4L2 Test Application) seems to work properly.

=== The Problem ===

The problem is that GstV4L2Src attempts to use a user space wrapper over V4L2 called libv4l2 if available (which is the case in most of the file systems). This library is a set of calls wrapping traditional V4L2 ioctls and exposing some additional features. This can be verified by noting that gstv4l2allocator.c calls v4l2_ioctl instead of the good old ioctl. However, by checking [https://github.com/philips/libv4l/blob/cdfd2992de72b93e588e0126491f17c30b9e6f43/libv4l2/libv4l2.c#L992 libv4l2 source code], the following shows:

<pre>
    case VIDIOC_REQBUFS:
      {
	struct v4l2_requestbuffers *req = arg;
	/* IMPROVEME (maybe?) add support for userptr's? */
	if (req->memory != V4L2_MEMORY_MMAP) {
	  errno = EINVAL;
	  result = -1;
	  break;
	}
</pre>

As it can be seen, USRPTR is not supported by this layer. [https://github.com/fastr/yavta YAVTA], on the other hand, does work because it calls the ioctl directly.

=== A Workaround ===

If you explore gst-plugins-good-1.9.1/sys/v4l2/v4l2_calls.h, around line 29 you’ll see something like:

<pre>
#ifdef HAVE_LIBV4L2 
#  include <libv4l2.h> 
#else
</pre>

You may get around this issue by avoiding the GStreamer elements from using libv4l2. To do so you’ll need to configure gst-plugins-good as:

  ./configure --without-libv4l2

This should get you past this issue.

== Additional Considerations ==

Another typical problem when using User Pointer mode, is that V4L2 fails to queue the buffers provided by the user. The debug shows as the following:

<pre>
0:00:04.677438294  6329   0x13ec00 ERROR          v4l2allocator gstv4l2allocator.c:1251:gst_v4l2_allocator_qbuf:<v4l2src0:pool:src:allocator> failed queueing buffer 0: Invalid argument
0:00:04.677526886  6329   0x13ec00 ERROR         v4l2bufferpool gstv4l2bufferpool.c:1127:gst_v4l2_buffer_pool_qbuf:<v4l2src0:pool:src> could not queue a buffer 0
</pre>

This is typically related to the nature of the memory passed to V4L2. Many drivers seem to be unable of handling non-contiguous, virtual, Linux memory to capture images. V4L2 documentation says that drivers may handle virtual memory by copying the data into each individual page, however by experience, I can tell you most of them will simply fail. Long story short, you need to use aligned, physically contiguous buffers with V4L2 userptr. Indeed, [https://github.com/fastr/yavta YAVTA] does a posix_memalign to allocate its userptr buffers, if you change it to a regular malloc it will fail.

==Contact Us==
{{ContactUs}}

[[Category:Jetson]][[Category:GStreamer]]