How to capture at 1080p with LeopardBoard DM36x

From RidgeRun Developer Connection

Jump to:navigation, search

Contents

Introduction

This whitepapers explains briefly how to configure your LeopardBoard DM368 to capture at 1080p.

Hardware needed

SDK Configuration

1. Select the MT9P031 driver to support your camera module

      Prompt: mt9p031 support
      Location:                                                                                                 
       -> Kernel configuration                                                                                    
         -> Device Drivers                                                                                        
           -> Multimedia support (MEDIA_SUPPORT [=y])                                                             
             -> Video capture adapters (VIDEO_CAPTURE_DRIVERS [=y])

2. Configure your SDK in order to be able to capture at 1080p as is shown below:

     Architecture configurations
           Video Output (Component)  --->                                                   
           Component Standard (1080I-30)  --->                                          
           Maximum Video Output Buffer Size (720P)  --->                            
           Maximum Video Input Buffer Size (1080P)  --->

3. Increase the reserved memory and the CMEM space to allocate the 1080p buffers.

    Proprietary Software
        [*] Reserve memory from the kernel for codec engine and friends automatically          
        (0x2bf7000) Amount of reserved memory from the kernel                     
        [*] Automatically setup CMEM at the beggining of the reserved memory area     
        (0x17f7000) Amount of reserved memory for cmemk 

4. If your target format is H264 and you are using an EVAL version of the SDK you will need to do the following modification into you MT9P031 driver:

Index: kernel/linux-2.6.32.17-psp03.01.01.39/drivers/media/video/mt9p031.c
===================================================================
--- kernel.orig/linux-2.6.32.17-psp03.01.01.39/drivers/media/video/mt9p031.c	2012-08-10 09:44:39.763840232 -0700
+++ kernel/linux-2.6.32.17-psp03.01.01.39/drivers/media/video/mt9p031.c	2012-08-10 09:45:20.195126497 -0700
@@ -204,7 +204,7 @@
     },
     {
     /* 1080P-HDTV */
-    .framesize = { 1920, 1080},
+    .framesize = { 1920, 1088},
     .frameintervals = {
         {  .numerator = 1, .denominator = 31 },
         },

This modification will allow use 1080p with the H264 encoder.

5. Re-built your SDK and install it into your board.

Use GStreamer to capture from the camera at 1080p

MJPEG Encoding

The following pipeline will capture a 1080p image and will encode it in JPEG to be saved to a file.

 IMAGE_FILE=image.jpeg

 gst-launch -e v4l2src num-buffers=1 always-copy=false ! 'video/x-raw-yuv,format=(fourcc)NV12, width=1920,height=1080' ! queue ! dmaiaccel ! dmaienc_mjpeg ! filesink location=$IMAGE_FILE

H264 Encoding

The following pipeline captures 1080p video and encode it in H264 to be saved into a file.

VIDEO_FILE=h264file.mp4

gst-launch -e v4l2src always-copy=false ! 'video/x-raw-yuv,format=(fourcc)NV12, width=1920,height=1088' ! queue ! dmaiaccel ! dmaienc_h264 encodingpreset=2 ratecontrol=2 ! qtmux ! filesink location=$VIDEO_FILE

NOTE: Due to codec limitations it is necessary to use 1920x1088 image size instead of 1920x1080. A specific modification must me done to accomplish this feature in the EVAL version (See previous section).

Live preview while saving to a file

Use case is a device that is capturing video with a camera, having an HD Monitor (HD 1080p TV) showing a live preview of the video being captured, along with the video being saved to file. A simple gst-launch type command is shown below. For a real product, you might want to consider using GStreamer daemon so that you can start and stop the recording.

Hardware is LeopardBoard 368 + L1-5M03 (MT9P031) Camera sensor board using SD card on leo368 and using leo368 component output.

 
H264ENC_PARMS="encodingpreset=2 ratecontrol=2 intraframeinterval=23 idrinterval=46 targetbitrate=3000000 profile=100 level=50 entropy=1 t8x8inter=true t8x8intra=true single-nalu=true"

gst-launch -e v4l2src input-src=camera chain-ipipe=true always-copy=false num-buffers=3000 ! 'video/x-raw-yuv,format=(fourcc)NV12,width=1280,height=720' ! dmaiaccel ! \ 
               tee name=t1 ! TIDmaiVideoSink enable-last-buffer=false videoStd=720P_60 videoOutput=COMPONENT \
            t1. ! queue ! dmaienc_h264 $H264ENC_PARMS ! queue ! dmaiperf print-arm-load=true ! qtmux ! queue ! \
                         filesink location=test.mp4 sync=false enable-last-buffer=false

Performance notes

For this example, testing was done with a LeopardBoard 368 and Aptina MT9P031 sensor (LI-5M03). We followed the GStreamer pipeline tuning steps, changing one part of the pipeline at a time so you can see the results of each change. We also ran into some problems so we included some of the GStreamer Debugging steps we followed.

The first simple pipeline that was tried ( v4l2src -> fakesink ):

CAPS='video/x-raw-yuv,format=(fourcc)NV12, width=1920,height=1080'
gst-launch -e v4l2src num-buffers=100 always-copy=false ! capsfilter caps="$CAPS" ! dmaiperf print-arm-load=true ! fakesink

The framerate is around 5.5 fps and the ARM CPU load is 100%. When you see the CPU load spike, it is due to an unwanted frame buffer copy. We avoid that using dmaiaccel element:

gst-launch -e v4l2src  num-buffers=100 always-copy=false ! capsfilter caps="$CAPS" ! dmaiaccel ! dmaiperf print-arm-load=true ! fakesink

The framerate is around 15.5 fps and the ARM CPU load is around 8%. We fixed the CPU load, but the framerate is too low. The problem is a buffer is not available when V4L2 needs it. So we add some more buffers (requires change in memory map and kernel command line parameters):

gst-launch -e v4l2src queue-size=5 num-buffers=100 always-copy=false ! capsfilter caps="$CAPS" ! dmaiaccel  ! dmaiperf print-arm-load=true ! fakesink

The framerate is 30 fps and the ARM CPU load is around 8%. Good, now we have video frames flowing properly so we can add more elements to the pipeline. Let's start with H.264 encoding:

gst-launch -e v4l2src queue-size=5 num-buffers=100 always-copy=false ! capsfilter caps="$CAPS" ! dmaiaccel ! dmaienc_h264 ! qtmux ! dmaiperf print-arm-load=true ! fakesink

Pipeline got stuck. I wonder why. Let's enable some debug:

gst-launch -e -v v4l2src queue-size=5 num-buffers=100 always-copy=false ! capsfilter caps="$CAPS" ! dmaiaccel ! dmaienc_h264 ! qtmux ! dmaiperf print-arm-load=true ! fakesink

With some of the output being:

 ../../src/src/gsttividenc1.c(343): gstti_videnc1_process (): /GstPipeline:pipeline0/dmaienc_h264:dmaienc_h2640: failed to encode video buffer

meaning the H.264 hardware encoder wasn't happy. Let's set the DMAI element specific debug output:

gst-launch --gst-debug-help # list what debug is available
gst-launch --gst-debug=tidmai*:5 v4l2src queue-size=5 num-buffers=100 always-copy=false ! capsfilter caps="$CAPS" ! dmaiaccel ! dmaienc_h264 ! qtmux ! dmaiperf print-arm-load=true ! fakesink

Now lets add some H.264 encoder parameters:

H264_PARMS="encodingpreset=2 ratecontrol=2 single-nalu=true targetbitrate=12000000 maxbitrate=12000000"


How about an MJPEG pipeline:

gst-launch -e v4l2src queue-size=5 num-buffers=100 always-copy=false ! capsfilter caps="$CAPS" ! dmaiaccel ! dmaienc_mjpeg ! dmaiperf print-arm-load=true ! fakesink

Notice the ARM load is low and the frame rate dropped. That means we can let the ARM work on more than one part of the pipeline at the same time. This is done using the GStreamer queue element:

gst-launch -e v4l2src queue-size=5 num-buffers=100 always-copy=false ! capsfilter caps="$CAPS" ! queue ! dmaiaccel ! dmaienc_mjpeg ! dmaiperf print-arm-load=true ! fakesink
Frame rate
(fps)
ARM CPU load Simplified pipeline
(real pipelines shown above)
5.5 100% v4l2src -> fakesink
15.5 8% v4l2src -> dmaiaccel -> fakesink
30 8% v4l2src queue-size=5 -> dmaiaccel -> fakesink
21.5 8% v4l2src queue-size=5 -> dmaiaccel -> dmaienc_mjpeg ! fakesink
21.5 8% v4l2src queue-size=5 -> queue -> dmaiaccel -> dmaienc_mjpeg ! fakesink
Navigation
Toolbox