DM385 DM813x MIPI CSI2 Video Capture

From RidgeRun Developer Connection
Jump to: navigation, search

Introduction

The DM385 can record video using the MIPI CSI-2 interface. MIPI CSI-2 is a serial interface for cameras that allows high performance at a very low energy cost. The Texas Instrument's DM385 has a MIPI CSI-2 interface, along with a Cortex-A8 general purpose processor and two Cortex-M3 used as dedicated co-processors for image capture and processing, which utilize the ISS hardware accelerator module as well as the VPSS and HDVICP2 modules.

The following image shows the hardware architecture involved in the video capture process using the ISS module to control the MIPI/CSI2 interface.

Figure 1. Hardware architecture for DM385 MIPI video capture

The DM385 and DM388 are also considered part of the DM813x SoC family.

ISS Hardware

The imaging subsystem (ISS) deals with the processing of the pixel data coming from an external image sensor, data from memory (image format encoding and decoding can be done to and from memory), or data from SL2 in IVA-HD for hardware encoding. With its subparts, such as interfaces and interconnects, image signal processor (ISP), and still image coprocessor (SIMCOP), the ISS is a key component for the following multimedia applications: camera viewfinder, video record, and still image capture. The ISS targets the following major use cases:

  • Viewfinder with digital zoom, video stabilization, and rotation
  • Up to 1080p video record at 30 fps
  • Up to 16 MPixel still image capture
  • High performance mode: Up to 200 MPixel/s throughput
  • High quality and low light modes: Up to 50 MPixel/s throughput
  • Still image capture during video record
Figure 2. DM385 Functional Diagram

V4L2 capture kernel module

ti81xxvin is the Linux V4L2 capture driver. Several structures and functions are added to this driver in order to enable correct communication with the ISS driver through the FVID2 interface.

To manage the MIPI CSI2 capture port, there is a new video device /dev/video7. With the driver installed, the following video port mapping will be active:

/dev/video0 → Capture VIP 0 Port A
/dev/video1 → Display 0
/dev/video2 → Display 1
/dev/video3 → Display 2
/dev/video4 → Capture VIP 0 Port B
/dev/video5 → Capture VIP 1 Port A
/dev/video6 → Capture VIP 1 Port B
/dev/video7 → Capture MIPI/CSI2 Port

VPPS kernel module

The VPSS module makes use of FVID2 interface to send information to Cortex-M3 firmware. Support for new structures and commands are required in order to enable MIPI video capture.

The FVID2 library is a HAL/functional layer running on the Cortex-A8 controlling the firmware running over the M3. It exposes a number of APIs controlling the video compositors, VENCs, and graphics/video pipelines to the user interface drivers like V4L2 and FBDEV. FVID2 library translates the V4L2/Fbdev data structures and ioctl to FVID2 data structures and commands, and calls the Linux Syslink IPC notify function to pass FVID2 data structures and commands to the firmware running on the M3 processor.

MIPI Capture using RidgeRun DM385 SDK

The EZ-SDK saLoopBack example is modified to demonstrate MIPI capture.

ldconfig issue

If you are running 64 bit Ubuntu 14.04 or newer, be sure you have the ldconfig fix in place.

saLoopBack application

This application demonstrates simple loop back from capture to display. The application takes input through the specified port (in this case, the application is modified to take input from CSI2 port). Capture buffers are displayed using V4L2 display driver. The sample application design incorporates a user pointer buffer mechanism for both capture and display. User pointer buffers are taken from the FBDEV driver. V4L2 capture driver outputs YUV422 (y and cbcr interleaved) data to memory and the display driver accepts the same format from memory. The capture driver detects the incoming resolution, then configures the capture and display driver for the same resolution. For example, with 1080P60 input resolution, saLoopBack sets the buffer size to 1920*1080*2 for both display and capture. The application changes the display resolution to 1080P60 using the sysfs entry. Review the saLoopBack code for details.

  • saLoopBack build and modifications to support MIPI capture:

Modify the file 'ezsdk5.class' found on <devdir>/bsp/classes/ezsdk5.class in the line 32 as follows

        - EZSDK_TARGETS+=omtb_v5 rpe_app_v5
        + EZSDK_TARGETS+=omtb_v5 rpe_app_v5 psp_examples_v5

Copy the examples folder that can be found inside the ti-ezsdk:

        ti-ezsdk_dm814x-evm_5_05_02_00/example-applications/linux-driver-examples-psp04.04.00.01

into your devdir at the following position:

        <devdir>/proprietary/ezsdk-5_05_02_00/ezsdk/example-applications/

Then you can build the examples by calling make in the folder:

        <devdir>/proprietary/ezsdk-5_05_02_00/

EXAMPLE:

        make psp_examples_clean
        make psp_examples_v5
        make psp_examples_install

After that the saLoopBack and other examples should be build in the following folder inside the platform filesystem:

        /usr/share/ti/ti-psp-examples/

Finally apply this patch to the original saLoopBack source code in order to enable CSI2 capture port support:

Index: ezsdk-5_05_02_00/ezsdk/example-applications/linux-driver-examples-psp04.04.00.01/video/saLoopBack.c
===================================================================
--- ezsdk-5_05_02_00.orig/ezsdk/example-applications/linux-driver-examples-psp04.04.00.01/video/saLoopBack.c    2014-11-21 14:03:16.706952184 -0600
+++ ezsdk-5_05_02_00/ezsdk/example-applications/linux-driver-examples-psp04.04.00.01/video/saLoopBack.c 2014-11-21 14:18:51.370997419 -0600
@@ -54,6 +54,7 @@
 #include <sys/mman.h>
 #include <errno.h>
 #include <string.h>
+#include <time.h>
 #include <linux/videodev2.h>
 #include <linux/fb.h>
 #include <linux/ti81xxfb.h>
@@ -67,7 +68,7 @@


 /* device node to be used for capture */
-#define CAPTURE_DEVICE         "/dev/video0"
+#define CAPTURE_DEVICE         "/dev/video7"
 #define CAPTURE_NAME           "Capture"
 /* device node to be used for display */
 #define DISPLAY_DEVICE         "/dev/video1"
@@ -75,7 +76,7 @@
 /* number of frames to be captured and displayed
  * Increase this for long runs
  */
-#define MAXLOOPCOUNT           500
+#define MAXLOOPCOUNT           25000

 /* Pixel format for capture and display. Capture supports
  * V4L2_PIX_FMT_NV12, V4L2_PIX_FMT_NV16, V4L2_PIX_FMT_RGB24 but display
@@ -87,7 +88,7 @@
 #define APP_NAME               "saLoopBack"

 /* Define below #def for debug information enable */
-#undef SALOOPBACK_DEBUG
+#define SALOOPBACK_DEBUG


 /* Structure for storing buffer information */
@@ -157,13 +158,13 @@
                        printf("Driver is capabled of scaling and cropping\n");

        }
-       system("echo 0 > /sys/devices/platform/vpss/display0/enabled");
+       system("echo 0 > /sys/devices/platform/vpss/graphics0/enabled");
        /* Query the preset. Set it to invalid and query for it */
        capt.dv_preset.preset = 0x0;
        if (ioctl(capt.fd, VIDIOC_QUERY_DV_PRESET, &capt.dv_preset)) {
                printf("Querying DV Preset failed\n");
                exit(2);
-       }
+       }
        switch (capt.dv_preset.preset) {
        case V4L2_DV_720P60:
                printf("%s:\n Mode set is 720P60\n", APP_NAME);
@@ -188,7 +189,8 @@
                printf("Setting DV Preset failed\n");
                exit(2);
        }
-       system("echo 1 > /sys/devices/platform/vpss/display0/enabled");
+       //system("echo 1 > /sys/devices/platform/vpss/display0/enabled"); tmp
+       printf("### -- saLoopBack: Capture init done\n");
        return 0;
 }
 /* Open display driver and set format according to resolution on capture */
@@ -225,9 +227,10 @@

        /* Get current format */
        capt.fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+       printf("### -- saLoopBack: getting format\n");
        ret = ioctl(capt.fd, VIDIOC_G_FMT, &capt.fmt);
        if (ret < 0) {
-               printf("Set Format failed\n");
+               printf("Get Format failed\n");
                return -1;
        }
        /* Set format according to mode detected */
@@ -244,6 +247,7 @@
        capt.fmt.fmt.pix.bytesperline = capt.fmt.fmt.pix.width * 2;
        capt.fmt.fmt.pix.sizeimage = capt.fmt.fmt.pix.bytesperline *
                capt.fmt.fmt.pix.height;
+
        ret = ioctl(capt.fd, VIDIOC_S_FMT, &capt.fmt);
        if (ret < 0) {
                printf("Set Format failed\n");
@@ -253,10 +257,9 @@
        capt.fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
        ret = ioctl(capt.fd, VIDIOC_G_FMT, &capt.fmt);
        if (ret < 0) {
-               printf("Set Format failed\n");
+               printf("Get Format failed\n");
                return -1;
        }
-       printFormat("Capture", &capt.fmt);

        /* Request buffers. We are operating in userPtr mode */
        capt.reqbuf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
@@ -526,6 +529,7 @@
         * can write on to them. Driver allocates contiguous memory for
         * three buffers. These buffers can be displayed one by one. */
        buffersize = fixinfo.line_length * varinfo.yres;
+       printf("## saLoopBack: buffer_size for mmap = %d", buffersize);
        buffer_addr[0] = (unsigned char *)mmap(0,
                        buffersize *  MAX_BUFFER,
                        (PROT_READ | PROT_WRITE),
@@ -602,10 +606,21 @@
 {
        int i = 0, ret = 0;
        struct v4l2_buffer temp_buf;
+       struct v4l2_buffer file_buf;
        struct timeval before, after, result;

+       const char *name[2];
+       name[0] = "capture_test0.yuv";
+       name[1] = "capture_test1.yuv";
+       name[2] = "capture_test2.yuv";
+       name[3] = "capture_test3.yuv";
+       name[4] = "capture_test4.yuv";
+
+       //system ("./ov5640_config.sh init");
+       //system ("./ov5640_config.sh write_reg");
+
        /* Divert fbdev to dvo2 so that it does  not do blending with display*/
-       system ("echo 1:dvo2 > /sys/devices/platform/vpss/graphics0/nodes");
+       //system ("echo 1:dvo2 > /sys/devices/platform/vpss/graphics0/nodes"); tmp
        /* Open the capture driver. Query the resolution from the capture driver
         */
        ret = initCapture();
@@ -689,28 +704,32 @@
         * 4 queue dislay buffer pointer to capture
         * 5 queue capture buffer pointer to display
         */
+       printf("### -- saLoopBack: starting capture loop\n");
+
        for (i = 0; i < MAXLOOPCOUNT; i++) {
                /* Dq capture buffer */
+               //printf("### -- saLoopBack: dequeuing capture buffer\n");
                ret = ioctl(capt.fd, VIDIOC_DQBUF, &capt.buf);
                if (ret < 0) {
-                       perror("VIDIOC_DQBUF\n");
+                       printf("VIDIOC_DQBUF ERROR\n");
                        return -1;
                }
                /* Because of IP bugs, capture hardware gets locked up once in
                 * a while. In that case DQbuf will return  V4L2_BUF_FLAG_ERROR
                 * in flags.
                 */
+/*
                if (capt.buf.flags & V4L2_BUF_FLAG_ERROR) {
-                       /* If DQbuf returned error check for the hardware lockup
-                        */
+                       // If DQbuf returned error check for the hardware lockup
+                        //
                        ret = ioctl(capt.fd, TICAPT_CHECK_OVERFLOW, &capt.over_flow);
                        if (ret < 0) {
                                perror("TICAPT_CHECK_OVERFLOW\n");
                                return -1;
                        } else {
-                               /* If hardware locked up, restart display and
-                                * capture driver
-                                */
+                               // If hardware locked up, restart display and
+                               // capture driver
+
                                if (capt.over_flow.porta_overflow) {
                                        printf("Port a overflowed\n\n\n\n\n\n\n");
                                        stopCapture();
@@ -726,17 +745,40 @@
                                }
                        }
                }
+*/
                /* DQ display buffer */
+               //printf("### -- saLoopBack: dequeuing display buffer\n");
                ret = ioctl(disp.fd, VIDIOC_DQBUF, &disp.buf);
                if (ret < 0) {
-                       perror("VIDIOC_DQBUF Display\n");
+                       printf("VIDIOC_DQBUF Display\n");
                        return -1;
                }
                /* Exchange display and capture buffer pointers */
+               //printf("### -- saLoopBack: exchange buffers\n");
                temp_buf.m.userptr = capt.buf.m.userptr;
                capt.buf.m.userptr = disp.buf.m.userptr;
                disp.buf.m.userptr = temp_buf.m.userptr;
+
+#ifdef CAPTURE_IMG
+               if ( ((i % 500) == 0) & (i > 0) ){
+                       printf("## saLoopBack: TEST FRAME TO MEM\n");
+                       /* Write output to binary file */
+                       //int new_file;
+                       unsigned int length = capt.fmt.fmt.pix.sizeimage;
+                       printf("## saLoopBack: buffer_length = %d\n",length);
+                       FILE *fh = fopen(name[(i/500)-1],"wb");
+                       if (fh != NULL){
+                               fwrite(temp_buf.m.userptr,1,length,fh);
+                       } else {
+                               printf("## saLoopBack: error with file handler, name[%d]\n", ((i/500)-1));
+                       }
+                       printf("## saLoopBack: New output file created\n");
+                       /* Close output file */
+                       fclose(fh);
+               }
+#endif
                /* Queue the capture buffer with updated address */
+               //printf("### -- saLoopBack: queuing capture buffer\n");
                ret = ioctl(capt.fd, VIDIOC_QBUF, &capt.buf);
                if (ret < 0) {
                        perror("VIDIOC_QBUF\n");
@@ -746,6 +788,7 @@
                disp.buf.memory = V4L2_MEMORY_USERPTR;
                disp.buf.length = capt.fmt.fmt.pix.sizeimage;
                /* Queue the display buffer back with updated address */
+               //printf("### -- saLoopBack: queuing display buffer\n");
                ret = ioctl(disp.fd, VIDIOC_QBUF, &disp.buf);
                if (ret < 0) {
                        perror("VIDIOC_QBUF Display\n");
@@ -758,7 +801,8 @@
        gettimeofday(&after, NULL);
        /* Calculate FPS */
        calc_result_time(&result, &after, &before);
-       printf("Frame rate = %lu\n", MAXLOOPCOUNT/result.tv_sec);
+       printf("\nsaLoopBack: Elapsed Time = %lu.%lu\n\n", result.tv_sec, result.tv_usec);
+       printf("\nsaLoopBack: FrameRate = %lu\n\n", MAXLOOPCOUNT/result.tv_sec);
        /* Stop capture driver */
        ret = stopCapture();
        if (ret < 0) {
@@ -784,10 +828,12 @@
                return ret;
        }
        close(fbdev_fd);
-       system("echo 0 > /sys/devices/platform/vpss/display0/enabled");
-       system ("echo 1080p-60 > /sys/devices/platform/vpss/display0/mode");
-       system("echo 0 > /sys/devices/platform/vpss/graphics0/enabled");
-       system("echo 1 > /sys/devices/platform/vpss/display0/enabled");
-       system("echo 1:hdmi > /sys/devices/platform/vpss/graphics0/nodes");
+       //system("echo 0 > /sys/devices/platform/vpss/display0/enabled");
+       //system ("echo 1080p-60 > /sys/devices/platform/vpss/display0/mode");
+       //system("echo 0 > /sys/devices/platform/vpss/graphics0/enabled");
+       //system("echo 1 > /sys/devices/platform/vpss/display0/enabled");
+       //system("echo 1:hdmi > /sys/devices/platform/vpss/graphics0/nodes");
+
+finish:
        return 0;
 }