Birds Eye View - Jetson GStreamer Pipelines
| ⇦ GStreamer/Generic Pipelines | Home | GStreamer/i.MX8 Pipelines ⇨ |
 |
| Getting Started |
|---|
| How BEV Works |
| User Guide |
| Calibration Guide |
| Calibration Guide Legacy |
| GStreamer |
| Performance |
| Contact Us |
RidgeRun's Jetson Bird's Eye View GStreamer pipelines show how to build and run multi-camera surround-view pipelines on NVIDIA Jetson using Jetson-accelerated elements such as nvjpegdec, nvv4l2decoder, nvvidconv, nvv4l2h264enc, nvarguscamerasrc, and nv3dsink. Use this page when you want Jetson-specific reference pipelines for JPEG images, UDP streams, MP4 files, CSI cameras, or CUDA-based undistortion.
Use Birds Eye View/GStreamer/Generic Pipelines for platform-agnostic Linux examples and Birds Eye View/GStreamer/i.MX8 Pipelines for NXP-specific pipelines.
These examples assume that your calibration file matches the number of cameras, camera order, and input resolution used in the pipeline. If the calibration file and the incoming streams do not match, the stitched output may fail or look incorrect. For calibration workflow details, see Birds Eye View/Calibration Guide. For plugin-level property details, see Birds Eye View/GStreamer/GstBEV Plugin.
| Use case | Input source | Recommended output | Main Jetson elements |
|---|---|---|---|
| Quick validation | JPEG images | JPEG file | nvjpegdec, nvvidconv |
| Network ingest | UDP H.264 RTP streams | MP4 file or UDP RTP stream | nvv4l2decoder, nvvidconv, nvv4l2h264enc |
| Offline playback | MP4 files | Display | nvv4l2decoder, nvvidconv |
| Live capture | CSI cameras on Jetson | Fakesink, display, or encoded stream | nvarguscamerasrc, nvvidconv |
| Distortion correction before BEV | Test streams or live streams | Display and/or MP4 file | rrcudaundistort, nvvidconv, nvv4l2h264enc |
Before You Start
Before running the pipelines on Jetson, confirm the following:
- You have the GstBev plugin installed.
- You have a valid Bird's Eye View calibration file.
- The camera order in the pipeline matches the order used during calibration.
- The input resolution matches the calibration file.
- The required Jetson GStreamer plugins are installed and available.
Set up Environment Variables
Locate the calibration file or create it with calibration tool. Adjust the number of sources, source order, and resolution to match your system. For the examples in this page, we use the Birds Eye View example calibration file, included with the sources and evaluation package. This calibration file expects an input of 6 cameras and a resolution of 1280x720.
Set the CALIBRATION_FILE environment variable to the path to the calibration file:
CALIBRATION_FILE=birds_eye_view.json
Set the CAPS environment variable to the caps expected by the bev element:
CAPS="video/x-raw,format=RGBA"
Note that we deliberately avoid setting the image resolution in the caps, since the image codecs should automatically parse the correct values.
Generate Birds Eye View from images
In this example, the pipeline will read the images presented in Fig. 1 and generate one single birds eye view image.
- Fig. 1: Example input images for generating birds eye view
-
Camera 0 -
Camera 1 -
Camera 2 -
Camera 3 -
Camera 4 -
Camera 5
Note that there is one filesrc branch per camera image in the pipeline, all images are sent to the bev element and the output is stored as a jpeg image. The images are decoded using nvjpegedec.
gst-launch-1.0 bev name=bev0 calibration-file=$CALIBRATION_FILE \ filesrc location=samples/bev_6_cameras/cam_0.jpg ! jpegparse ! nvjpegdec ! nvvidconv ! video/x-raw ! queue ! bev0.sink_0 \ filesrc location=samples/bev_6_cameras/cam_1.jpg ! jpegparse ! nvjpegdec ! nvvidconv ! video/x-raw ! queue ! bev0.sink_1 \ filesrc location=samples/bev_6_cameras/cam_2.jpg ! jpegparse ! nvjpegdec ! nvvidconv ! video/x-raw ! queue ! bev0.sink_2 \ filesrc location=samples/bev_6_cameras/cam_5.jpg ! jpegparse ! nvjpegdec ! nvvidconv ! video/x-raw ! queue ! bev0.sink_3 \ filesrc location=samples/bev_6_cameras/cam_4.jpg ! jpegparse ! nvjpegdec ! nvvidconv ! video/x-raw ! queue ! bev0.sink_4 \ filesrc location=samples/bev_6_cameras/cam_3.jpg ! jpegparse ! nvjpegdec ! nvvidconv ! video/x-raw ! queue ! bev0.sink_5 \ bev0. ! queue ! videoconvert ! jpegenc ! filesink location=$OUTPUT
The expected output from the previous pipeline is presented in Fig. 2.
Birds Eye view from UDP SRC
This section provides pipelines for reading the input video stream from UDP and saving the resulting birds eye view video as an MP4 recording or streaming it through UDP. The examples in this section assume that the input video streams are available in the UDP ports from 4000 to 4005.
Save the result to a MP4 file
The pipeline below receives six UDP streams, decodes them, uses the bev element to generate a single top-down view, encodes the result and saves it as a MP4 file. The pipeline uses nvv4l2decoder to decode the input streams and nvv4l2h264enc to encode the resulting video.
gst-launch-1.0 -ve \ udpsrc port=4000 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_0 \ udpsrc port=4001 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_1 \ udpsrc port=4002 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_2 \ udpsrc port=4003 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_3 \ udpsrc port=4004 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_4 \ udpsrc port=4005 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_5 \ bev name=bev calibration-file=$CALIBRATION_FILE ! queue ! nvvidconv ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mp4mux ! filesink location=bev_result.mp4
Stream the result via UDP+RTP
Similarly to the previous example, the following pipeline receives six UDP video streams and generates the birds eye view. The resulting video stream is then encoded and streamed through UDP. You must set the HOST to the IP address and PORT to the UDP port of the birds eye view video receiver:
Sender
gst-launch-1.0 -v \ udpsrc port=4000 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_0 \ udpsrc port=4001 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_1 \ udpsrc port=4002 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_2 \ udpsrc port=4003 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_3 \ udpsrc port=4004 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_4 \ udpsrc port=4005 ! application/x-rtp ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! queue ! bev.sink_5 \ bev name=bev calibration-file=$CALIBRATION_FILE ! queue ! nvvidconv ! nvv4l2h264enc bitrate=20000000 ! rtph264pay config-interval=10 ! queue ! udpsink host=$HOST port=$PORT
You can use the following pipeline to receive and display the birds eye view stream:
Receiver
gst-launch-1.0 udpsrc port=$PORT ! 'application/x-rtp,media=video,encoding-name=H264' ! queue ! rtph264depay ! avdec_h264 ! videoconvert ! xvimagesink
Generate Birds Eye view from mp4 videos
The following pipeline reads the input from six mp4 videos, decodes them and applies the birds eye view. The result is sent to display. Make sure to set the INPUT_N variables to your corresponding input videos paths.
gst-launch-1.0 bev name=bev0 calibration-file=$CALIBRATION_FILE \ filesrc location=$INPUT_0 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! bev0.sink_0 \ filesrc location=$INPUT_1 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! bev0.sink_1 \ filesrc location=$INPUT_2 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! bev0.sink_2 \ filesrc location=$INPUT_3 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! bev0.sink_3 \ filesrc location=$INPUT_4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! bev0.sink_4 \ filesrc location=$INPUT_5 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvidconv ! $CAPS ! bev0.sink_5 \ bev0. ! queue ! nvvidconv ! xvimagesink
Generate Birds Eye View from live cameras
The following pipeline captures from six cameras, applies the birds eye view and sends the result to display. Make sure to set the video device ID and the caps property according to your own setup.
gst-launch-1.0 bev name=bev0 calibration-file=$CALIBRATION_FILE \ nvarguscamerasrc sensor-id=0 ! nvvidconv ! $CAPS ! bev0.sink_0 \ nvarguscamerasrc sensor-id=1 ! nvvidconv ! $CAPS ! bev0.sink_1 \ nvarguscamerasrc sensor-id=2 ! nvvidconv ! $CAPS ! bev0.sink_2 \ nvarguscamerasrc sensor-id=3 ! nvvidconv ! $CAPS ! bev0.sink_3 \ nvarguscamerasrc sensor-id=4 ! nvvidconv ! $CAPS ! bev0.sink_4 \ nvarguscamerasrc sensor-id=5 ! nvvidconv ! $CAPS ! bev0.sink_5 \ bev0. ! queue ! nvvidconv ! nv3dsink -v
Generate Birds Eye View from live cameras with lens undistortion
Often, fisheye lens are used in surround-systems camera setups to achieve larger field of views. The images captured from cameras with fisheye lens must be undistorted before the birds eye view processing. Plugins like RidgeRun's CUDA Undistort provide a fast solution to rectify the fisheye images undistortion. The following pipeline is an example of how CUDA Undistort can work alongside Birds Eye View to generate elevated surround views from cameras with fisheye lens. The pipeline captures from six live sources (we use videotestsrc for this example but this can be replaced with nvarguscamerasrc), apply the undistortion, and then process the images with birds eye view and generate an MP4 recording and render the result to display simultaneously.
gst-launch-1.0 bev name=bev0 calibration-file=$CALIBRATION_FILE \
videotestsrc is-live=true ! video/x-raw,height=720,width=1280 ! capsfilter caps=$CAPS ! rrcudaundistort distortion-model=fisheye camera-matrix="\"{\"fx\":$FX,\"fy\":$FY,\"cx\":$CX,\"cy\":$CY}\"" distortion-parameters="\"{\"k1\":$K1,\"k2\":$K2,\"k3\":$K3,\"k4\":$K4}\"" valid-pixels=0 ! nvvidconv ! $CAPS ! queue ! bev0.sink_0 \
videotestsrc is-live=true ! video/x-raw,height=720,width=1280 ! capsfilter caps=$CAPS ! rrcudaundistort distortion-model=fisheye camera-matrix="\"{\"fx\":$FX,\"fy\":$FY,\"cx\":$CX,\"cy\":$CY}\"" distortion-parameters="\"{\"k1\":$K1,\"k2\":$K2,\"k3\":$K3,\"k4\":$K4}\"" valid-pixels=0 ! nvvidconv ! $CAPS ! queue ! bev0.sink_1 \
videotestsrc is-live=true ! video/x-raw,height=720,width=1280 ! capsfilter caps=$CAPS ! rrcudaundistort distortion-model=fisheye camera-matrix="\"{\"fx\":$FX,\"fy\":$FY,\"cx\":$CX,\"cy\":$CY}\"" distortion-parameters="\"{\"k1\":$K1,\"k2\":$K2,\"k3\":$K3,\"k4\":$K4}\"" valid-pixels=0 ! nvvidconv ! $CAPS ! queue ! bev0.sink_2 \
videotestsrc is-live=true ! video/x-raw,height=720,width=1280 ! capsfilter caps=$CAPS ! rrcudaundistort distortion-model=fisheye camera-matrix="\"{\"fx\":$FX,\"fy\":$FY,\"cx\":$CX,\"cy\":$CY}\"" distortion-parameters="\"{\"k1\":$K1,\"k2\":$K2,\"k3\":$K3,\"k4\":$K4}\"" valid-pixels=0 ! nvvidconv ! $CAPS ! queue ! bev0.sink_3 \
videotestsrc is-live=true ! video/x-raw,height=720,width=1280 ! capsfilter caps=$CAPS ! rrcudaundistort distortion-model=fisheye camera-matrix="\"{\"fx\":$FX,\"fy\":$FY,\"cx\":$CX,\"cy\":$CY}\"" distortion-parameters="\"{\"k1\":$K1,\"k2\":$K2,\"k3\":$K3,\"k4\":$K4}\"" valid-pixels=0 ! nvvidconv ! $CAPS ! queue ! bev0.sink_4 \
videotestsrc is-live=true ! video/x-raw,height=720,width=1280 ! capsfilter caps=$CAPS ! rrcudaundistort distortion-model=fisheye camera-matrix="\"{\"fx\":$FX,\"fy\":$FY,\"cx\":$CX,\"cy\":$CY}\"" distortion-parameters="\"{\"k1\":$K1,\"k2\":$K2,\"k3\":$K3,\"k4\":$K4}\"" valid-pixels=0 ! nvvidconv ! $CAPS ! queue ! bev0.sink_5 \
bev0. ! queue ! tee name=t t. ! queue ! nvvidconv ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=output.mp4 t. ! queue ! nvvidconv ! nv3dsink
FAQ
- When should I use the Jetson pipelines instead of the generic pipelines?
- Use the Jetson pipelines when you want to use NVIDIA Jetson-specific GStreamer elements for hardware-accelerated decode, camera capture, color conversion, display, or encode.
- When should I use CUDA Undistort before Bird's Eye View?
- Birds Eye View requires rectilinear images as input. If your camera produces fisheye images, then they must be rectified before the bev element can process them. This page provides an example where the
rrcudaundistortelement runs before the BEV element so that fisheye or distorted camera images are corrected before stitching.
Related Pages
- Birds Eye View
- Birds Eye View/GStreamer/GstBEV Plugin
- Birds Eye View/GStreamer/Generic Pipelines
- Birds Eye View/GStreamer/i.MX8 Pipelines
- Birds Eye View/Calibration Guide
- Birds Eye View/Performance/NVIDIA Jetson
- CUDA Accelerated GStreamer Camera Undistort
Contact Us
If you need help adapting these pipelines to a specific Jetson carrier board, camera topology, or performance target, contact RidgeRun through Birds Eye View/Contact us. For larger integration work, RidgeRun's engineering team can help with camera bring-up, pipeline optimization, calibration workflow, and benchmark-driven tuning.