DS3D Framework

Deepstream 7.0 has implemented a 3D framework, interfaces and custom-libs for DeepStream-3D processing. This DS3D framework adds support for LIDAR and RADAR with BEVFusion and V2X models. These interfaces are capable of different types of data fusion like: dataloader, datafilter and datarender.

DS3D framework recognizes ds3d/datamap as the data format for data buffers in GStreamer plugins. The type ds3d/datamap are key-value pairs where key is a string and value, a pointer, structure or tensor frame to data. DS3D Framework source code can be find in the path /opt/nvidia/deepstream/deepstream/sources/libs/ds3d.

The 3D Lidar Sensor Fusion example use a series of DS3D components configured in YAML format to perform the following pipeline.

Follow the DeepStream 7.0 Lidar example guide to check out and test the Lidar example.

Each component is configured in YAML format and implemented on a custom library of types: ds3d::dataloader, ds3d::databridge. s3d::datamixer and ds3d::datafilter.

Load LIDAR data

To load LIDAR data is required to use the data type type: ds3d::dataloader which is implemented in the library libnvds_lidarfileread.so. or libnvds_3d_dataloader_realsense.so. The library can be found in the path: /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/dataloader/lidarsource.

LIDAR info can be load from a yaml file and output as a ds3d/datamap data type using file loader custom library or read lidar data frame by frame and create a ds3d::datamap per frame.

You can check more details in the Custom Dataloader libnvds_lidarfileread Configuration Specifications

LIDAR preprocess

LiDAR data preprocess generates features, coordinates, and point numbers (feats/coords/N) tensors. Which can be achieved with the library libnvds_3d_lidar_preprocess_datafilter.so which process data for v2x sensor fusion model.

This library can be found in the path: /opt/nvidia/deepstream/deepstream/sources/libs/ds3d/datafilter/lidar_preprocess/ and you can check specifications on Custom Datafilter libnvds_3d_lidar_preprocess_datafilter Specifications

This ca also be perform with the gstreamer element nvds3dfilter with the corresponding configuration file.

LIDAR Bridge

This bridge is designed to loads custom libraries for data conversion to and from DS3D. For example to convert 2D video (video/x-raw(memory:NVMM) into ds3d::datamap format use the library libnvds_3d_video_databridge.so of type ds3d::databridge.

This ca also be perform with the gstreamer element nvds3dbridge with the corresponding configuration file.

Data and LIDAR Mixer

The ds3d::datamixer combines video data 2D and LIDAR into single data of type ds3d::datamap.

This is implemented in the library libnvds_3d_multisensor_mixer.so and also can be performed with the gstreamer element nvds3dmixer with the corresponding configuration file.

Data Alignment

The library libnvds_3d_alignment_datafilter.so of type ds3d::datafilter applies a series of transformations to align the LiDAR data with the camera image coordinate system.

This ca also be perform with the gstreamer element nvds3dfilter with the corresponding configuration file.

Data Inference

This library libnvds_tritoninferfilter.so of type ds3d::datafilter executes data inference using Triton Inference Server. It supports both Triton CAPI and gRPC modes. Custom pre-processing and post-processing might be required depending on the specific inference task.

You can check more details in the libnvds_tritoninferfilter Configuration Specifications

Data Render

You can render a 3D scene using OpenGL ES (GLES) with various elements such as textures, LiDAR points, bounding boxes within a single window. This is implemented in the library libnvds_tritoninferfilter.so of type ds3d::datafilter.

You can check more details in the Custom datarender libnvds_3d_gles_ensemble_render Configuration Specifications

RidgeRun Resources