NVIDIA GTC 2024 360 VR demo description
| 360 VR with Generative AI |
|---|
|
| GTC Demo Description |
| Partnerships |
| Using the Demo |
| RidgeRun Products in the Demo |
| Potential applications and use cases |
| Low Latency Streaming |
| Contact Us |
Understanding this demo for GTC 2024 requires familiarity with the key components enabling 360-degree video control via a web-based graphical user interface (GUI) and viewing through a virtual reality (VR) headset.
Components
This demo is composed of the next elements shown in the chart.
List of components:
- 2 Fish Eye Lenses with 185 degrees or more for field of view.
- 2 Camera Modules from Framos
- Carrier Board from Connect Tech
- Jetson AGX Orin Module.
- Docker Container.
- WebPage.
- VR Headset.
- RidgeRun Software products
- Python Demo
Fish Eye Lenses
For the Fish Eye Lenses, the demo uses the 185° HFOV Lens, with an FOV of 185-degree. More information
Camera Modules
Framos developed the camera modules FSM:GO with IMX676 Sensor, providing a resolution of 3552 x 3556 at up to 64 FPS.
Carrier Board
Connect Tech developed the Anvil Carrier Board, prepared to endure the demands of the most computationally intensive AI applications by using the cutting-edge NVIDIA® Jetson AGX Orin™ and AGX Orin™ Industrial supercomputers.
Jetson AGX Orin Module
The NVIDIA® Jetson Orin™ module gives up to 275 trillion operations per second (TOPS) for multiple AI inference pipelines with 64GB of memory, giving more than 8X the performance of Jetson AGX Xavier™.
Driver for GMSL Interface
GMSL (Gigabit Multimedia Serial Link), operating as an asymmetric full-duplex SerDes technology, adeptly manages the simultaneous transmission of high-rate downlink data and lower-rate uplink data. It facilitates the conveyance of power, bidirectional control data, Ethernet, audio, and multiple unidirectional video streams over a single coaxial or shielded twisted pair cable. Additionally, GMSL supports both unprotected and encrypted video transport through HDCP.
Docker Container
RidgeRun has crafted this container to encompass the requisite software dependencies, products, and applications necessary for executing the demo. This demo uses GStreamer with Python to create and control the pipelines via the Web Page. You can have more information in the RidgeRun Wiki Guide on Docker Images with Demos and Products Evaluation Versions.
For this demo, the container includes the repository where it runs the demo with the required dependencies as you can see in the next image.
Generative AI
Generative AI Agent developed by NVIDIA. This agent provides information regarding the actual snapshot taken by the demo.
Important:
- In the AGX Orin used in the demo, an NVME was installed in order to have more space to play with the Jetson AI lab containers.
- The Gen AI agent must run inside the container provided by NVIDIA, the instructions below show how to run everything
- The demo repository must be placed inside the mount point of the container. In the AGX Orin for the demo is mounted in /orin_ssd/jetson_containers/data/
- Instructions to download jetson containers can be found here.
- Github repository with Gen AI reference samples here. The Llava reference code we are using is this one.
Instructions to run:
- Make sure your NVME is mounted in /orin_ssd
- After the board boots up, it is important to change the docker images directory as follows (In the GTC AGX Orin /orin_ssh directory there is an script to run these commands called change_docker_mount.sh):
sudo mount --rbind /orin_ssd/docker /var/lib/docker sudo systemctl stop docker sudo systemctl start docker
- Go to the NVME mount point:
cd /orin_ssd
WebPage
This web page handles the features to control the demo and a thumbnail to see a snapshot of the video sent to the VR Headset. The web page can be executed in any browser.
VR Headset
The MetaQuest 2 provides 360° videos with visuals at a resolution of 20 pixels per degree and are showcased on a Fast-Switch LCD Display, boasting a total span of 1832 x 1920 pixels per eye.