View source for NVIDIA GTC 2024 360 VR Demo/GTC Demo Description

<noinclude>
{{NVIDIA GTC 2024 360 VR Demo/Head|previous=|next=Partnerships}}
</noinclude>
 
{{DISPLAYTITLE:NVIDIA GTC 2024 360 VR demo description|noerror}}

Understanding this demo for GTC 2024 requires familiarity with the key components enabling 360-degree video control via a web-based graphical user interface (GUI) and viewing through a virtual reality (VR) headset.

== Components ==

This demo is composed of the next elements shown in the chart. 
<br>
<br>
[[File:GTC STRUCTURE.png.png|800x447px|thumb|center|Structure of the 360 video demo]]
<br>
List of components:
* 2 Fish Eye Lenses with 185 degrees or more for field of view. 
* 2 Camera Modules from Framos
* Carrier Board from Connect Tech
* Jetson AGX Orin Module.
* Docker Container.
* WebPage.
* VR Headset.
* RidgeRun Software products
* Python Demo

=== Fish Eye Lenses ===

For the Fish Eye Lenses, the demo uses the 185° HFOV Lens, with an FOV of 185-degree. 
[https://www.framos.com/en/products/fsmgo-with-imx676-sensor-and-185-hfov-lens-27357 More information]

=== Camera Modules ===

Framos developed the camera modules FSM:GO with IMX676 Sensor, providing a resolution of 3552 x 3556 at up to 64 FPS. 

[https://www.framos.com/en/products/fsmgo-with-imx676-sensor-27363 More information]

=== Carrier Board ===

Connect Tech developed the Anvil Carrier Board, prepared to endure the demands of the most computationally intensive AI applications by using the cutting-edge NVIDIA® Jetson AGX Orin™ and AGX Orin™ Industrial supercomputers.

[https://connecttech.com/product/anvil-embedded-system-with-nvidia-jetson-agx-orin/ More Information]

=== Jetson AGX Orin Module ===

The  NVIDIA® Jetson Orin™ module gives up to 275 trillion operations per second (TOPS) for multiple AI inference pipelines with 64GB of memory, giving more than 8X the performance of Jetson AGX Xavier™.

[https://www.arrow.com/en/products/900-13701-0050-000/nvidia More information]

===  Driver for GMSL Interface ===

GMSL (Gigabit Multimedia Serial Link), operating as an asymmetric full-duplex SerDes technology, adeptly manages the simultaneous transmission of high-rate downlink data and lower-rate uplink data. It facilitates the conveyance of power, bidirectional control data, Ethernet, audio, and multiple unidirectional video streams over a single coaxial or shielded twisted pair cable. Additionally, GMSL supports both unprotected and encrypted video transport through HDCP.

=== Docker Container === 

RidgeRun has crafted this container to encompass the requisite software dependencies, products, and applications necessary for executing the demo. This demo uses GStreamer with Python to create and control the pipelines via the Web Page. You can have more information in the RidgeRun Wiki Guide on [[Docker_Images_with_Demos_and_Products_Evaluation_Versions | Docker Images with Demos and Products Evaluation Versions]]. 

For this demo, the container includes the repository where it runs the demo with the required dependencies as you can see in the next image. 

[[File:Docker dependencies.png|641x591px|thumb|center|Docker dependencies required for the demo.]]

=== Generative AI ===

Generative AI Agent developed by [https://www.jetson-ai-lab.com/tutorial_llava.html NVIDIA]. This agent provides information regarding the actual snapshot taken by the demo.

'''Important''':
* In the AGX Orin used in the demo, an NVME was installed in order to have more space to play with the Jetson AI lab containers.
* The Gen AI agent must run inside the container provided by NVIDIA, the instructions below show how to run everything
* The demo repository must be placed inside the mount point of the container. In the AGX Orin for the demo is mounted in ''/orin_ssd/jetson_containers/data/''
* Instructions to download jetson containers can be found [https://www.jetson-ai-lab.com/tutorial_llava.html here].
* Github repository with Gen AI reference samples [https://github.com/dusty-nv/jetson-containers here]. The Llava reference code we are using is [https://github.com/dusty-nv/jetson-containers/blob/master/packages/llm/llava/benchmark.py this one].

Instructions to run:
* Make sure your NVME is mounted in '''/orin_ssd'''
* After the board boots up, it is important to change the docker images directory as follows (In the GTC AGX Orin ''/orin_ssh'' directory there is an script to run these commands called '''change_docker_mount.sh'''):
<pre>
sudo mount --rbind /orin_ssd/docker /var/lib/docker
sudo systemctl stop docker
sudo systemctl start docker
</pre>
* Go to the NVME mount point:
<pre>
cd /orin_ssd
</pre>

=== WebPage ===

This web page handles the features to control the demo and a thumbnail to see a snapshot of the video sent to the VR Headset. The web page can be executed in any browser. 

=== VR Headset ===

The MetaQuest 2 provides 360° videos with visuals at a resolution of 20 pixels per degree and are showcased on a Fast-Switch LCD Display, boasting a total span of 1832 x 1920 pixels per eye. 

[https://www.meta.com/quest/products/quest-2/ More information]

<noinclude>
{{NVIDIA GTC 2024 360 VR Demo/Foot||Partnerships}}
</noinclude>