Realistic 3D simulation is becoming a cornerstone of modern AI and graphics, from training autonomous vehicles (AV) to powering robotics and digital twins. Neural rendering techniques like NeRFs and 3D Gaussian Splatting (3DGS) have revolutionized how 3D scenes are reconstructed and visualized from raw sensor data.
In this post, we introduce the implementation of 3D Gaussian Unscented Transform (3DGUT), a cutting-edge method that supercharges neural rendering in the gsplat library—with support for real-world camera effects. With this integration, the performance and fidelity benefits are more readily available and accessible to developers and researchers to create rich virtual worlds for autonomous machines, robotics, and other physical AI applications.
How 3DGUT advances 3D rendering
3DGUT is revolutionizing 3D rendering and scene reconstruction. Built on the widely adopted 3DGS framework, 3DGUT replaces traditional Elliptical Weighted Average (EWA) splatting with the more flexible Unscented Transform. This advancement enables developers to handle complex camera models—including distortions for fisheye lenses and rolling shutter effects—with unprecedented accuracy and without sacrificing rendering speed.
3DGUT also introduces powerful capabilities for simulating secondary visual phenomena, such as reflections and refractions, by aligning seamlessly with the NVIDIA ray-tracing method, 3DGRT. To accelerate adoption and research, 3DGUT and 3DGRT are open-source and available through the nv-tlabs/3dgrut GitHub repo.
Implementation of 3DGUT in gsplat
The popular open-source Gaussian splatting library gsplat is designed specifically for developing and training Gaussian Splatting methods. It is central to harnessing the full potential of 3DGUT. gsplat offers Python bindings compatible with PyTorch and an optimized CUDA backend with numerous features that enhance the optimization of Gaussian Splatting models.
Actively maintained on GitHub under Apache License 2.0, gsplat powers numerous popular neural rendering repositories, including Nerfstudio. Combining the flexibility of 3DGUT and the modular, highly optimized framework of gsplat provides an ideal environment for scalable, high-performance 3D rendering.
Scalable 3D rendering
The integration of 3DGUT into the popular open-source gsplat library enables rendering (and training from) complex camera models including distortions and rolling shutters. This is critical for production-level pipelines. When paired with the lightweight gsplat CUDA backend with batch and multi-GPU support, users can explore larger environments with fewer resources, all while maintaining real-time feedback.
Streamlined workflow and rapid experimentation
The modular, plug-and-play architecture of gsplat is the ideal complement for the flexibility of 3DGUT. Researchers and developers can quickly prototype new architectures by switching optimization routines, camera models, or even Gaussian representations—without rewriting core infrastructure.
Pairing 3DGUT with gsplat reduces friction in every step of the workflow—from dataset preparation to visual evaluation—speeding up iteration cycles and boosting research productivity. Watch for upcoming support for Nerfstudio’s interactive visualization tools, and a wide range of data import/export formats.
Fueling a growing, community-driven ecosystem
The open-source nature of gsplat, combined with the power of 3DGUT, invites a broader community of researchers, developers, and creators to contribute, extend, and innovate. Whether you’re building new neural rendering algorithms, integrating into existing 3D pipelines, or simply exploring ideas, this integration provides a robust foundation backed by cutting-edge NVIDIA research and the gsplat collaborative ecosystem. Together, they democratize high-performance 3D rendering and accelerate progress across academia and industry.
Empowering physical AI development
3DGUT is a powerful enabler for physical AI applications where visual realism is vital, including robotics, autonomous vehicles, and embodied agents. By modeling complex camera phenomena such as fisheye lens distortion and rolling shutter effects, 3DGUT solves many of the challenges that typically accompany data collected in real world applications. This level of fidelity is essential for training robust vision systems that must operate reliably under dynamic motion and wide-angle optics typical of real-world robotic sensors.
3DGUT also supports secondary lighting effects, including reflections, refractions through unification with 3DGRT. This is critical for synthetic object insertion and multibounce illumination. These lighting cues significantly affect how AI agents interpret depth, surface boundaries, and object material properties—making their inclusion vital for testing and improving behavior in complex, uncertain conditions.
Integrated with gsplat’s fast, flexible rendering backend and the interactive tools from the Nerfstudio ecosystem, this setup enables rapid experimentation and iteration, which are critical for advancing research and development in physical AI.
Check out Videos 3 and 4 for a comparison of reconstructed scenes from a regular pinhole camera (Video 3) and a fisheye camera enabled with 3DGUT (Video 4).
Get started with 3DGUT
3DGUT and 3DGRT are open-source and available to the global developer and research community. Explore the code and contribute through the?nv-tlabs/3dgrut GitHub repo. In addition, 3DGUT is now integrated into the widely adopted open-source gsplat library, enabling seamless experimentation, plug-and-play rendering pipelines, and deployment-ready performance.
To get started, check out the comprehensive documentation and quick-start guides for both the NVIDIA 3DGUT implementation and the gsplat integration with 3DGUT, including tutorials on using it for real-time visualization and multiformat dataset support.?
NVIDIA 3DGUT Sweepstakes?
Researchers and developers are invited to showcase their creativity by generating innovative scenes using 3DGUT in gsplat or 3DGRUT (3DGUT plus 3DGRT) in the NVIDIA 3DGUT Sweepstakes. Participants will have the opportunity to win an NVIDIA RTX 4090 GPUs, accelerating future research and development.
