Paper summary: 3D Gaussian Splatting for Real-Time Radiance Field Rendering

Problem

Radiance-field methods (NeRF and successors) produce excellent novel views of a scene from photos, but the best-quality ones are slow to train and far too slow to render in real time. Faster methods trade away quality. For full, unbounded scenes at 1080p, no method rendered in real time at top quality.

Method

Instead of a neural network queried along rays, the scene is represented explicitly as a set of 3D Gaussians:

1. Start from the sparse points that camera calibration (structure-from-motion) already produces.
2. Optimise each Gaussian’s position, anisotropic covariance (its stretched ellipsoid shape), opacity and view-dependent colour, while adding and removing Gaussians where needed (density control).
3. Render with a visibility-aware, tile-based rasteriser that “splats” Gaussians onto the image. It’s differentiable, so the same renderer drives training.

Key results

The authors report state-of-the-art visual quality with competitive training times, and real-time rendering at ≥30 fps at 1080p on several standard datasets.

Limitations

From the paper’s own limitations section:

• Artifacts appear in poorly observed regions, where other methods struggle too.
• Gaussians can become elongated or “splotchy”.
• “Popping” artifacts occasionally appear, which the authors trace partly to their simple culling and visibility ordering.

They also note that, while compact compared with earlier point-based approaches, memory use is significantly higher than NeRF-based methods. Peak GPU memory during training of large scenes can exceed 20 GB in their unoptimised prototype.