Gaussian Splatting

Gaussian Splatting is a 3D rendering technique that represents scenes as millions of overlapping semi-transparent ellipsoids, enabling real-time playback without training a neural network or using ray marching.

What It Is

Gaussian Splatting was introduced in 2023 as a way to capture and render 3D scenes without the slow ray marching that NeRF requires. The core idea is to store a scene not as voxels or a polygon mesh, but as a cloud of millions of 3D Gaussian functions — each one an ellipsoidal blob with a position in space, a shape (covariance matrix), an opacity level, and color information encoded via spherical harmonics.

Think of it like describing a sculpture by specifying thousands of translucent soap bubbles. Each bubble knows where it sits, how big and stretched it is, how see-through it is, and what color light it reflects. Stack enough bubbles together and the sculpture appears.

Unlike NeRF’s implicit neural network, Gaussian Splatting stores the scene as explicit geometry: you can inspect, move, and render individual Gaussians without querying a network. During rendering, the Gaussians are sorted by depth and projected (or “splatted”) onto a 2D image plane via GPU rasterization. Rasterization runs at the speed GPUs were designed for, which is why Gaussian Splatting can render complex scenes in real time where NeRF might take seconds per frame.

For text-to-3D pipelines specifically, Gaussian Splatting matters because it can serve as both an intermediate and a final representation. Tools that use score distillation sampling to optimize a 3D asset can work in Gaussian space, then export to a mesh for downstream use in game engines or 3D reconstruction workflows.

How It’s Used in Practice

Most people first encounter Gaussian Splatting in photorealistic 3D capture: a team photographs or films an object or environment from many angles, runs structure-from-motion to get initial camera positions, and trains a Gaussian Splat model over those images. The result is a dense cloud of Gaussians that loads into a viewer and can be panned around in real time.

In text-to-3D workflows, Gaussian Splatting plays a different role. Rather than starting from photos, the Gaussians are initialized randomly and then optimized using gradients from a diffusion model. The optimization loop adjusts each Gaussian’s position, shape, and color until the scene matches the text prompt from every viewing angle.

Pro Tip: Gaussian Splats do not export to polygon meshes on their own. If you need a mesh for a game engine or 3D print, add a mesh extraction step after the splat is trained. Tools like SuGaR or Gaussian Opacity Fields handle this conversion.

When to Use / When Not

Common Misconception

Myth: Gaussian Splatting produces a mesh you can load directly into Blender or a game engine.

Reality: A Gaussian Splat is a cloud of ellipsoids, not a polygon mesh. Blender and game engines work with meshes. You need a separate conversion step — such as SuGaR or Gaussian Opacity Fields — to extract usable geometry from the splat.

One Sentence to Remember

Gaussian Splatting gives you real-time, photorealistic rendering from a cloud of ellipsoids. If your pipeline needs a mesh afterward, it is the starting point, not the final answer.

FAQ

Q: How is Gaussian Splatting different from NeRF?
A: NeRF encodes a scene inside a neural network and needs ray marching to render it, which is slow. Gaussian Splatting stores the scene as explicit ellipsoids and renders via GPU rasterization, enabling real-time speeds.

Q: Can I use Gaussian Splatting for text-to-3D generation?
A: Yes. Diffusion models can optimize Gaussian Splats by scoring rendered views against a text prompt, using a process called score distillation sampling. The resulting Gaussians can later be converted to a mesh.

Q: How many images do you need to train a Gaussian Splat?
A: For photo-based capture, dozens to hundreds of overlapping images typically produce good results. In text-to-3D, Gaussians are initialized randomly and optimized against rendered views, so no input photos are needed.