What Is Text-to-3D: How NeRF, Gaussian Splatting, and Mesh Diffusion Turn Text Prompts into 3D Assets

The strange thing about text-to-3D generation is that the most capable systems have never seen a 3D object during training. They learn geometry from photographs — from how shadows fall across a surface, how specularity shifts with viewing angle, how occlusion reveals depth. What looks like spatial understanding is, at its core, a very sophisticated theory of light.

That observation opens something important: if a diffusion model can predict what a scene would look like from any viewpoint, those predictions can serve as constraints to carve a 3D representation out of empty space — iterating until the shape renders consistently from every direction. No 3D scan required. No annotated depth map. Just 2D predictions, applied repeatedly to a 3D problem.

How a 2D Diffusion Model Carves 3D Space

Before 2022, generating 3D from text required training data that barely existed at useful scale: millions of labeled, textured 3D models with consistent coordinate systems and material properties. The engineering question wasn’t architectural — it was epistemic. Could a model trained exclusively on 2D images carry enough implicit knowledge about three-dimensional structure to be useful? And if so, how do you extract that knowledge into a form a renderer can actually use?

The answer turned out to require a single conceptual shift: treating 3D reconstruction not as a direct generation problem, but as an optimization problem with a 2D critic.

What is text-to-3D generation?

Text-to-3D generation is the task of producing a 3D representation from a natural language prompt — a full geometric structure with surface properties, not a flat image. The output can take several forms: an implicit volumetric field, a collection of geometric primitives, or a polygon mesh ready for a game engine. What distinguishes it from image generation is the additional degree of freedom: the output must remain consistent across any viewing angle, not just look plausible from a single chosen perspective.

The breakthrough that made practical text-to-3D possible was DreamFusion, published by Poole, Jain, Barron, and Mildenhall (Google Research and UC Berkeley) in September 2022 (arXiv). DreamFusion’s contribution was not a new 3D architecture. It was a new training signal — a method for using a pretrained 2D model as a geometric critic without ever touching 3D data.

How do AI models turn a text prompt into a 3D mesh or volumetric scene?

The mechanism is Score Distillation Sampling (SDS). The procedure is worth understanding precisely, because the failure modes follow directly from its structure.

Begin with a NeRF — a neural function that maps any 3D position and viewing direction to a predicted color and volume density. Initialize it randomly. Render it from a randomly sampled camera angle to produce a 2D image. Pass that image into a pretrained diffusion model as a noisy sample, conditioned on the text prompt. The diffusion model estimates how it would denoise the image toward something consistent with the prompt. That denoising gradient gets backpropagated through the differentiable renderer, back into the NeRF’s parameters — nudging the 3D field toward configurations that the diffusion model finds plausible from that viewing angle.

Not an opinion about 3D structure. A gradient signal extracted from 2D consistency, applied iteratively.

This loop runs thousands of times, sampling camera positions across a hemisphere. The NeRF converges toward geometry that the diffusion model finds consistent with the text from every direction simultaneously. Spatial structure emerges from viewpoint consistency alone.

The analogy to computed tomography is instructive: CT reconstruction infers volume density from 2D X-ray projections taken at multiple angles. SDS infers 3D geometry from 2D diffusion predictions taken at multiple viewpoints. Both methods recover a three-dimensional structure from a set of two-dimensional constraints — the difference is that CT uses physical attenuation physics, while SDS uses a learned prior over what scenes look like.

Two failure modes follow directly from SDS’s reliance on the 2D critic. The Janus problem occurs because the diffusion model has a strong frontal-view prior: without explicit orientation conditioning, the optimized NeRF grows plausible-looking geometry on every face of a head, because each individual view passes the 2D check in isolation. Research quantifies the severity: roughly 65% of naive SDS samples show multi-face artifacts; orientation-conditioned variants reduce this to below 10% (Springer). The second failure mode is over-saturation: SDS applies high classifier-free guidance weights to keep the optimization on-target, which produces outputs that are often visually oversaturated and lack fine surface detail.

Both are properties of the training signal. Not rendering bugs.

Three Ways to Hold Shape in Memory

DreamFusion established the synthesis loop; the remaining question is representational. A 3D field can be stored implicitly (as a neural function computed on demand), explicitly (as a collection of geometric primitives), or as a polygon surface. Each choice makes a different engineering contract with the downstream pipeline.

What is the difference between NeRF, Gaussian splatting, and direct mesh output in text-to-3D?

Three representations, three different trade-off profiles.

Neural Radiance Fields (NeRF, Mildenhall et al., ECCV 2020) encode a scene as a continuous function. Given a 5D input — spatial location (x, y, z) plus viewing direction (θ, ϕ) — the network outputs color and volume density. Rendering requires marching rays through the field and integrating accumulated color, which produces high-quality results but is computationally expensive: rendering a single frame at moderate resolution takes seconds on a GPU. NeRF is differentiable by construction, making it a natural target for SDS optimization. It is not directly importable into game engines or modeling software — the implicit representation has no equivalent in standard asset formats. (arXiv)

Gaussian Splatting (Kerbl, Kopanas, Leimkühler, Drettakis, ACM Transactions on Graphics / SIGGRAPH 2023) replaces the implicit field with explicit learnable 3D Gaussians — each parameterized by position, covariance (encoding shape and orientation), opacity, and spherical harmonic color coefficients. Rendering projects these Gaussians onto the image plane in depth order via a differentiable rasterizer, which is orders of magnitude faster than volumetric ray marching: real-time rendering at 1080p becomes possible. The scene representation is explicit and editable in principle, but the Gaussians are not polygon geometry — converting a Gaussian splat to an exportable mesh still requires a separate surface reconstruction step, and the resulting topology is rarely clean enough for rigging. (Inria project page)

Direct mesh diffusion — the approach pursued by tools like Meta 3D AssetGen and Microsoft TRELLIS — skips the radiance field entirely and produces a polygon surface as the primary output. Meta 3D AssetGen (July 2024) applies a Multiview Diffusion pipeline: the text prompt generates multiple views, which feed a signed distance function reconstruction stage, followed by a texture refinement transformer that operates directly in UV Mapping space. The pipeline produces PBR Materials — albedo, normal, roughness, and metallic maps — ready for physically based rendering in Unity or Unreal Engine.

Microsoft TRELLIS (CVPR 2025 Spotlight) defines a Structured LATent (SLAT) representation built on sparse voxels, training a model with up to 2 billion parameters on 500,000 3D objects. A single forward pass decodes into a Radiance Field, a Gaussian splat, or a polygon mesh — the same latent supports all three output formats. The follow-up TRELLIS.2 scales to 4 billion parameters, introduces an “O-Voxel” sparse structure, and adds full PBR material output with an image-to-3D focus. (Microsoft GitHub)

The pipeline that most practical workflows settle on sits between these poles: a Image-to-3D intermediate stage. A multiview synthesis model — such as Zero123 — takes a 2D image and predicts the scene from novel viewpoints; a reconstruction network lifts those views into geometry. One-2-3-45++ executes the full chain — text to image to multi-view synthesis to mesh — in approximately one minute.

The representational choice isn’t about which produces better geometry in a vacuum. It’s about what the output needs to do: SDS over NeRF for maximum prompt fidelity, Gaussian splatting for real-time preview, direct mesh diffusion when the asset must enter a production pipeline without intermediate reconstruction.

What the Geometry Predicts — and Where It Fails

The SDS framework generates testable predictions about output quality. If the text prompt contains strong spatial constraints — “a ceramic mug, cylindrical body, loop handle, flat base” — the diffusion critic receives consistent per-view gradients, and the 3D field converges to a coherent shape. If the prompt is vague — “a cool cup” — the critic has too much freedom, and the optimization finds a local minimum that satisfies the 2D consistency constraint without forming coherent 3D structure. Prompt specificity is not a stylistic choice here; it is a direct parameter of the loss surface.

If you need PBR outputs for a real-time renderer, the SDS-over-NeRF path is structurally wrong. NeRF outputs view-dependent color and volume density — not albedo, roughness, or metallic values. Recovering physically based material properties requires either a dedicated UV parameterization and material estimation stage, or a pipeline that was designed to output those maps natively. This is precisely what Meta 3D AssetGen and TRELLIS.2 address; both treat PBR material output as a primary design constraint rather than a post-processing step.

The bottleneck that benchmarks don’t capture is topology. Commercial tools have narrowed the gap on visual fidelity and generation speed considerably. Meshy AI (Meshy 6, up to approximately 600,000 faces, roughly one minute per model with PBR textures, Pro tier at $20 per month for 1,000 credits) handles concept iteration across an entire asset library at a pace no manual workflow can match (Meshy Docs). Tripo AI’s Tripo P1.0, a native 3D diffusion model introduced at GDC 2026, targets production-ready assets in seconds; Rodin Gen-2.5 prioritizes character and hero asset fidelity with high-polygon meshes and 4K PBR output, suited for baking workflows rather than real-time deformation.

None of these produce geometry ready for skeletal animation rigging. Mesh Topology — clean edge loops, controlled pole placement, predictable deformation under joint rotation — is optimized by human retopologists for animator intent, not by a diffusion loss for visual reconstruction fidelity. The two objectives produce systematically different mesh structures. As of mid-2026, this remains an open problem across all commercial text-to-3D tools and research models; the geometry looks correct as a still render and fails under animation.

Rule of thumb: SDS-based pipelines serve concept iteration and pre-visualization; mesh-diffusion pipelines serve production assets that will be lit but not rigged.

When it breaks: Skeletal animation rigging requires clean mesh topology that no current text-to-3D system produces; the output mesh must be manually retopologized before deformation, which offsets the generation speed advantage for character and creature assets.

Security & compatibility notes:

• OpenAI Point-E / Shap-E: No active development since 2023; both models have been superseded by commercial tools and Microsoft TRELLIS. Do not build production pipelines around them — use only for reference or archival purposes.

The Data Says

Text-to-3D works because a 2D diffusion model’s denoising gradient, backpropagated through a differentiable 3D renderer, is a surprisingly effective geometry signal — not because the model reasons about space, but because coherent 3D structure is the only configuration consistent with its 2D predictions across all viewing angles simultaneously. The three competing representations are not competing answers to the same question; they are answers to different questions about rendering speed, export compatibility, and material fidelity. The engineering choice should follow the output requirement — NeRF for fidelity under optimization, Gaussian splatting for interactive preview, direct mesh diffusion when the asset needs to enter a production pipeline without reconstruction.