TRELLIS.2

Product Introduction

1. Definition: TRELLIS.2 is a state-of-the-art, large-scale 3D generative AI model and framework developed by Microsoft. It is a 4-billion-parameter Diffusion Transformer (DiT) model specifically engineered for high-fidelity image-to-3D generation. Its core innovation is the "O-Voxel," a novel, field-free sparse voxel representation that serves as a native and compact structured latent space for 3D data.
2. Core Value Proposition: TRELLIS.2 exists to solve the fundamental inefficiencies and quality limitations in 3D content generation. It provides a structured, memory-efficient, and high-fidelity pipeline for converting 2D images into fully textured, production-ready 3D assets with complex geometries and Physically Based Rendering (PBR) materials, significantly accelerating 3D content creation workflows.

Main Features

1. O-Voxel Representation: This is the foundational technology of TRELLIS.2. Unlike traditional implicit neural representations (INRs) or signed distance fields (SDFs) that define surfaces through continuous functions, O-Voxel is a discrete, sparse voxel grid. It directly represents occupancy and attributes, enabling "field-free" modeling. This allows it to natively handle arbitrary topologies—including open surfaces (cloth, leaves), non-manifold geometry, and internal structures—without the artifacts or lossy conversions associated with iso-surface extraction from neural fields.
2. High-Resolution & Efficient Generation: The model leverages a Sparse 3D Variational Autoencoder (SC-VAE) with 16x spatial downsampling to compress 3D assets into a compact latent space. This enables the 4B-parameter DiT backbone to generate high-resolution outputs (up to 1536³ voxels) efficiently. For example, it can generate a 512³ resolution asset in approximately 3 seconds on an NVIDIA H100 GPU. The pipeline is fully feed-forward, requiring no test-time optimization (like NeRF) or lengthy post-processing.
3. Full PBR Texture Modeling: Beyond geometry, TRELLIS.2 generates comprehensive surface material attributes. It models a full PBR material set, including Base Color, Roughness, Metallic, and Opacity maps. This enables photorealistic rendering with realistic light interaction and support for transparent materials directly out of the model, moving beyond simple vertex colors or albedo textures.
4. Bidirectional Instant Processing: The framework provides near-instant conversion between standard 3D formats and its internal O-Voxel representation. It can convert a textured mesh to O-Voxel in under 10 seconds on a single CPU core and reconstruct a mesh from O-Voxel in under 100 milliseconds on CUDA. This facilitates seamless integration into existing pipelines for editing, rendering, or further processing.

Problems Solved

1. Pain Point: Traditional 3D generative models and neural fields (e.g., NeRFs, 3D Gaussian Splatting) often produce unstructured, "soup-like" geometry that is inefficient for storage, editing, and rendering. They struggle with complex topologies and high-resolution, textured output, requiring significant post-processing and optimization time.
2. Target Audience: The primary users are researchers and developers in 3D computer vision and generative AI. Secondary users include 3D artists, game developers, and professionals in VFX, AR/VR, and digital twin creation who need to rapidly prototype or generate 3D assets from reference images.
3. Use Cases: Key scenarios include: Concept Art to 3D Model for game asset creation; E-commerce Product Visualization by generating 3D models from product photos; Rapid Prototyping in industrial design; and Dataset Augmentation for training other vision models by synthesizing diverse 3D objects with ground-truth geometry and materials.

Unique Advantages

1. Differentiation: Compared to other image-to-3D tools, TRELLIS.2 uniquely combines high-fidelity geometry and material generation in a single, efficient feed-forward pass. Unlike methods that output point clouds or untextured meshes, it produces production-ready assets. Versus pure neural field approaches, its O-Voxel output is inherently structured, compact, and instantly convertible to standard mesh formats without Marching Cubes.
2. Key Innovation: The core innovation is the structured latent space built upon the O-Voxel representation. This structure allows the model to learn and generate 3D data in a native, organized format that mirrors how 3D content is ultimately used (as textured meshes), bridging the gap between neural generation and practical application. The use of sparse convolutions (via FlexGEMM) and a hierarchical VAE enables scaling to 4B parameters while maintaining efficiency.

Frequently Asked Questions (FAQ)

1. What is the difference between TRELLIS.2 and other AI 3D generators like TripoSR or Large Reconstruction Model (LRM)? TRELLIS.2 is a 4B-parameter model focused on generating both high-quality geometry and full PBR textures in a structured O-Voxel format, suitable for direct use in 3D pipelines. Many other models are smaller, may not generate high-quality textures, or output unstructured representations like NeRFs that require additional conversion.
2. What are the hardware requirements to run TRELLIS.2 inference? Running the full TRELLIS.2-4B model for inference requires an NVIDIA GPU with at least 24GB of VRAM (tested on A100/H100). The installation has specific dependencies like CUDA 12.4 and requires compilation of several custom libraries (FlexGEMM, CuMesh, O-Voxel).
3. Can I train or fine-tune TRELLIS.2 on my own custom 3D dataset? Yes, the repository provides the complete training code. You must first preprocess your textured mesh dataset into the O-Voxel format using the provided data_toolkit. The training scripts support distributed training across multiple GPUs and nodes for large-scale experiments.
4. What 3D file formats can TRELLIS.2 import and export? The framework primarily works with its internal O-Voxel format for efficiency. However, it includes utilities to convert standard textured meshes (like OBJ/GLTF with materials) to O-Voxel and can export generated assets to GLB format with PBR textures (Base Color, Roughness/Metallic, Normal maps) and optimized geometry.
5. Does TRELLIS.2 support text-to-3D generation? The currently released model and demo are specifically for image-to-3D generation. The underlying architecture and latent space could potentially be adapted for text conditioning, but this is not a feature of the initial release.