DeepGEMM

Product Introduction

1. Definition: DeepGEMM is a high-performance, unified CUDA kernel library specifically designed for accelerating core linear algebra operations, particularly General Matrix Multiplication (GEMM), on NVIDIA GPUs. It falls under the technical categories of GPU-accelerated computing, deep learning infrastructure, and scientific computing libraries.
2. Core Value Proposition: DeepGEMM exists to solve the critical performance bottleneck of matrix multiplication in modern AI workloads, such as large language model (LLM) training and inference. Its primary value is delivering near-peak hardware performance through meticulously hand-optimized, runtime-compiled kernels that support cutting-edge data types like FP8 and FP4, while maintaining a clean and accessible codebase for educational and production use.

Main Features

1. Unified High-Performance Kernels: The library consolidates essential primitives for modern LLMs into a single codebase. This includes optimized GEMMs for FP8, FP4, and BF16 data types, fused Mixture-of-Experts (MoE) operations, and Multi-Query Attention (MQA) scoring kernels. All kernels are compiled Just-In-Time (JIT) via its DeepJIT system, eliminating the need for lengthy CUDA compilation during installation and allowing for runtime specialization.
2. Fused Mega MoE with Overlapped Communication: A flagship feature is the "Mega MoE" kernel, which fuses expert routing, two linear projections (FP8xFP4 or FP8xFP8), SwiGLU activation, and result combination into a single mega-kernel. Crucially, it overlaps NVLink inter-GPU communication with tensor core computation, dramatically reducing latency in distributed MoE model execution. It requires symmetric memory allocation across processes.
3. Advanced Data Type & Layout Support: DeepGEMM provides robust support for next-generation numerical formats critical for AI efficiency. It natively handles FP8 and even 4-bit FP4 weights for GEMM, alongside traditional BF16. For SM100 (Blackwell) GPUs, it utilizes a packed UE8M0 format for scaling factors. The library supports various memory layouts (NT, TN, NN, TT) on SM100 and offers specialized "contiguous" and "masked" grouped GEMM APIs optimized for MoE workloads during training/prefill and inference/decoding, respectively.
4. Lightweight & Educational Design: Unlike monolithic frameworks like CUTLASS, DeepGEMM is intentionally built with a limited number of core kernel functions. It leverages concepts from CUTLASS and CuTe but avoids heavy template metaprogramming, making the source code a valuable, clean resource for developers and researchers to study state-of-the-art GPU kernel optimization techniques for tensor cores.

Problems Solved

1. Pain Point: Suboptimal computational throughput in deep learning and HPC applications due to inefficient GEMM kernel implementations that fail to saturate modern GPU tensor cores, especially with new data types like FP8.
2. Target Audience: The primary users are deep learning engineers, AI researchers, and HPC developers working on GPU-accelerated applications. This includes teams building or optimizing LLM inference/training frameworks, those implementing custom MoE layers, and performance engineers seeking to extract maximum FLOPs from NVIDIA H100, H200, and Blackwell architecture GPUs.
3. Use Cases:
   • Training and Inference of Large Language Models: Accelerating the dense and MoE linear layers that form the computational backbone of models like DeepSeek-V3.
   • Scientific Computing: Providing high-performance BLAS-level primitives for computational physics, chemistry, and finance simulations.
   • Kernel Development & Education: Serving as a reference implementation for learning advanced CUDA optimization, warp-level programming, and tensor core usage.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros:
     • Peak Performance: Demonstrated performance matching or exceeding expert-tuned libraries (e.g., CUTLASS) across various matrix shapes, achieving up to 1550 TFLOPS on H800 GPUs.
     • Modern Feature Support: Early and efficient support for FP8, FP4, and fused MoE operations, addressing immediate needs in cutting-edge AI model development.
     • JIT Compilation (DeepJIT): Removes install-time compilation overhead and allows for agile kernel configuration without pre-building a vast template space.
     • Code Simplicity: The codebase is significantly more approachable than alternatives for those wanting to understand the underlying optimizations.
   • Cons:
     • NVIDIA-Only: Exclusively targets NVIDIA GPUs with SM90 (Hopper) or SM100 (Blackwell) architecture, offering no support for AMD or Intel GPUs.
     • Specialized Scope: Focused primarily on GEMM and a specific set of fused operations (MoE, MQA). It is not a general-purpose linear algebra library like cuBLAS.
     • Increased User Responsibility: Requires users to handle data layout transformations (e.g., input transposition, FP8 casting, scaling factor packing) externally, as the library focuses solely on the optimized kernel execution.
     • Advanced Configuration: Tuning parameters like SM count, tensor core utilization, and memory alignment require user knowledge for optimal results.

2. Key Alternatives & Differentiation:

   • NVIDIA cuBLAS/cuBLASLt: The standard vendor library. DeepGEMM differentiates by offering higher performance for specific shapes and data types (FP8/FP4), fused operations (Mega MoE), and a transparent, educational codebase, whereas cuBLAS is a closed-source, general-purpose black box.
   • CUTLASS: NVIDIA's open-source template library for CUDA core and tensor core GEMM. DeepGEMM is inspired by CUTLASS but is functionally different by being a curated collection of pre-defined, high-performance kernels with a JIT compilation model. DeepGEMM avoids CUTLASS's complex template metaprogramming, resulting in a simpler codebase but less flexibility for defining entirely new kernel structures.
   • Triton: A Python-like JIT compiler for GPU kernels. DeepGEMM is lower-level (written directly in CUDA/C++) and specifically hyper-optimized for GEMM and related operations, potentially offering higher performance for its target workloads. Triton offers greater productivity and flexibility for a wider range of custom kernels but may not reach the same peak performance for dense linear algebra.

Frequently Asked Questions (FAQ)

1. What is DeepGEMM used for? DeepGEMM is used to dramatically accelerate matrix multiplication and related fused operations (like Mixture-of-Experts layers) on NVIDIA GPUs, primarily for training and inferencing large language models and other deep learning workloads where computational throughput is critical.
2. How does DeepGEMM's performance compare to cuBLAS? For the matrix shapes and data types it targets (particularly FP8 and FP4 GEMM, fused MoE), DeepGEMM is designed to match or exceed the performance of NVIDIA's proprietary cuBLAS library by applying hand-tuned optimizations and leveraging JIT compilation for specific problem sizes.
3. Does DeepGEMM support AMD GPUs or older NVIDIA architectures? No, DeepGEMM is specifically optimized for NVIDIA's latest tensor core architectures, requiring an SM90 (Hopper, e.g., H100) or SM100 (Blackwell, e.g., B200) GPU. It does not support AMD GPUs or NVIDIA architectures prior to Hopper.
4. What is the difference between contiguous and masked grouped GEMM in DeepGEMM? Contiguous grouped GEMM is for training or prefill where token counts per expert are known upfront; tokens are concatenated into a single contiguous tensor with alignment. Masked grouped GEMM is for inference decoding with CUDA Graphs, where a mask tensor indicates valid computations, allowing dynamic token counts without CPU awareness.
5. What are the main benefits of the Mega MoE kernel? The Mega MoE kernel's main benefit is latency reduction through kernel fusion and overlap. It fuses multiple operations (dispatch, two linear layers, activation, combine) into one, eliminating intermediate memory writes, and most importantly, overlaps the necessary inter-GPU communication (via NVLink) with the tensor core computation, hiding communication overhead.