Nemotron 3.5 Transcribe

Product Introduction

1. Definition: NVIDIA Nemotron 3.5 Transcribe is an advanced open-weights speech-augmented language model (SALM) engineered for intelligent transcription and speech recognition tasks. Operating as a unified 30B-parameter hybrid Mamba-Transformer mixture-of-experts (MoE) architecture—featuring approximately 4.3B active parameters—it combines a Transformer speech encoder, a frozen Nemotron 3 Diarization model, and the Nemotron 3.5 Lightning LLM to process and transcribe audio signals with high precision.

2. Core Value Proposition: The model solves the rigid limitations of conventional automatic speech recognition (ASR) systems by allowing users to select transcription styles, formats, and behaviors via natural language instructions without requiring fine-tuning or retraining. Supporting 40 languages and handling recordings up to one hour long, Nemotron 3.5 Transcribe provides verbatim or written-form output, speaker attribution, contextual entity biasing, and word-level timestamps, setting a new benchmark on both the English and multilingual Open ASR Leaderboards.

Main Features

1. Intelligent Instruction-Driven Transcription: Nemotron 3.5 Transcribe utilizes natural language prompting to dynamically alter output formats, allowing users to instantly switch between verbatim transcription, clean written-form text, punctuation styles, domain-specific terminologies, and rich transcription formatting within a single model architecture.

2. Speaker-Attributed Transcription via Parallel Expert Encoder: The model incorporates a frozen diarization model whose speaker activity outputs are injected directly into the ASR encoder representations via a Parallel Expert Encoder. This technical mechanism enables high-accuracy, speaker-attributed transcription for heavily overlapping multi-speech environments, ranging from brief one-minute segments to lengthy conference calls and meetings.

3. Contextual Entity Biasing (ContextASR): Given a user-supplied list of expected domain-specific keywords, proper nouns, or brand names, the model dramatically reduces named-entity error rates (achieving up a six-fold reduction on ContextASR-Bench), ensuring highly accurate recognition of niche vocabulary without human post-correction.

4. Speaker-Attributed Word Timestamps: A lightweight Connectionist Temporal Classification (CTC) adapter aligns the generated transcript tokens directly with the underlying audio stream post-decoding, producing exact word-level timing markers mapped to individual speakers even across complex, overlapping multi-speaker recordings.

5. High-Performance Efficient Inference: Nemotron 3.5 Transcribe supports native Multi-Token Prediction (MTP) speculative decoding alongside an optional DFlash2 draft model. When deployed via the Nemotron Transcribe NIM using the NVFP4 checkpoint paired with DFlash2 on a single NVIDIA RTX PRO 6000 Blackwell GPU, the system achieves inference speeds of up to 2,238 seconds of audio per second.

Problems Solved

1. Pain Point: Traditional ASR systems commit irrevocably to a single output format (e.g., forcing raw verbatim text with disfluencies or strictly cleaned-up written prose), lack robust multi-speaker attribution in overlapping audio environments, struggle with proprietary domain-specific vocabulary, and fail to adapt to varied downstream user requirements without costly model retraining.

2. Target Audience: AI Engineers, Speech Recognition Researchers, Enterprise Software Architects, Call Center Analytics Platforms Developers, Legal and Medical Transcription Service Providers, and Multilingual Natural Language Processing (NLP) Teams.

3. Use Cases: Automated meeting minute generation with precise speaker diarization and word timestamps; real-time customer support call monitoring and analytics requiring contextual entity recognition; cross-lingual speech translation and spoken language identification across 40 distinct languages; and batch transcription processing for media archives requiring strict stylistic adherence (verbatim vs. clean written forms).

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros: State-of-the-art transcription accuracy (mean Word Error Rate of 4.71% on English Open ASR and 4.62% on multilingual Open ASR); extreme flexibility driven by natural language instructions; massive context window handling up to 1-hour audio segments; exceptional throughput via speculative decoding (NVFP4 and DFlash2); and availability under an open-weights license (OpenMDW-1.1).
   • Cons: High hardware demands for optimal local deployment due to its 30B MoE base footprint; requires modern enterprise-grade hardware (such as NVIDIA Blackwell GPUs) to fully maximize the 2,238x real-time inference throughput claims; and relies on fine-tuning toolkits like NeMo Speech for custom domain adaptations.

2. Key Alternatives & Differentiation: Compared to traditional cascaded automatic speech recognition pipelines (which chain separate acoustic, language, and diarization models), Nemotron 3.5 Transcribe functions as a unified speech LLM, eliminating error propagation between pipeline stages. Against standard end-to-end open-source models like OpenAI Whisper or Meta MMS, Nemotron 3.5 Transcribe introduces dynamic instruction-based formatting, superior contextual entity biasing, native overlapping speaker attribution, and multi-token prediction speculative inference.

Frequently Asked Questions (FAQ)

1. What is NVIDIA Nemotron 3.5 Transcribe? Nemotron 3.5 Transcribe is an open-weights speech-augmented language model (SALM) with ~4.3B active parameters (30B MoE total) designed for intelligent transcription, speaker attribution, and multi-format text generation across 40 languages using natural language instructions.

2. How does Nemotron 3.5 Transcribe handle speaker diarization and overlapping speech? The model integrates a frozen speaker diarization model whose activity representations are fed directly into the ASR encoder via a Parallel Expert Encoder, allowing it to attribute text to specific speakers accurately even during overlapping conversations.

3. What license governs the use of Nemotron 3.5 Transcribe weights? The model checkpoints are released under the OpenMDW-1.1 license, permitting flexible academic and commercial utilization alongside the NVIDIA NeMo Speech fine-tuning toolkit.

4. What hardware is required to run Nemotron 3.5 Transcribe efficiently? While the model provides multiple quantization formats (BF16, FP8, NVFP4, and GGUF), achieving maximum inference throughput of over 2,200 seconds of audio per second requires utilizing the NVFP4 checkpoint with the DFlash2 draft model running on NVIDIA Blackwell architecture GPUs like the RTX PRO 6000.

5. Can Nemotron 3.5 Transcribe be adapted to recognize proprietary industry terminology? Yes. Through its contextual transcription capabilities, users can supply a list of expected entities or domain-specific terms to dramatically reduce named-entity recognition errors, or fine-tune the model on custom tasks within 0.5 to 2.3 GPU-hours on a single GPU using NeMo Speech.