YuE

Product Introduction

1. Definition: YuE2 is a frontier, multimodal AI music generation system that unifies symbolic music planning and high-fidelity audio synthesis within a single, end-to-end model architecture. It is a 3.58-billion parameter neural network capable of generating complete, structured songs from text prompts.
2. Core Value Proposition: YuE2 exists to bridge the gap between editable symbolic music composition (like a digital score) and state-of-the-art audio generation quality. Its primary value is enabling controllable, plan-based music AI where users can generate a song from a text description, receive an editable symbolic score (melody, chords, structure), and then have the model render that plan into a high-quality audio recording with vocals and instrumentation, all while maintaining competitive audio quality with leading commercial models like Suno v5/v6.

Main Features

1. Symbolic Planning & Audio Generation Unification: YuE2 operates through a two-stage process within a unified model. First, it performs symbolic planning, generating an editable lead sheet in ABC notation that details the song's melody, chords, rhythmic structure, and lyrical phrasing. This symbolic plan is then used to condition the audio generation stage, where the model produces semantic music tokens and decodes them into full-song, high-fidelity audio. This is enabled by a Mixture-of-Experts architecture with separate Autoregressive (AR) and Non-Autoregressive (NAR) experts sharing attention computation.
2. Zero-Shot Cover & Agentic Music Editing: Beyond text-to-song generation, YuE2 supports zero-shot music covering, where it can re-render an existing song with significant alterations to genre, instrumentation, tempo, or lyrics based on a prompt. Furthermore, it enables agentic music editing, allowing for iterative, conversational refinement of a generated song (e.g., "make the chorus louder," "change the key to major"), where each edit is grounded in the evolving symbolic score.
3. Integrated Analysis Models (MERT2 & SheetSage2): The system is supported by two auxiliary, state-of-the-art models. MERT2 is a 632M-parameter music audio representation model that provides the foundational acoustic and semantic features used for both analysis and generation within YuE2. SheetSage2 is an autoregressive audio-to-score transcription model that converts existing audio recordings into editable symbolic scores (beats, keys, chords, melody), which also supplies the symbolic training data for YuE2's planning capability.

Problems Solved

1. Pain Point: Existing AI music generators often act as "black boxes," producing audio that is difficult to meaningfully edit or iterate upon after the initial generation. There is a disconnect between the creative, structural intent (the "composition") and the generated audio output.
2. Target Audience: Music Producers & Composers seeking AI co-creation tools with fine-grained control; Researchers in AI and music information retrieval; Developers building music creation applications requiring editable output; Content Creators needing customizable, royalty-free music with specific structural elements.
3. Use Cases: Rapid Prototyping: Generating a complete song draft with a structured score from a text idea. Educational Tool: Demonstrating how changes in a score (chord progression, melody) affect the final audio. Interactive Music Creation: Using the agentic editing feature to refine a track through natural language conversation. Music Analysis Pipeline: Using SheetSage2 to transcribe existing music into a manipulable symbolic format for remixing or study.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros:
     • Unprecedented Control: The symbolic plan provides a human-readable, editable intermediate representation, offering significantly more control over the musical structure than purely audio-based generators.
     • Frontier Audio Quality: Benchmarks (WildSongBench) show its best-of-8 outputs achieve a higher mean SongBench score than Suno v5 and v6, indicating top-tier perceived audio quality and text alignment.
     • Multifunctional Pipeline: Integrates generation, covering, and editing into a cohesive system powered by a shared family of models (YuE2, MERT2, SheetSage2).
     • Transparency & Research Focus: As a research release with detailed paper, benchmarks, and available models on Hugging Face, it offers greater transparency than closed commercial APIs.
   • Cons:
     • Computational Complexity: The 3.58B parameter model and best-of-8 selection strategy imply high inference costs compared to smaller or single-pass models.
     • Niche Skill Requirement: Fully leveraging the symbolic planning feature requires some basic music literacy to understand and edit ABC notation or lead sheets.
     • Accessibility: While a free demo is hosted, full-scale access likely requires significant technical expertise to run the models locally or via API, unlike consumer-facing apps.

2. Key Alternatives & Differentiation:

   • Suno AI: The primary competitor in terms of audio quality. Differentiation: Suno is a closed, consumer-focused product optimized for ease-of-use and viral output. YuE2 is an open-research system focused on controllability via symbolic plans, editing, and transparency. YuE2 trades some user-friendliness for compositional control and reproducibility.
   • Stable Audio / AudioCraft: Open-source audio generation frameworks from Stability AI and Meta. Differentiation: These are general-purpose audio generation models (not exclusively music) or earlier-generation music models. YuE2 differentiates through its explicit two-stage symbolic-audio architecture, integrated transcription (SheetSage2), and benchmark-leading performance on specific music generation tasks.
   • Traditional DAWs with AI Plugins (e.g., AIVA): Differentiation: Tools like AIVA generate MIDI/symbolic music but often lack integrated, frontier-quality audio rendering. YuE2 unifies both in one model, and its generation is prompt-based rather than rule-based or style-imitation-based.

Frequently Asked Questions (FAQ)

1. Is YuE2 free to use? Yes, a hosted free demo is available on the NOIZ Research website to try basic generation. The full models and code are also available on GitHub and Hugging Face for research and non-commercial use, though running them requires significant computational resources.
2. How does YuE2 compare to Suno AI? YuE2 matches or exceeds Suno v5/v6 in benchmarked audio quality (SongBench on WildSongBench) while adding a unique symbolic planning layer for editable scores and agentic editing, making it more suitable for controlled music production. Suno remains more accessible as a polished consumer application.
3. What is symbolic planning in AI music generation? Symbolic planning refers to the AI first generating a structured musical blueprint—including melody notes, chords, rhythm, and song sections—in a format like ABC notation before creating the audio. This plan acts as an editable intermediate step, allowing users to modify the composition before it is rendered into sound.
4. Can YuE2 create music in different languages and genres? Yes, as demonstrated by the Genre Explorer and the WildSongBench evaluation, YuE2 is trained on a diverse dataset and can generate music across multiple languages and genres based on the text prompt provided by the user.
5. What are MERT2 and SheetSage2 used for? MERT2 is the core audio understanding model that extracts semantic features from music, used by both YuE2 and SheetSage2. SheetSage2 is a transcription model that converts audio into symbolic scores. Together, they form the analysis backbone that enables YuE2's understanding of music and its ability to learn from audio-score pairs.