Molmo 2

Product Introduction

1. Definition: Molmo 2 is an open-source suite of vision-language models (VLMs) developed by the Allen Institute for AI (AI2). It belongs to the technical category of multimodal foundation models, specifically engineered to process and reason over videos, multiple images, and text simultaneously.
2. Core Value Proposition: Molmo 2 exists to provide the research and developer community with state-of-the-art video understanding capabilities—including spatio-temporal grounding, object tracking, and dense captioning—under a permissive open license (Apache 2.0). It delivers this performance while being significantly more data-efficient and computationally accessible than larger proprietary models or prior open alternatives.

Main Features

1. Native Video & Multi-Image Understanding:

   • How it works: Molmo 2 processes sequences of video frames or sets of related images using a vision transformer (ViT) encoder. Visual tokens from each frame/image are interleaved with text tokens, frame timestamps, and image indices via a lightweight connector module before being fed into the language model backbone (Qwen 3 or Olmo). Crucially, visual tokens can attend bidirectionally across frames/images, enabling joint reasoning over space and time.
   • Technical Specs: Supports input of up to 128 frames per video clip (sampled at ≤2 fps). Employs 3x3 patch pooling to manage long-context visual sequences efficiently.

2. Spatio-Temporal Grounding & Tracking:

   • How it works: The model outputs bounding box coordinates (x, y, width, height) and precise timestamps in response to queries. For tracking, it assigns persistent object IDs that persist across occlusions and scene re-entries. This is achieved through specialized training objectives (e.g., pointing loss, tracking consistency loss) on datasets like Molmo2-VideoPoint and Molmo2-VideoTrack.
   • Capabilities: Performs counting-by-pointing (e.g., "How many times did X happen?" returns event timestamps/locations), multi-object tracking, anomaly detection, and referring expression resolution (e.g., "Find the window above the sink").

3. Efficient, Open Architecture Variants:

   • Molmo 2 (8B): Based on Qwen 3, optimized for highest accuracy in video QA and grounding.
   • Molmo 2 (4B): Also Qwen 3-based, optimized for inference speed and lower resource requirements.
   • Molmo 2-O (7B): Uses the fully open-source Olmo LLM backbone, providing an end-to-end open stack (vision encoder, connector, LLM) for researchers requiring full control and auditability.

Problems Solved

1. Pain Point: Lack of open, high-performance models capable of fine-grained video understanding (grounding answers in specific spatial regions and temporal segments). Prior open VLMs struggled with temporal reasoning, object permanence, and dense event description.
2. Target Audience:
   • AI Researchers: Needing transparent, modifiable models for video ML experimentation.
   • Robotics Engineers: Requiring real-time video scene understanding for navigation/manipulation.
   • Video Analytics Developers: Building applications for security, industrial monitoring, or content moderation needing object tracking/event detection.
   • Scientific Researchers: Analyzing experimental video data (e.g., biology, physics).
3. Use Cases:
   • Industrial Monitoring: Counting objects on a conveyor belt & flagging anomalies in real-time video feeds.
   • Autonomous Systems: Tracking multiple agents (vehicles, pedestrians) in complex traffic scenes.
   • Content Search & Summarization: Generating searchable, timestamped dense captions for long videos.
   • Scientific Video Analysis: Quantifying behaviors or events in lab recordings (e.g., cell movement, material stress tests).

Unique Advantages

1. Differentiation vs. Competitors:
   • Vs. Proprietary (Gemini 3 Pro, GPT-5): Molmo 2 outperforms Gemini 3 Pro on video tracking and is competitive on image/video QA, despite being significantly smaller (8B vs 100B+ params). It offers full transparency (weights, data, code) unlike closed APIs.
   • Vs. Open Models (PerceptionLM): Achieves superior tracking and grounding accuracy using <1/8th the video training data (9.19M vs 72.5M videos), demonstrating superior data efficiency via curated datasets and targeted objectives.
2. Key Innovations:
   • Bidirectional Visual Token Attention: Allows tokens from different frames/images to interact directly, drastically improving multi-image/video reasoning.
   • Molmo2-Cap Dataset: 431k long-form, highly descriptive video captions (avg. hundreds of words/clip) providing unprecedented supervision density for event understanding.
   • Token-Weighted Fine-Tuning: Dynamically balances learning across diverse tasks (captioning, QA, pointing, tracking) during SFT for optimal multi-task performance.

Frequently Asked Questions (FAQ)

1. What license is Molmo 2 released under?
   Molmo 2 models, training code, and newly created datasets (e.g., Molmo2-Cap, Molmo2-VideoTrack) are released under the Apache 2.0 license, permitting commercial use. Note: Some integrated academic datasets may have academic/non-commercial only restrictions – check the Tech Report for specifics.
2. What hardware is needed to run Molmo 2?
   The 4B variant can run efficiently on a single high-end consumer GPU (e.g., RTX 4090, 24GB VRAM). The 8B and 7B-O variants benefit from multiple GPUs or data center-grade A100/H100 GPUs for full-context video inference, especially with long clips.
3. How does Molmo 2 handle long videos (>128 frames)?
   It uses a SlowFast-inspired strategy: Processing key frames at high resolution and intermediate frames at lower resolution/rate. This maintains high accuracy on tasks like long-video QA while significantly reducing vision token count and computational load.
4. Can Molmo 2 process images AND videos?
   Yes, natively. Its architecture handles single images, sets of 2-5 related images (Multi-Image QA/Pointing), and video clips seamlessly within the same model, using the same core mechanisms for spatio-temporal reasoning.
5. Is Molmo 2 suitable for real-time applications?
   The 4B variant is optimized for low-latency inference and can be used in near-real-time applications depending on hardware and frame sampling rate. For strict real-time requirements (e.g., autonomous drones), further optimization or distillation may be needed.