miles

Product Introduction

1. Definition: Miles is an open-source, production-grade reinforcement learning (RL) framework specifically engineered for the post-training phase of large language models (LLMs) and vision models. It falls into the technical category of distributed machine learning systems for agentic AI.
2. Core Value Proposition: Miles exists to solve the significant engineering complexity and instability associated with scaling reinforcement learning for frontier-scale models. Its primary value is enabling AI labs and enterprises to conduct simple, efficient, and stable training for tasks like Supervised Fine-Tuning (SFT), RL (e.g., PPO, GRPO), and on-policy distillation (OPD) on models with hundreds of billions to trillions of parameters.

Main Features

1. Native Agentic RL Engine: Miles is built from the ground up for agentic workloads. It features a native SGLang integration for high-throughput rollouts, a fully asynchronous architecture, and a Token-in-Token-Out (TITO) data path that preserves exact token streams and metadata from tool-using agents. Its Rollout Routing Replay (R3) and peer-to-peer (P2P) weight update systems optimize data flow and model synchronization during distributed training.
2. Stable & Scalable Training Backend: The framework provides a stable training foundation with support for advanced numerical formats like NVFP4 and MXFP8 for memory efficiency. It offers features such as optimizer state disk offload and zero-KL stabilization techniques. Crucially, Miles supports multiple distributed training backends, including Megatron-LM for 3D parallelism and FSDP2, allowing teams to choose the optimal strategy for their model architecture and hardware cluster.
3. Comprehensive Post-Training Recipes: Miles provides ready-to-use, battle-tested recipes for the entire post-training stack. This includes implementations for SFT, RL algorithms like GRPO and PPO, and on-policy distillation. These recipes support both full-parameter fine-tuning and parameter-efficient methods like LoRA, offering flexibility for different computational budgets and research goals.

Problems Solved

1. Pain Point: The extreme engineering burden and instability of running long-horizon, large-scale reinforcement learning training jobs. Traditional methods struggle with coordinating rollouts across thousands of GPUs, managing variable-length trajectories from tool-using agents, and maintaining training stability over weeks-long runs.
2. Target Audience: AI research scientists and engineering teams at labs and enterprises working with frontier models (e.g., DeepSeek, Qwen, Nemotron). This includes roles like Machine Learning Engineers, Research Scientists, and DevOps engineers focused on LLM post-training, agent fine-tuning, and scalable RL infrastructure.
3. Use Cases: Training a trillion-parameter model on a scientific reasoning environment with thousands of rollout steps; fine-tuning a coding agent with tool use across a massive cluster of NVIDIA Blackwell GPUs; performing stable, efficient on-policy distillation from a large teacher model to a smaller student model using a custom reward function.

Unique Advantages

1. Differentiation: Unlike general-purpose ML frameworks or narrower RL libraries, Miles is a vertically integrated solution specifically for post-training at scale. It combines a high-performance, agent-aware rollout engine (SGLang) with robust, multi-backend distributed training (Megatron-LM/FSDP), which is not typically found in a single, cohesive open-source package.
2. Key Innovation: The Token-in-Token-Out (TITO) architecture and Rollout Routing Replay (R3). TITO ensures perfect fidelity of the agent's interaction token stream (including tool calls and observations) between the rollout environment and the training loop. R3 intelligently manages the lifecycle of trajectory data, enabling fully asynchronous training where rollout workers and trainers operate concurrently without blocking, dramatically improving hardware utilization.

Frequently Asked Questions (FAQ)

1. How does Miles handle training data privacy and model checkpoint control? Miles is designed to run on your own infrastructure (on-premise or private cloud). All training data, generated trajectories, intermediate checkpoints, and final model weights remain within your controlled environment, giving you full ownership and governance over the entire AI training pipeline.
2. Can I use Miles with my existing custom agent environment or evaluation harness? Yes. Miles provides explicit integrations for popular agent frameworks like Harbor, HUD, and NeMo Gym, and sandboxes like Modal and E2B. More importantly, it allows you to plug in custom rollout, reward calculation, and data-processing functions without modifying the core training loop, ensuring compatibility with proprietary environments.
3. What hardware accelerators does the Miles framework support? Miles offers extensive hardware support, including NVIDIA's Ampere, Hopper, and Blackwell GPU architectures with native CUDA support. It also supports AMD Instinct GPUs via the ROCm software platform, providing flexibility in data center hardware selection.
4. Is Miles only suitable for reinforcement learning, or can it be used for Supervised Fine-Tuning (SFT)? Miles is a comprehensive post-training framework. While its architecture is optimized for RL, it includes high-performance, scalable recipes for Supervised Fine-Tuning (SFT) and On-Policy Distillation (OPD), making it a unified tool for multiple critical stages of the model alignment pipeline.
5. What is the fastest way to evaluate Miles for a specific model like Qwen or DeepSeek? The recommended starting point is the Docker-based quick start guide, which runs a complete GRPO training loop on a supported model architecture. You can then incrementally replace the demo model, dataset, and reward function with your own targets, allowing for rapid prototyping without building an RL stack from scratch.