omlx

Product Introduction

1. Definition: oMLX is a macOS-native, Apple Silicon-optimized LLM (Large Language Model) inference server built on the MLX framework. It functions as a persistent, high-performance backend for running local AI models.
2. Core Value Proposition: It exists to eliminate the slow, repetitive recomputation plaguing AI coding agents (like those in Claude Code, Cursor, and OpenClaw) by implementing a revolutionary paged SSD KV (Key-Value) cache. This slashes response times from 90+ seconds to under 5 seconds for repeated queries, making local AI on Mac viable for professional developer workflows.

Main Features

1. Paged SSD KV Caching: This is the core innovation. Instead of keeping the entire KV cache in volatile RAM, oMLX persists individual cache blocks to SSD in Safetensors format. It uses a two-tier architecture: frequently accessed ("hot") blocks remain in RAM, while less-used ("cold") blocks are stored on disk with an LRU (Least Recently Used) eviction policy. This allows previously computed contexts to be restored from disk in milliseconds across requests and even server restarts, eliminating recomputation.
2. Continuous Batching & Multi-Model Serving: The server utilizes mlx-lm's BatchGenerator to handle concurrent requests efficiently, achieving up to a 4.14x token generation speedup at 8x concurrency. It can load and serve multiple LLM, VLM (Vision-Language Model), embedding, and reranker models simultaneously, with LRU eviction for system memory management.
3. Native macOS Application & Comprehensive API Compatibility: oMLX includes a signed, notarized menu bar app (not Electron) for easy server control and monitoring, plus a web dashboard for model management. It provides drop-in OpenAI-compatible (/v1/chat/completions) and native Anthropic-compatible (/v1/messages) API endpoints, ensuring seamless integration with major AI coding tools. It also supports multiple tool-calling formats (JSON, Qwen, Gemma, GLM, MiniMax) and MCP (Model Context Protocol).

Problems Solved

1. Pain Point: The fundamental inefficiency of in-memory KV caches with coding agents. Agents constantly shift context (editing, debugging, reviewing), which invalidates the entire in-memory cache in traditional servers (Ollama, LM Studio), forcing a full, slow recomputation of the prompt (30-90 seconds TTFT).
2. Target Audience: macOS developers and engineers using AI-powered coding assistants (Claude Code, Cursor, OpenClaw) who require fast, persistent, and locally managed AI. Specifically, developers working on Apple Silicon Macs (M1 or later) with 16GB+ RAM (64GB+ recommended).
3. Use Cases: Essential for any development session involving an AI agent that iteratively edits or discusses the same codebase. Scenarios include: refactoring large files, debugging complex functions, or having an extended conversation where the agent needs to recall earlier code context, all requiring sub-5-second response times on the second and subsequent turns.

Unique Advantages

1. Differentiation: Unlike Ollama or LM Studio which rely on volatile in-memory caching, oMLX's persistent SSD-based cache makes previously computed token states permanently recoverable. This architectural difference directly targets and solves the specific workflow of stateful, context-shifting coding agents, whereas competitors are optimized for stateless chat.
2. Key Innovation: The implementation of a paged, two-tier (RAM/SSD) KV cache system for LLM inference on macOS. This approach, analogous to virtual memory, is the specific technology that enables infinite cache persistence (subject to disk space) and the dramatic reduction in Time-To-First-Token (TTFT) for repeated context prefixes.

Frequently Asked Questions (FAQ)

1. How does oMLX's performance compare to LM Studio or Ollama on Mac? oMLX significantly outperforms LM Studio and Ollama in coding agent workflows due to its persistent SSD KV cache. While raw single-request speed may be similar, oMLX eliminates recomputation penalties, reducing TTFT from 90+ seconds to under 5 seconds on subsequent turns within the same session, a critical advantage for iterative coding.
2. What are the minimum system requirements to run oMLX effectively? You need an Apple Silicon Mac (M1, M2, M3, or later) running macOS 15 (Sequoia) or higher. A minimum of 16GB of unified memory is required, but 64GB or more is strongly recommended for comfortable operation with larger quantized models (e.g., 70B+ parameter models).
3. Do I need to redownload all my Hugging Face models to use oMLX? No. oMLX reads from the standard Hugging Face cache directory (~/.cache/huggingface/hub), which is shared with other tools like Transformers and llama.cpp. It can also import models from your LM Studio application folder and custom directories, so existing model downloads are fully compatible.
4. Is oMLX compatible with Claude Code and Cursor? Yes, oMLX provides full drop-in compatibility. It offers both an OpenAI-compatible API endpoint (/v1/chat/completions) for Cursor and a native Anthropic-compatible endpoint (/v1/messages) for Claude Code and OpenClaw. Its web dashboard includes a config generator that provides the exact terminal command needed for each tool.
5. What model formats and types does oMLX support? oMLX supports any model in the MLX format available on Hugging Face. This includes major families like Qwen, Llama, Mistral, Gemma, DeepSeek, GLM, and MiniMax. It also supports Vision-Language Models (VLMs) and automatically handles reasoning tags (e.g., <think>) for models like DeepSeek.