oMLX

Product Introduction

1. Definition: oMLX is a native macOS inference server application built on Apple's MLX framework. It is a technical solution that transforms Apple Silicon Macs into a local, high-performance server for running Large Language Models (LLMs), Vision-Language Models (VLMs), embedding models, and reranker models.
2. Core Value Proposition: oMLX exists to eliminate the prohibitive latency—specifically Time-To-First-Token (TTFT)—experienced when using local LLMs for interactive tasks like AI-powered coding. Its primary value is enabling near-instantaneous agent responses (under 5 seconds) by implementing a persistent, tiered Key-Value (KV) cache that survives context shifts and server restarts, a critical failure point for other local inference solutions.

Main Features

1. Paged SSD KV Caching: This is the core innovation. oMLX implements a two-tier caching architecture where KV cache blocks are managed in pages. Frequently accessed "hot" blocks reside in RAM, while "cold" blocks are persisted to SSD in the efficient safetensors format using an LRU (Least Recently Used) policy. This allows the server to instantly restore previously computed context from disk, bypassing recomputation. It makes the cache effectively infinite and persistent across requests and application restarts.
2. Continuous Batching with MLX-LM: The server utilizes mlx-lm's BatchGenerator to handle concurrent inference requests simultaneously instead of queuing them. This feature maximizes hardware utilization on Apple Silicon, providing up to a 4.14x speedup in token generation throughput at 8x concurrency compared to processing single requests.
3. Native macOS Menu Bar Application & Dashboard: oMLX is a fully native Swift application, not an Electron wrapper. It runs from the macOS menu bar, providing a system-integrated way to start, stop, and monitor the inference server. A companion web dashboard offers model management, a chat interface, real-time performance metrics, and a built-in Hugging Face model downloader.
4. Multi-Model Serving & API Compatibility: The server can load and manage multiple model types (LLM, VLM, embedding, reranker) simultaneously in memory with LRU eviction. It exposes both OpenAI-compatible (/v1/chat/completions) and native Anthropic-compatible (/v1/messages) API endpoints, making it a drop-in replacement for services like OpenAI in tools such as Claude Code, Cursor IDE, and OpenClaw.
5. Advanced Tool Calling & MCP Support: oMLX supports multiple tool-calling formats including JSON mode, Qwen, Gemma, GLM, and MiniMax styles. It integrates with the Model Context Protocol (MCP) and includes features like tool result trimming to manage context window usage efficiently.

Problems Solved

1. Pain Point: The extreme latency (30-90 seconds TTFT) for local LLM coding agents when context windows shift or the cache is invalidated, which breaks developer workflow and makes local AI impractical for real-time use.
2. Target Audience: Software developers and engineers using AI-powered coding assistants (Cursor, Claude Code), researchers running local LLM inference on Apple hardware, and developers building applications that require a local, private OpenAI/Anthropic API endpoint.
3. Use Cases: Providing a sub-5-second response time for coding agents within IDEs; running a private, high-throughput inference endpoint for application development; conducting local AI research and model testing on macOS; using vision-language models with persistent caching for document analysis.

Unique Advantages

1. Differentiation: Unlike Ollama or LM Studio which hold the KV cache purely in volatile RAM, oMLX's persistent SSD caching ensures no computational work is ever permanently lost. This fundamentally changes the economics of repeated, iterative local inference, especially for agentic workflows. Its native macOS integration and dual API compatibility also offer a more seamless user experience.
2. Key Innovation: The implementation of a paged, tiered (RAM + SSD) KV cache that is explicitly designed for the stateful, non-linear nature of AI agent interactions. This approach, combined with deep integration with the MLX framework for Apple Silicon, is its technological cornerstone.

Frequently Asked Questions (FAQ)

1. How does oMLX's performance compare to LM Studio or Ollama on Mac? oMLX is specifically optimized for iterative, agentic workflows where context changes frequently. While raw single-request speed may be similar, oMLX dramatically reduces Time-To-First-Token (TTFT) on subsequent turns in a conversation—from minutes to seconds—by restoring cached context from SSD, whereas other tools must recompute it entirely.
2. What are the minimum system requirements to run oMLX effectively? You need an Apple Silicon Mac (M1 or later) running macOS 15 Sequoia or higher. A minimum of 16GB RAM is required, but 64GB or more is strongly recommended for running larger, more capable models like the 122B parameter Qwen3.5 MoE model comfortably.
3. Can I use models I've already downloaded for LM Studio or llama.cpp with oMLX? Yes. oMLX reads from the standard Hugging Face cache directory (~/.cache/huggingface/hub), which is shared by most popular AI tools. It can also directly import models from your LM Studio models folder, eliminating the need to re-download model files.
4. Is oMLX compatible with the Claude Code extension and Cursor IDE? Yes, fully. oMLX provides an Anthropic-compatible /v1/messages API endpoint. The built-in web dashboard includes a configuration generator that provides the exact terminal command needed to configure Claude Code, Cursor, or OpenClaw to use your local oMLX server as its backend.
5. What model architectures and types does oMLX support? oMLX supports any model converted to the MLX format, which includes major families like Qwen, LLaMA, Mistral, Gemma, DeepSeek, GLM, and MiniMax. It supports Vision-Language Models (VLMs) for image analysis, standard text generation, embedding models, and rerankers. It also has special handling for reasoning tags (e.g., <think>) in models like DeepSeek.