BaseRT

Product Introduction

1. Definition: BaseRT is a high-performance, local inference runtime specifically engineered for running Large Language Models (LLMs) on Apple Silicon Macs (M-series chips). It falls into the technical category of on-device AI inference engines.
2. Core Value Proposition: BaseRT exists to provide developers and power users with the fastest possible method for running open-source LLMs locally on macOS, eliminating reliance on cloud APIs and ensuring complete data privacy. Its primary value is unmatched inference speed on Apple Silicon, enabling practical local AI applications.

Main Features

1. Apple Silicon-Optimized Runtime: BaseRT is built from the ground up to leverage the unique architecture of Apple's M-series chips, including the Neural Engine (ANE), CPU, and GPU (Unified Memory Architecture). It uses low-level Metal Performance Shaders and custom kernels to maximize hardware utilization, which is how it achieves significant performance gains over generic runtimes.
2. One-Command Installation & Serving: The product emphasizes extreme developer ergonomics. Installation is handled via a single shell command using a curl script. Serving a model is equally simple (basert serve <model-name>), abstracting away complex compilation, dependency management, and configuration typically associated with local model deployment.
3. Broad Model Format & Architecture Support: BaseRT supports a wide range of popular open-source model families and architectures, including Qwen, Llama, Gemma, Mistral, and Phi. It is compatible with common quantization formats (like Q4, Q8), allowing users to balance model size, speed, and accuracy for their specific Apple Silicon device (e.g., M5 Pro).

Problems Solved

1. Pain Point: Slow and inefficient local LLM inference on macOS. Existing solutions like llama.cpp or MLX, while functional, do not fully exploit the potential of Apple Silicon, leading to suboptimal token generation speeds (tokens/sec) and long prefill times, which hinders interactive use.
2. Target Audience: The primary personas are macOS developers building on-device AI applications, researchers prototyping with local models, and privacy-conscious power users (like engineers, writers, analysts) who want to use AI coding assistants or chatbots without sending data to third-party servers.
3. Use Cases: Essential scenarios include: running a fully local coding agent (e.g., with the pi agent framework); prototyping and testing different LLMs privately on a MacBook; building macOS-native AI applications that require fast, offline inference; and conducting performance benchmarking for Apple Silicon.

Unique Advantages

1. Differentiation: Direct benchmarks provided by Base Compute show BaseRT significantly outperforms the two main alternatives. It demonstrates up to 6.4x faster prefill than llama.cpp and 3.9x faster prefill than MLX on specific models (e.g., Qwen3 30B), with decode speed advantages of up to 1.33x. This positions it as the performance leader for this specific platform.
2. Key Innovation: The core innovation is its deep, proprietary optimization for the Apple Silicon stack. While competitors offer cross-platform support or research-focused frameworks, BaseRT's singular focus on maximizing throughput and latency on M-series chips through custom Metal shaders and runtime scheduling is its defining technical edge.

Frequently Asked Questions (FAQ)

1. How does BaseRT achieve faster speed than MLX on Apple Silicon? BaseRT uses highly optimized, low-level Metal kernels and a runtime specifically tuned for production inference workloads, whereas MLX is a broader machine learning framework designed for flexibility and research, which can incur overhead.
2. Is BaseRT free to use? Yes, BaseRT is an open-source project available on GitHub, allowing for free local use, modification, and inspection of its codebase for running LLMs on personal Apple devices.
3. What models are compatible with the BaseRT runtime? BaseRT supports a growing list of transformer-based models, including major families like Llama 3.2, Gemma 4, Qwen3.5/3.6, Mistral, and Phi-3, particularly in GGUF or similar quantized formats.
4. Can I use BaseRT to serve an API for other applications? Yes, when you run basert serve, it launches a local inference server. You can then point compatible applications (like coding agent plugins or custom scripts) to this local endpoint (typically localhost:port), similar to a local OpenAI-compatible API.
5. Does BaseRT work on Intel-based Macs or other platforms? No, BaseRT is exclusively built for and optimized for Apple Silicon Macs (M1, M2, M3, M4 series and beyond). It does not support Intel Macs, Windows, or Linux.