colibri

Product Introduction

1. Definition: colibrì is a pure C, zero-dependency inference engine specifically designed for running frontier-scale Mixture-of-Experts (MoE) large language models (LLMs). It is a technical framework that enables efficient, token-exact model execution outside of specialized data centers.
2. Core Value Proposition: It exists to democratize access to state-of-the-art AI by solving the critical problem of massive model size. Its primary innovation is streaming expert weights from disk (NVMe) instead of loading the entire multi-trillion-parameter model into RAM, enabling models like the 2.8-trillion-parameter Kimi K3 to run on consumer hardware with as little as 32 GB of memory.

Main Features

1. Hierarchical Expert Staging: The engine treats the thousands of experts in an MoE model as a storage hierarchy. It uses measured routing heat from the model's router (which runs a layer ahead for prefetching) to dynamically decide which experts reside in VRAM, which are kept in pinned RAM, and which are streamed from high-speed NVMe storage. This intelligent data staging is the core mechanism that enables running models larger than total system memory.
2. Pure C & Zero-Dependency Codebase: The entire engine is written in pure C with no external dependencies (Apache 2.0 license). This results in exceptional portability, minimal binary size, and direct hardware control, allowing it to run on a vast range of systems from servers to laptops without complex dependency management.
3. Multi-Backend & Multi-Model Support: colibrì supports multiple compute backends (CUDA, Metal, CPU) and currently runs eight different frontier model families (including GLM-5.2, Qwen3.8-Flash-Next, DeepSeek V4 Flash, and Kimi K3). It provides a unified engine for diverse architectures, handling features like gated recurrence, MLA attention, sparse attention, and vision/audio towers while maintaining token-exact validation against reference implementations.

Problems Solved

1. Pain Point: The prohibitive hardware cost and resource requirement (often hundreds of GBs of VRAM) to run cutting-edge Mixture-of-Experts models, which traditionally seals them inside corporate data centers and limits access for researchers, developers, and enthusiasts.
2. Target Audience: AI researchers conducting ablations or model analysis, software developers integrating frontier models into applications, hobbyists and tinkerers curious about state-of-the-art AI, and organizations seeking to run large models on existing or cost-effective hardware.
3. Use Cases: Running a 2.8T parameter model on a single 32GB machine for research; performing quality ablations and quantization tests on the OLMoE model; deploying a local, private API server with an OpenAI-compatible interface using models like GLM-5.2; benchmarking model performance across diverse community hardware configurations.

Unique Advantages

1. Differentiation: Unlike conventional inference engines (e.g., vLLM, llama.cpp) that require the active working set of the model to fit in memory, colibrì is specifically architected for out-of-core computation of MoE models. It prioritizes correct, bit-identical output ("speed never buys drift") over pure speed, and its pure C codebase offers unparalleled simplicity and transparency compared to complex, dependency-heavy frameworks.
2. Key Innovation: The "expert-as-storage-hierarchy" paradigm and the ahead-of-time router execution for latency hiding. The system does not view all weights as resident state but as data to be staged, which is a fundamental shift in inference engine design for the MoE era. This is complemented by the publication of "expert atlases" (like the visualized 13,260 experts of GLM-5.2) for model interpretability.

Frequently Asked Questions (FAQ)

1. How does colibrì run models bigger than my RAM? colibrì uses a hierarchical staging system that streams only the necessary Mixture-of-Experts weights from your NVMe SSD into RAM and VRAM as needed for each token, based on the model's routing decisions, instead of loading the entire multi-terabyte model into memory.
2. Is colibrì's output accurate compared to standard implementations? Yes. The colibrì engine is rigorously validated to be token-exact against the original reference transformer implementations (like Hugging Face), ensuring no drift in model quality or reasoning capability despite its streaming optimization.
3. What hardware do I need to run colibrì? You can run it on a wide range of hardware. The minimum proven configuration is a system with 32 GB of RAM and a fast NVMe SSD for the 2.8T parameter Kimi K3 model. It supports CPU-only execution (e.g., on AWS Graviton4), NVIDIA CUDA GPUs, and Apple Silicon via a Metal backend.
4. What models are supported by the colibrì engine? As of v1.10.2, colibrì supports eight model families: GLM-5.2/5.3, GLM-5.3-Flash, Qwen3.8-Flash-Next, Kimi K3, Inkling, DeepSeek V4 Flash, Qwen3.6, and OLMoE, covering a range of capabilities from text to vision and audio.
5. Can I use colibrì as an API server? Yes. colibrì includes a built-in, OpenAI-compatible API server and web dashboard, allowing you to deploy it as a local drop-in replacement for services like OpenAI's API, making it easy to integrate with existing applications.