turbovec

Product Introduction

1. Definition: turbovec is a high-performance, open-source vector similarity search and indexing library. Technically, it is a vector database index built on the TurboQuant algorithm, implemented in Rust with Python bindings via PyO3/Maturin.
2. Core Value Proposition: It exists to enable fast, memory-efficient, and private nearest neighbor search for AI applications like Retrieval-Augmented Generation (RAG), without requiring a separate training phase or managed cloud service. Its primary value is delivering FAISS-level recall with superior compression (2-4 bit) and integrated filtering, all while running entirely on local hardware.

Main Features

1. Online, Training-Free Quantization: Unlike traditional Product Quantization (PQ) methods that require a separate, costly "train" step on a sample dataset, turbovec uses the data-oblivious TurboQuant algorithm. It applies a random rotation to normalized vectors, leveraging the asymptotic distribution of coordinates to pre-compute optimal Lloyd-Max quantization codebooks. A per-coordinate calibration (TQ+) step fits shift/scale parameters on the first batch of data and freezes them, allowing subsequent vectors to be added immediately without retraining or index rebuilds.
2. SIMD-Optimized Search Kernels: For maximum query throughput, turbovec implements hand-optimized assembly kernels using SIMD intrinsics. On ARM architectures (like Apple Silicon), it uses NEON instructions, outperforming FAISS IndexPQFastScan by 10-19%. On x86, it utilizes AVX-512BW (with an AVX2 fallback) for competitive performance, winning in 4-bit configurations. The kernel uses a nibble-split lookup table strategy adapted from FAISS FastScan for efficient scoring.
3. In-Kernel Filtered Search (Hybrid Retrieval): A key differentiator is the ability to perform filtered searches at the SIMD kernel level. Users can pass an allowlist of candidate IDs (e.g., from a SQL query or keyword search) to the search() function. The kernel checks a bitmask at 32-vector block granularity, skipping entire blocks with no allowed candidates and filtering individual disallowed slots during heap insertion. This avoids the common "over-fetch then filter" pattern, preserving recall and performance for selective queries.
4. Length Renormalization for Unbiased Scoring: Scalar quantization introduces a systematic downward bias in inner product estimates. turbovec corrects for this by computing and storing a per-vector correction scalar: the inner product of the rotated unit vector with its quantized reconstruction. During search, scores are multiplied by this scalar, turning the biased estimator into an unbiased one at zero runtime cost, significantly boosting recall at low bit widths (2-bit, 4-bit).
5. Stable External ID Management (IdMapIndex): Beyond the basic index, turbovec offers IdMapIndex, which maintains a mapping between internal slot indices and user-provided 64-bit external IDs. This allows for O(1) deletion by external ID and ensures that search results return these stable IDs, which is critical for production systems where documents have primary keys from an external database.

Problems Solved

1. Pain Point: Excessive Memory Footprint for Vector Search. Storing embedding vectors (e.g., float32) for large corpora consumes massive RAM (e.g., 31 GB for 10M docs). turbovec's 2-4 bit quantization reduces this to ~4 GB for the same corpus, a 16x compression, enabling larger datasets to fit in memory on affordable hardware.
2. Pain Point: Operational Complexity of Vector Index Management. Many vector indexes require careful parameter tuning, a separate training step, and periodic full rebuilds as data grows. turbovec's online, training-free ingestion eliminates this complexity; vectors can be added incrementally with no tuning or rebuilds.
3. Target Audience: Developers building privacy-sensitive or air-gapped RAG pipelines. This includes engineers in healthcare, finance, legal tech, and enterprise IT who cannot send data to external vector DB services. Also targets ML engineers and researchers needing a high-performance, local ANN benchmark that supports advanced features like in-kernel filtering.
4. Use Cases: 1) On-device or On-premise RAG: Deploying a fully local question-answering system where user data never leaves the device or VPC. 2) Hybrid Search Systems: Efficiently combining dense vector search (from turbovec) with sparse keyword search (e.g., BM25) by using the keyword results as an allowlist for the dense reranking stage. 3) Real-time Filtering: Searching a vector index within a specific tenant, category, or time window by dynamically generating an allowlist from a metadata store.

Unique Advantages

1. Differentiation vs. FAISS: While FAISS is a broader toolkit, turbovec specializes in ultra-low-bit quantization with integrated filtering. It matches or exceeds FAISS IndexPQ recall while offering online ingestion (vs. FAISS's train step) and native, efficient filtered search (vs. FAISS's often cumbersome IDSelector or post-filtering). Its SIMD kernels are competitive, winning on ARM and 4-bit x86 scenarios.
2. Differentiation vs. Managed Vector Databases (Pinecone, Weaviate): turbovec is a library, not a service. It offers zero data egress, no network latency, and no recurring costs. It provides the core search algorithm for developers to embed directly into their applications, aligning with the "bring your own index" trend for maximum control and privacy.
3. Key Innovation: The integration of the theoretically-grounded TurboQuant algorithm (data-oblivious quantization) with practical systems engineering (TQ+ calibration, length renormalization, and SIMD-optimized filtered search kernels). This combination delivers robust, production-ready performance straight from academic research, without the typical "algorithm-to-system" gap.

Frequently Asked Questions (FAQ)

1. How does turbovec's compression and search speed compare to FAISS? turbovec compresses vectors to 2 or 4 bits per dimension, achieving up to 16x memory reduction over float32. In benchmarks, its search speed beats FAISS IndexPQFastScan by 10-19% on ARM (Apple Silicon) and is competitive on x86, winning in 4-bit configurations. It maintains comparable or better recall, especially with its length renormalization correction.
2. Does turbovec require a separate training step before adding vectors? No. A core innovation of turbovec is its online, training-free ingestion. The first batch of vectors undergoes a lightweight calibration (TQ+), and subsequent batches use the frozen calibration parameters. Vectors are indexed immediately upon addition, with no separate train() call or full index rebuilds required.
3. Can I perform filtered or hybrid search with turbovec? Yes. turbovec natively supports filtered search via an allowlist parameter. You can pass a list of allowed external IDs (when using IdMapIndex) to restrict the search scope. The filtering happens inside the high-speed SIMD kernel, avoiding the performance and recall penalty of fetching and then discarding irrelevant results.
4. What frameworks does turbovec integrate with? turbovec provides drop-in replacements for popular Python frameworks. You can install extras for LangChain (turbovec[langchain]), LlamaIndex (turbovec[llama-index]), Haystack (turbovec[haystack]), and Agno (turbovec[agno]), allowing you to swap the default in-memory vector store without changing your pipeline code.
5. Is turbovec suitable for production use? turbovec is marked as "Alpha" (Development Status 3), indicating it is under active development. However, it is built in Rust for safety and performance, uses trusted publishing on PyPI, and is designed with production features like stable external IDs and efficient filtering. It is suitable for evaluation and integration into pipelines where its core advantages (compression, local operation, filtering) are critical.