Open Index

Product Introduction

1. Definition: Open Index is an open-source structured context management layer and MCP (Model Context Protocol) server for AI agents. It is a technical framework designed to build, maintain, and query a domain-specific knowledge graph, referred to as a "brain."
2. Core Value Proposition: It exists to solve the critical challenges of AI agent context management, specifically context poisoning, contradictions, non-determinism, and navigation difficulties. Its primary value is enabling developers to build self-learning, context-aware, and domain-specialized AI agents with a persistent, validated, and structured memory system.

Main Features

1. Structured Brain Primitives: The system is built on four core primitives. A doc_type defines a concept (e.g., customer, service). A doc_schema defines the fields and search configuration (semantic or syntactic with custom boost weights) for that type. An entity is a single instance of a doc_type, stored as JSON. The related_to field within every entity creates graph edges using a target and relationship_edge_meaning, forming the knowledge graph without a separate graph database.
2. Dual Storage Policy: Open Index features a sophisticated storage model. Entities can be storage: file, where JSON files in entities/ are the git-tracked source of truth, ideal for curated data. Alternatively, storage: index entities are owned by the search database (SQLite or OpenSearch), perfect for high-volume, connector-pulled, or temporal data, preventing repository churn.
3. Full MCP Server Integration: It operates as a full-featured MCP server over stdio or HTTP, providing agents with dynamic navigation_guidelines() for prompt injection, search_brain() and get_entity() for context retrieval, and put_entity()/create_doc_type() for validated context writing. The default mode is read-write, enabling agents to both consume and maintain knowledge, with a --read-only flag for consumption-only use cases.
4. Pluggable Search Backend & Connector Framework: The system uses a pluggable search interface. The default is SQLite with FTS5 for zero-dependency local development. For production, it supports OpenSearch for scalable, cloud-based agent deployments. An optional connectors/ framework allows scheduled ingestion of entities from external MCP servers, automating brain population.

Problems Solved

1. Pain Point: It directly addresses the unstructured nature of markdown and text-based context for AI agents, which leads to hallucinations, outdated information (context decay), and inefficient retrieval. It replaces "prompt stuffing" with a queryable, structured knowledge base.
2. Target Audience: The primary users are AI engineers and developers building domain-specialized agents (e.g., for customer support, legal, sales, infrastructure/SRE). Secondary users are technical teams (DevOps, platform engineering) needing to model complex domains like microservices architecture or incident management for AI-assisted operations.
3. Use Cases: Essential for building a customer support agent that understands products, common issues, and customer segments. Critical for an SRE agent that knows all services, their dependencies, runbooks, and dashboards. Vital for a sales agent with structured context on leads, accounts, and deal stages. It is also used for personal knowledge management agents.

Unique Advantages

1. Differentiation: Unlike vector databases or simple document stores, Open Index enforces a strict schema (doc_schema) and entity relationship model (related_to), providing structure and validation that pure semantic search lacks. Compared to building a custom graph database, it offers a batteries-included, agent-native layer with MCP integration and a management CLI out of the box.
2. Key Innovation: The dual storage: file | index policy is a key innovation. It seamlessly blends human/agent-curated, version-controlled data (file) with volatile, machine-generated data (index) in a single system, resolving the conflict between git-based workflows and dynamic agent memory. The related_to field as a universal graph edge constructor is a simple yet powerful abstraction for building context graphs.

Frequently Asked Questions (FAQ)

1. How does Open Index handle context updates and avoid staleness? Open Index manages staleness through its connector framework for scheduled ingestion from sources like MCP servers, agent write-back via the put_entity MCP tool, and a manual open-index index command to reconcile storage: file entities. The analytics dashboard tracks fetch patterns, highlighting gaps for proactive updates.
2. Can Open Index be used with cloud-based AI agents like ChatGPT? Yes, for production cloud agents, you deploy the open-index serve HTTP MCP server with bearer-token authentication. Agents connect remotely, and you can configure the search backend to use OpenSearch instead of SQLite for scalability and multi-agent access to a shared brain.
3. What is the difference between Open Index and a vector database like Pinecone? While both aid retrieval, Open Index is a structured context layer. It combines semantic and syntactic (boosted field) search with enforced schemas and explicit relationship graphs. A vector database is an unstructured embedding store; Open Index can use one as a backend (via OpenSearch) but adds the crucial schema, entity, and relationship management on top.
4. Is Open Index suitable for very large-scale knowledge graphs? For large-scale deployments, the recommended path is to use the OpenSearch backend, which is designed for distributed, large-scale search. The SQLite + FTS5 backend is intended for local development and smaller, single-brain instances.
5. How do I get started building a brain for my specific domain? Run open-index init my-brain to scaffold the directory structure. Use open-index add-doc-type to define your domain concepts (e.g., product, bug). Add entity JSON files manually, via open-index import, or let an agent create them via the MCP put_entity tool. Finally, run open-index index to build the searchable brain and open-index ui to explore it.