xerj

Product Introduction

1. Definition: XERJ is an AI-native, Elasticsearch-compatible search engine built as a single, static Rust binary. It is a specialized retrieval engine designed to index structured and unstructured data—including source code, documents, logs, and PDFs—enabling AI agents to query a semantic index instead of repeatedly reading raw files.
2. Core Value Proposition: XERJ exists to drastically reduce the token consumption and associated costs of AI agent workflows. It solves the problem of wasteful context window usage by providing precise, indexed answers, cutting down on the expensive output tokens agents spend when looping through files with tools like grep. Its primary value is enabling efficient agentic search, reference coding, and long-term agent memory.

Main Features

1. AST-Powered Code Search: XERJ parses source code using Tree-sitter (supporting 36 languages) into an Abstract Syntax Tree (AST). It extracts symbols, their kinds, line numbers, and a searchable definitions field. This allows an agent to retrieve a specific function with its full signature and contract in one query, rather than a list of line matches that require further context to interpret.
2. Zero-Config, Format-Sniffing Ingest (xerj autoindex): The engine automatically walks a folder, identifying files by content (not extension), covering 13 format families (JSONL, CSV, SQLite, PDF, DOCX, HTML, etc.). It infers field types and date encodings, writes explicit Elasticsearch mappings, and streams data with idempotent IDs. The process is resumable and kill-safe, recording junk files instead of failing.
3. Integrated Hybrid Search & Agent Memory API: XERJ provides native hybrid search, fusing lexical (BM25) and vector (kNN) relevance scoring using Reciprocal Rank Fusion (RRF) in a single query tree. It includes a dedicated /_memory REST API for agents to store, recall by semantic meaning or keyword, filter, and forget information, with physical namespace isolation for per-agent data separation—all without external embedding services.

Problems Solved

1. Pain Point: AI agents waste significant computational budget (output tokens) and time when they must read entire files or repositories into their limited context window to find information, a process exacerbated by using grep which only provides location hints.
2. Target Audience: Developers and engineering teams building or utilizing AI coding assistants (e.g., for reference coding); security professionals conducting AI-driven audits; teams implementing RAG (Retrieval-Augmented Generation) pipelines; and developers building AI agents that require persistent, queryable memory.
3. Use Cases:
   • Reference Coding: Clone and index open-source repositories (e.g., sled), allowing a coding agent to retrieve exact function definitions and contracts before writing code, preventing hallucination of APIs.
   • Security Audits: Efficiently query across thousands of files (e.g., 1,492 PHP files in a WordPress audit) using a fraction of the token context required to read the entire codebase.
   • Document RAG: Index folders of mixed-format documents (PDFs, DOCX, CSV) with one command, enabling complex Q&A over corporate documentation without building a custom pipeline.
   • Agent Memory: Provide a persistent, semantically searchable memory layer for AI agents to recall user preferences, session history, or learned facts across interactions.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros:
     • Token Economics: Demonstrated 2.7x reduction in output token usage for coding tasks, directly lowering LLM API costs.
     • Operational Simplicity: Single static binary (Apache 2.0); no runtime dependencies (JVM); sub-second startup; seamless Elasticsearch API compatibility.
     • Honest Hybrid Retrieval: Built-in lexical embedder works fully offline; neural encoder is an optional upgrade. Avoids "black box" neural search where it isn't needed.
     • Resilient Ingest: Handles corrupt or junk files gracefully, resumes from interruptions, and provides a self-describing data catalog.
   • Cons:
     • Built-in Semantic Limits: The default feature-hashing embedder provides vocabulary-overlap similarity, not deep neural understanding. For complex semantic tasks, the neural encoder must be explicitly enabled or an external proxy used.
     • Accuracy Trade-off: In a controlled test, its accuracy tied a grep-and-Python baseline (9/10 vs. 10/10), winning on structural efficiency, not raw precision.
     • New Ecosystem: As a newer tool, it lacks the vast plugin ecosystem and managed cloud services of established players like Elastic.

2. Key Alternatives & Differentiation:

   • Elasticsearch / OpenSearch: XERJ differentiates by being AI-agent-first, with built-in code AST parsing, a zero-config ingest CLI (autoindex), and a dedicated agent memory API. It is a fraction of the size (single binary vs. JVM-based stack) and starts instantly. It is a migration-compatible bridge for agent use cases, not a general-purpose clone.
   • grep, ripgrep, sed: These are file search tools. XERJ is an answer engine. It returns precise passages (function definitions) from a pre-built index, saving the agent the subsequent step of reading the found file into context. It solves the token cost problem that line-oriented tools do not address.
   • Vector Databases (e.g., Pinecone, Weaviate): XERJ integrates vector search as one component of a unified system that also includes superior lexical search (BM25), code AST search, and a document pipeline. It avoids the complexity of running and fusing results from separate lexical and vector databases.

Frequently Asked Questions (FAQ)

1. How does XERJ save money on AI agent usage? XERJ reduces the number of expensive output tokens an AI agent consumes by providing precise answers from an index, preventing the agent from needing to read large files or entire search results into its context window. Benchmark data shows a 2.7x reduction in output tokens for coding tasks.
2. Can I use my existing Elasticsearch clients and dashboards with XERJ? Yes. XERJ is wire-compatible with the Elasticsearch API (1,366/1,369 conformance tests pass), meaning existing clients, libraries (e.g., elasticsearch-py), and dashboards like Kibana can connect to it with little to no modification.
3. What types of files and data can XERJ index? The xerj autoindex command automatically sniffs and indexes 13 format families, including source code (via Tree-sitter), JSON, JSONL, CSV, log files, PDF, DOCX, HTML, XML, YAML, SQL dumps, SQLite files, and gzip archives.
4. Is XERJ suitable for large-scale, production document search? Its architecture supports scalable, resumable ingestion and efficient hybrid search. However, for massive, petabyte-scale general search workloads, established platforms like Elasticsearch may offer more mature clustering and management tools. XERJ's sweet spot is AI-augmented workflows and targeted retrieval.
5. How does the built-in agent memory work? The /_memory API provides a simple REST interface for an agent to POST text with metadata to a namespaced index (e.g., agent-demo) and later _recall it using semantic (vector), keyword, or filtered searches. Data is physically isolated per namespace, ensuring agents cannot access each other's memories.