IQ Routing

Product Introduction

1. Definition: IQ Routing is a vendor-agnostic AI gateway and intelligent routing layer. Technically, it is a drop-in proxy that sits between an application and multiple Large Language Model (LLM) providers like OpenAI, Anthropic, and Google.
2. Core Value Proposition: It exists to drastically reduce AI infrastructure costs by dynamically routing each LLM request to the cheapest model capable of maintaining a defined quality threshold. Its primary value is automated cost optimization for AI applications without sacrificing performance or output quality.

Main Features

1. Intelligent Per-Step Routing: The core technology is a purpose-built classifier that analyzes the semantic complexity and intent of each individual request within an agentic workflow. It evaluates live cost and latency data across integrated model providers (e.g., GPT-4o, Claude Haiku, Gemini) and selects the most cost-effective model that meets a pre-configured quality bar for that specific task, such as retrieval vs. synthesis.
2. Semantic Caching: This feature intercepts and deduplicates requests before they are sent to a paid model. It uses semantic matching to identify not just identical queries but also paraphrased or semantically equivalent prompts, serving cached responses instantly (~11ms latency) and charging $0.00 for the repeat call. Caches are isolated per team or API key.
3. Session-Aware Agent Governance: Unlike simple per-call routers, IQ Routing treats an entire agent loop (a trajectory) as a single accountable session. It tracks cumulative cost, token usage, and step classes across the session and can enforce spending caps (e.g., on "thinking" tokens) to prevent runaway costs in complex, multi-turn autonomous agent operations.
4. Unified Billing & Audit Ledger: It provides granular, four-way cost attribution (input, output, cached, thinking tokens) written to an immutable ledger. This enables per-team budgeting, per-API-key rate limiting, and detailed, auditable spend reports that break down the traditionally opaque "one-line" LLM invoice by model, team, and task.

Problems Solved

1. Pain Point: Uncontrolled and opaque LLM spending. Companies default to using a single, powerful (and expensive) frontier model for all tasks—from simple classification to complex reasoning—leading to massive, un-itemized bills. Finance teams struggle to categorize costs or attribute spend to specific teams or projects.
2. Target Audience: Engineering leaders and DevOps teams managing production AI workloads; FinOps professionals responsible for cloud and AI spend management; Product teams building chatbots, RAG pipelines, or autonomous agent applications.
3. Use Cases: Essential for cost-effective scaling of: Production chatbot backends, Multi-step Retrieval-Augmented Generation (RAG) pipelines, Long-running autonomous agent loops (e.g., LangChain, LlamaIndex), and any finance or analytics workload with high, repetitive LLM call volume.

Unique Advantages

1. Differentiation: Compared to basic API gateways or load balancers, IQ Routing's differentiation is its context-aware routing. It doesn't just round-robin or failover; it makes a quality-preserving economic decision for each unique step. Versus manual model selection, it automates optimization in real-time based on live pricing and prompt intent.
2. Key Innovation: The insight that "an agent is a trajectory, not a stream of independent calls." Its session envelope and per-step classifier represent a novel architectural approach that understands the structure of AI workflows, allowing for granular optimization at the step level while maintaining conversational state and coherence across different models within a single session.

Frequently Asked Questions (FAQ)

1. How does IQ Routing save money on LLM API calls? IQ Routing saves money by implementing a semantic cache to eliminate charges for duplicate queries and, more importantly, by using an intelligent classifier to route each request to the least expensive LLM (like GPT-4o-mini or Claude Haiku) that is capable of handling the specific task's complexity, avoiding unnecessary use of premium models like Claude Opus.
2. Is IQ Routing compatible with existing AI applications built with OpenAI's SDK? Yes, IQ Routing offers drop-in compatibility. You can typically replace your base OpenAI or Anthropic API endpoint URL with IQ Routing's single gateway URL without changing your application code, making integration possible in under thirty seconds for SDK-based projects.
3. How does the per-step routing work for AI agent loops? For agent loops (e.g., planning, retrieval, tool call, synthesis), IQ Routing's classifier evaluates each step independently. A simple step like "format arguments" may route to a cheap, fast model, while a critical "reasoning-heavy" synthesis step routes to a frontier model. This step-level optimization across a single session is where the majority of cost savings (40-80%) are achieved.
4. What is the performance latency impact of using an AI gateway like IQ Routing? IQ Routing adds minimal latency. Cache-hit responses are served in approximately 11 milliseconds. For non-cached requests, the intelligent routing logic executes quickly, and the primary latency is determined by the downstream LLM provider it selects, often resulting in faster responses when it correctly routes simple tasks to faster models.
5. Can I set budgets and control costs for different teams with IQ Routing? Yes, a core feature of IQ Routing is its granular financial controls. You can set hard per-team or per-project budgets and per-API-key rate limits. All spending is split into input, output, cached, and thinking tokens in a detailed audit ledger, providing complete cost visibility and preventing budget overruns.