Respan Gateway

Product Introduction

1. Definition: Respan Gateway is an AI Gateway platform and unified LLM routing layer that sits between an application and a multitude of large language model providers. It functions as a single API endpoint for managing, securing, and observing all interactions with models like GPT-4, Claude, Gemini, and 500+ others, providing both router (multi-model failover) and passthrough (direct provider access) modes.
2. Core Value Proposition: Respan exists to make production AI reliable, observable, and cost-effective by replacing fragmented toolchains. Its core proposition is delivering unified LLM routing, built-in observability, spend controls, and prompt management in one platform, eliminating the need to stitch together separate tools for routing, monitoring, and evaluation.

Main Features

1. Unified Routing & Intelligent Failover: This feature provides a single OpenAI-compatible API endpoint (https://api.respan.ai/api/) to access 500+ models. It works by accepting standard chat completion requests and intelligently routing them to the specified provider or model. If the primary model encounters an error or rate limit, the gateway automatically retries the request with the next model in a pre-configured fallback list (fallback_models), implementing backoff strategies to prevent cascading failures. This ensures high availability for AI-powered applications.
2. Granular Spend Controls & Alerting: Respan implements per-API-key cost management and alerting. Administrators can set soft warnings and hard caps on token usage or spending. The system tracks usage against these limits in real-time and triggers alerts via Slack or email when thresholds are crossed. This provides financial governance and prevents unexpected bill shocks from runaway LLM usage.
3. Full Observability and Tracing: Every call through the gateway, whether routed or passthrough, is automatically logged as a detailed trace tree. Each trace captures latency per span, the model used, and custom metadata like customer_identifier or thread_identifier. This enables developers to filter logs by feature, tenant, or conversation thread, providing deep LLM observability for debugging and performance analysis without additional instrumentation.
4. Response Caching and Optimization: The platform includes built-in LLM response caching to reduce latency and cost. It allows caching of repeat prompts with configurable TTL (cache_ttl). Advanced options include cache_by_customer to prevent cache collisions between different users and is_cached_by_model to ensure cache entries are model-specific, preventing stale responses when switching between different LLMs.

Problems Solved

1. Pain Point: Unreliable and Unobservable AI Production Systems. Teams calling LLM providers directly face provider key sprawl, lack automatic failover on the hot path, uncoordinated retries that can cascade, and fragmented logging that makes debugging nearly impossible. Each provider's dashboard is a silo.
2. Target Audience: This product is essential for AI/ML Engineers, Backend Developers, DevOps/SRE Teams, and Platform Engineers responsible for deploying and maintaining production AI features. It also serves CTOs and Engineering Managers who need to control costs and ensure reliability without complex tool integration.
3. Use Cases: Critical for any customer-facing AI application (chatbots, agents, copilots) where downtime is costly; for AI agent frameworks requiring multi-model resilience; for enterprise teams needing to attribute LLM costs to specific customers, features, or projects; and for any organization aiming to implement robust LLM evaluations and prompt management in a centralized system.

Unique Advantages

1. Differentiation: Unlike simple API proxies or single-model routers, Respan combines routing, observability, evaluations, prompt management, and cost controls in one integrated platform. Competitors may offer point solutions for logging or routing, but Respan eliminates the toolchain complexity of using separate services like a logging aggregator, a cost tracker, and a routing library.
2. Key Innovation: The platform's key innovation is its unified trace and metadata model. By automatically tagging every request—whether it's a routed call, a passthrough call, or a cache hit—with rich context like customer_identifier and custom metadata, it transforms raw API logs into actionable, filterable observability data. This provides a single pane of glass for understanding performance, cost, and behavior across all AI interactions in a product.

Frequently Asked Questions (FAQ)

1. How does Respan Gateway handle LLM provider outages and rate limits? Respan automatically implements failover and retries. You can configure a list of fallback models per request or in the dashboard settings. If the primary model returns an error or a 429 rate-limit status, the gateway will automatically retry the request with the next model in the list, using configurable backoff parameters to protect upstream services.
2. What is the difference between the "Unified Router" and "Provider Passthrough" modes? The Unified Router uses Respan's single, OpenAI-compatible endpoint to access multiple models with added features like failover and caching. Provider Passthrough allows you to use native SDK endpoints (e.g., Anthropic's API) while still routing the request through Respan, ensuring the call is logged and tagged with metadata for observability, but without Respan's routing or caching layer.
3. Can I track LLM costs for specific customers or features using Respan? Yes. By sending a customer_identifier and custom metadata (e.g., feature: "chatbot", environment: "production") on each request, you can filter your Logs and Traces in the Respan dashboard. This allows you to monitor usage, latency, and costs broken down by tenant, feature flag, or A/B test cohort.
4. How does the caching prevent one user's data from being served to another? Respan's caching system has an option called cache_by_customer. When enabled, the cache key incorporates the customer_identifier, creating isolated cache namespaces for each user. Additionally, you can use cache_options.is_cached_by_model to ensure a cache entry generated for GPT-4 is not served for a subsequent request to Claude.
5. What compliance standards does Respan adhere to for handling sensitive data? Respan is built for enterprise and compliance-sensitive workloads. It holds SOC 2 and ISO 27001 certifications, operates under GDPR principles, and offers a HIPAA Business Associate Agreement (BAA) for healthcare organizations, ensuring data privacy and security controls are in place.