Switchyard

Product Introduction

1. Definition: Switchyard is a Rust-based proxy and library for LLM (Large Language Model) traffic, designed for multi-agent conversational AI systems. It functions as a protocol translation and intelligent routing layer between AI clients and various model backends.
2. Core Value Proposition: It exists to solve the complexity of orchestrating multi-agent AI applications by providing a unified framework for routing requests, translating between incompatible LLM APIs, and enabling rigorous evaluation and benchmarking of different models and routing strategies.

Main Features

1. Protocol Translation: Switchyard performs real-time translation between major LLM provider APIs. It can accept a request in OpenAI Chat Completions format, route it to an Anthropic Claude endpoint using the Anthropic Messages API, and translate the response back to the OpenAI format for the client. This works bidirectionally across OpenAI, Anthropic, and OpenAI-compatible endpoints (like vLLM, NVIDIA NIM, Ollama).
2. Multi-Backend Routing Algorithms: The framework provides several typed, composable routing algorithms. The LLM Classifier uses a separate model call to analyze request content and route between "weak" (cost-effective) and "strong" (powerful) model tiers. The Stage Router uses signals already present in the conversation (like tool call results or errors) to make routing decisions without an extra LLM call. The Escalation Router sends every request to a weak model first, then uses a "judge" model to decide if a re-run on a strong model is needed. Random routing enables fixed traffic splits for A/B testing.
3. Operational Metrics & Observability: Switchyard emits Prometheus metrics covering key operational data. This includes request counts, error rates, latency percentiles (P50, P90, P99), token usage (input, output, total), and the overhead introduced by the routing logic itself, enabling performance monitoring and cost analysis.

Problems Solved

1. Pain Point: Developers and researchers face significant friction when building multi-agent systems that require different specialized AI models from various providers, due to incompatible APIs and the complexity of managing routing logic, fallbacks, and evaluation.
2. Target Audience: The primary users are AI Researchers evaluating model performance, ML Engineers building production-grade conversational AI (advanced chatbots, virtual assistants), and Developers creating interactive AI platforms that require orchestration between multiple agents.
3. Use Cases: Essential scenarios include: enabling a coding agent like Claude Code to use open-source models via API translation; conducting rigorous A/B benchmarking of different LLMs (e.g., GPT-4 vs. Claude 3) for a specific task; building a cost-optimized customer support agent that uses a small model for simple queries and escalates only complex issues to a larger, more expensive model.

Unique Advantages

1. Differentiation: Unlike simple API wrappers or gateways, Switchyard is not just a pass-through. Its core innovation is providing programmable routing intelligence. Compared to manual orchestration code, it offers pre-built, optimized algorithms and a clean abstraction layer, significantly reducing development time and complexity for multi-agent systems.
2. Key Innovation: The separation of the routing logic library (switchyard-libsy) from the HTTP server. This allows developers to embed the sophisticated routing algorithms directly into their existing Rust-based proxy, gateway, or agent runtime without being forced into a specific server architecture, offering unparalleled integration flexibility.

Frequently Asked Questions (FAQ)

1. What is Switchyard used for? Switchyard is used to build, evaluate, and operate multi-agent conversational AI systems by intelligently routing requests between different large language models, translating between their APIs, and providing metrics for performance analysis.
2. How does Switchyard routing work? Switchyard uses configurable algorithms like LLM Classifiers and Stage Routers to decide which backend model (e.g., a fast/cheap model vs. a powerful/expensive one) should process a given user request, based on content analysis or conversation signals, to optimize cost and performance.
3. Is Switchyard ready for production use? No, Switchyard is currently pre-alpha software labeled as experimental. NVIDIA explicitly states it is "Not for production use," as the API and algorithms are expected to change significantly before a stable v1.0 release.
4. Can Switchyard work with locally hosted models? Yes, Switchyard can route requests to any OpenAI-compatible endpoint, which includes popular local inference servers like vLLM, Ollama, and NVIDIA NIM, allowing you to use open-source models within its orchestration framework.
5. What is the difference between switchyard-server and switchyard-libsy? switchyard-server is a standalone HTTP proxy you can run. switchyard-libsy is a Rust library that provides the core routing algorithms, allowing you to embed Switchyard's intelligence directly into your own application's codebase without using the provided server.