RunInfra

Product Introduction

1. Definition: RunInfra is a hosted Model API platform that provides OpenAI-compatible and Anthropic-compatible endpoints for accessing high-performance open-weight large language models (LLMs). It is a technical infrastructure service designed specifically for AI agents and applications.
2. Core Value Proposition: RunInfra exists to provide developers and businesses with a cost-effective, high-speed, and privacy-focused gateway to leading open-source models. Its primary value is reducing the effective cost of AI inference through intelligent, session-aware caching that achieves cache hit rates above 99% for agent workloads, directly translating to lower operational expenses.

Main Features

1. Session-Aware Routing & High-Efficiency Caching: This is RunInfra's core technical innovation. The system intelligently routes all subsequent calls from an AI agent's session to the same backend replica that holds its cached context. This dramatically increases cache hit probability compared to random or round-robin routing. How it works: When an agent initiates a session, a session ID (derived from the conversation or workspace) is used to pin requests to a specific replica. This ensures that repeated prompts within a session are served from the in-memory cache on that replica, billed at a much lower "cached input" token rate.
2. Unified API Compatibility: RunInfra provides a single API endpoint (https://api.runinfra.ai/v1) that is compatible with both the OpenAI SDK and the Anthropic Messages API SDK. This means developers can switch their existing applications from OpenAI's GPT models or Anthropic's Claude models to RunInfra's open models by simply changing the base URL and API key, with minimal code changes.
3. Coding Agent Integration & Flat-Rate Plans: Beyond pay-as-you-go API access, RunInfra offers "Coding Plans" starting at $10/month. These are designed for individual developers using AI coding assistants. Setup is simplified to a one-prompt process for agents like Claude Code, Cline, and OpenCode, where the agent itself runs a CLI tool to configure the integration. The flat monthly fee provides access to a bundle of model calls, with the high cache hit rate extending the utility of the plan.

Problems Solved

1. Pain Point: The high and unpredictable cost of running AI agents, which often involve long, repetitive context windows (like codebases) sent with each inference request. Traditional model providers charge the full price for every input token, making iterative agent workflows prohibitively expensive.
2. Target Audience: The primary users are Developers building AI-powered applications (especially multi-step agents), DevOps/ML Engineers managing inference infrastructure, and Individual Developers/Engineers using AI coding assistants (Claude Code, Cline, etc.) who seek a cheaper, private alternative to closed-source models.
3. Use Cases: Essential for cost-sensitive deployment of coding assistants that re-analyze project files, long-running conversational agents that maintain context, applications performing repeated analysis on static documents, and any production workload where prompt templates or context is reused across multiple user sessions.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros:
     • Substantial Cost Reduction: Effective input token costs can be 3.5x to 13.6x lower than list prices due to high cache hit rates (97-99.5%).
     • Performance: Advertises high output token speeds (e.g., 254 tok/s for GLM 5.3 Flash, 540 tok/s for Nemotron 3.5) measured at the model level.
     • Developer Experience: Seamless integration with popular SDKs and coding agents lowers the barrier to entry.
     • Strong Privacy Stance: Explicit zero data retention policy, no training on user data, and SOC 2 Type II compliance.
   • Cons/Limitations:
     • Cache Dependency: The major cost benefit is heavily dependent on achieving high cache hits, which may vary based on user workload patterns.
     • Model Selection: Focuses on a curated list of ~7 open models (DeepSeek, GLM, Qwen, etc.), not the vast universe of all open-source models or market leaders like GPT-4o or Claude 3.5.
     • Best-Effort Cache: Cache hits are not guaranteed, as stated, which introduces a variable element in cost forecasting.

2. Key Alternatives & Differentiation:

   • OpenAI / Anthropic API: The direct alternative. RunInfra differentiates on price (via caching), privacy (no training, zero retention), and model choice (open vs. closed). Trade-off: Potential differences in model capability, latency, and reliability.
   • Other Model Hosting Platforms (Together AI, Fireworks AI): These also offer open model APIs. RunInfra's key differentiator is its agent-optimized, session-aware caching architecture as a first-class feature, whereas others may offer general-purpose caching. Its pricing narrative is tightly coupled to agent-specific cache performance metrics.
   • Self-Hosting Open Models: RunInfra eliminates the DevOps complexity, hardware cost, and engineering effort required to deploy, scale, and optimize model inference and implement efficient caching layers.

Frequently Asked Questions (FAQ)

1. How does RunInfra's caching work and how does it save money? RunInfra uses session-aware routing to direct an AI agent's repeated calls to the same server replica where its context is cached. This means repeated input tokens (like a project's codebase) are served from memory instead of being re-processed by the model, and you are billed at a significantly lower "cached input" rate (e.g., $0.01/M tokens vs. $0.15/M tokens).
2. Is RunInfra compatible with my existing code using the OpenAI Python library? Yes, RunInfra provides an OpenAI-compatible API endpoint. You can typically switch by changing the base_url parameter in your OpenAI client initialization to https://api.runinfra.ai/v1 and using your RunInfra API key, with no other code changes required for basic chat completions.
3. What is the difference between RunInfra's pay-as-you-go API and the $10/month Coding Plan? The pay-as-you-go API is for general application development, billed per token used. The $10/month Coding Plan is a simplified, flat-rate subscription designed specifically for individual developers using AI coding assistants (like Claude Code), providing a bundle of usage that is consumed through the agent integration.
4. Does RunInfra store or use my prompt data for training? No. RunInfra explicitly states a zero data retention policy by default and does not use customer prompts or completions to train their or any third-party models, which is a key privacy differentiation from some major providers.
5. Which models does RunInfra support and how do they compare to GPT-4? RunInfra focuses on high-performance open models like DeepSeek V4.1 Flash, GLM 5.3 Flash, and Qwen3.8 27B. These models are competitive for many coding and reasoning tasks but may have different strengths/weaknesses compared to GPT-4 in areas like very complex reasoning, long-context fidelity, or specific modalities.