Inferock Bench

Product Introduction

1. Definition: Inferock Bench is a local, diagnostic proxy server designed for cost-tracking and billing-integrity verification of Large Language Model (LLM) API calls. It operates as a middleware layer between an application and major AI provider APIs, including OpenAI, Anthropic, Gemini, and pinned OpenRouter endpoints.
2. Core Value Proposition: It exists to provide independent, per-call auditing of AI API consumption, addressing the critical gap where providers act as both the service vendor and the sole auditor of their own billing. Its primary function is to generate verifiable receipts that cross-check token usage, failures, and retries against the provider's invoice, identifying potential overcharges and delivery failures.

Main Features

1. Independent Receipt Generation: The proxy captures every API call's metadata, including request/response payloads, status codes, timing, and the provider's reported token usage. It then processes this data against "The Inferock Standard" specification to generate a local receipt. This receipt separates observed provider spend, bill-bounded monetary loss (e.g., charges for failed calls), time loss, and invoice-check exposure (like cache discount risks).
2. Multi-Provider Failure Detection: The system employs detectors to flag specific failure modes that can lead to incorrect billing. This includes identifying billed-empty outputs (charged for no content), refusals, truncations, token-recount mismatches, duplicate request IDs, and evidence of provider-fault retries that may not be credited.
3. Local-First & Key-Secure Architecture: All processing occurs on the user's local machine. Provider API keys are stored locally with strict file permissions and are only attached to outbound requests to the respective AI providers. The Inferock servers never receive these keys. The local proxy generates a separate, local-only bench key (ibl_*) for applications to use, maintaining security and control.
4. Measured Provider Planes & Coverage States: It provides calibrated measurement for specific API surfaces (or "planes") from OpenAI, Anthropic, Gemini, and select OpenRouter models. Each surface is assigned a clear coverage state—watched-clean, signal (monitored with detectors), or not-openable—providing transparency about what is and isn't being actively audited.

Problems Solved

1. Pain Point: The lack of independent verification for AI API billing, often described as "the provider grading their own homework." Users cannot easily verify if they were charged for failed calls, network errors, or incomplete responses.
2. Target Audience: DevOps engineers and platform teams managing substantial LLM budgets; FinOps professionals responsible for cloud and AI cost optimization; developers and companies building LLM-powered applications who need to debug cost spikes and performance issues; auditors and technical leads requiring evidence for billing disputes.
3. Use Cases: Auditing a monthly OpenAI or Anthropic invoice for discrepancies; debugging unexpected cost spikes in an LLM application; measuring the real cost impact of model failures and retries in a production system; providing tangible evidence for billing support requests to an AI provider; establishing a baseline for cost-per-call in development and staging environments.

Unique Advantages

1. Differentiation: Unlike generic API monitoring or logging tools, Inferock Bench is purpose-built for LLM billing integrity with a formal specification (The Inferock Standard). Unlike provider dashboards, it offers an independent, adversarial perspective. Compared to simple cost calculators, it works on real traffic and correlates cost with delivery success/failure.
2. Key Innovation: Its "receipt contract" philosophy clearly separates observations (raw event data) from interpretations (dollar figures calculated using published thresholds and assumptions). This transparency prevents conflating evidence with claims. The system is designed to move more factors from "interpretation" to "observation" over time, increasing auditability.

Frequently Asked Questions (FAQ)

1. Is Inferock Bench a replacement for my AI provider's dashboard? No, it is a complementary audit tool. It uses your provider's API and reported data but adds an independent layer of analysis, failure detection, and evidence preservation specifically for billing integrity, which provider dashboards typically do not offer.
2. Does using Inferock Bench slow down my API calls? As a local proxy, it adds minimal latency (typically sub-millisecond) for request/response routing and logging. The primary latency is still determined by the network call to the external AI provider (e.g., OpenAI, Anthropic). The tool is designed for diagnostic and auditing purposes, not for ultra-low-latency production routing.
3. Can Inferock Bench audit traffic that doesn't go through it? No. It can only measure and generate receipts for API calls that are routed through its local proxy endpoint. It cannot retroactively analyze logs or audit traffic that bypassed it, which is why it's implemented as a proxy to be integrated into the application's request flow.
4. How does Inferock Bench handle my API keys securely? Provider keys are stored locally in a configuration file (~/.inferock-bench/) with strict owner-only file permissions. They are never transmitted to Inferock's servers. The keys are only used locally to authenticate outbound requests from the proxy to the respective AI provider APIs.
5. What does "invoice-check exposure" mean on the receipt? This refers to potential billing discrepancies that require manual verification against your official provider invoice, not automatic monetary loss. A key example is CACHE_DISCOUNT_AT_RISK, which highlights calls where context caching was likely used but the corresponding discount may not have been applied correctly on the invoice. It is presented separately from "money loss" to avoid conflating evidence with confirmed overcharges.