oqoqo

Product Introduction

1. Definition: Oqoqo is a specialized platform for building, running, and analyzing large-scale evaluation (eval) experiments for AI agents and language models (LLMs). It falls under the technical categories of AI/ML evaluation, agent benchmarking, and developer tooling.
2. Core Value Proposition: It exists to provide developers and product teams with a managed, scalable infrastructure to objectively measure how effectively AI agents can interact with and use real-world products, APIs, and tools. Its core value is enabling data-driven decisions on agent performance, model selection, and product interface design through reproducible, private benchmarks.

Main Features

1. Managed Experiment Infrastructure: Oqoqo provides fully managed cloud infrastructure to run eval experiments at scale. How it works: Users define experiments, and Oqoqo orchestrates the execution of each task run in isolated, ephemeral sandbox environments. This includes durable workflow management, parallel execution, and separate timing for setup, execution, and teardown phases to ensure realistic and reproducible conditions.
2. Custom Task & Rubric Definition: The platform allows users to build private benchmarks by defining custom task sets and evaluation rubrics in plain language. How it works: Users create tasks (e.g., "Create a PaymentIntent for $15.00 USD") and attach specific, requirement-based rubrics (e.g., "Creates the product," "Writes both ids"). These tasks and rubrics are versioned with each experiment, ensuring comparability across different runs and iterations over time.
3. Full Trajectory Capture & Friction Analysis: Beyond a simple pass/fail metric, Oqoqo captures the complete step-by-step trajectory of an agent's attempt. How it works: It records all tool calls (e.g., CLI commands, API requests), bash commands, errors, and the agent's final stopping point. This data is analyzed to generate dynamic insights, such as identifying specific frictions in a product's interface, token inefficiencies, or common failure patterns in the agent's reasoning.
4. Multi-Agent & Multi-Treatment Comparison: The platform is designed for comparative analysis. How it works: An experiment configuration crosses multiple variables: different AI agents (e.g., Claude Code, Codex), different "treatments" (e.g., using a raw agent vs. an MCP server), and repeated trials of the same tasks. This A/B testing framework allows users to measure the lift in pass rates and efficiency from one treatment to another under controlled conditions.
5. Multi-Interface Access (Web, CLI, MCP): Oqoqo can be operated through a web interface for human users and is also accessible to AI agents themselves via a Command Line Interface (CLI) and Model Context Protocol (MCP) server. This "agent-first" design allows other AI agents to programmatically launch and inspect experiments, creating a recursive evaluation loop.

Problems Solved

1. Pain Point: The inability to systematically and reliably test if AI agents can successfully use a software product, API, or developer tool. Manually testing agents is anecdotal, not scalable, and lacks reproducibility.
2. Target Audience: Developer tool companies (e.g., Stripe, Twilio), AI engineering teams, MCP server developers, product managers for API-first products, and researchers focused on agent capabilities and benchmarking.
3. Use Cases:
   • Building Private Benchmarks: Turning internal support tickets, documentation examples, or common customer workflows into a standardized, private test suite for agents.
   • Evaluating MCP Servers & Skills: Quantifying the performance improvement an MCP server or custom skill provides to an agent when interacting with a specific product.
   • CI/CD for Agent Interfaces: Triggering automated eval experiments when a product's API, CLI, or SDK changes to prevent breaking agent-based workflows.
   • Model Selection: Determining which LLM or AI agent (e.g., Claude Code vs. GitHub Copilot) performs best on a specific set of practical, domain-specific tasks.
   • Identifying Product Frictions: Analyzing failure trajectories to find confusing API parameters, missing documentation, or inefficient workflows that hinder agent usability.

Unique Advantages

1. Differentiation: Unlike generic LLM evaluation platforms focused on question-answering, Oqoqo is specifically engineered for evaluating agentic interaction with tools and environments. It provides the isolated sandboxes, real tooling, and trajectory-level analysis that traditional benchmarks lack. Compared to manual testing, it offers scale, reproducibility, and quantitative metrics.
2. Key Innovation: The integration of isolated, tool-ready sandboxes within a comparative experiment framework. The platform doesn't just simulate an environment; it provisions real, ephemeral machines (e.g., a Stripe CLI sandbox) with project state and context, allowing agents to execute actual commands and API calls. Coupling this with a flexible A/B testing engine for agents and treatments is its core technical innovation.

Frequently Asked Questions (FAQ)

1. What is Oqoqo used for? Oqoqo is used for building evals and private benchmarks to test how well AI agents can use real-world products like APIs, CLIs, and SDKs. It helps teams compare agent performance, find failure points, and optimize their tools for agent users.
2. How does Oqoqo compare to manual agent testing? Manual testing is single-run and anecdotal. Oqoqo automates reproducible experiments at scale, running the same tasks across multiple agents and treatments to generate statistically significant metrics like pass rate lift, token usage, and identified frictions.
3. Can I use Oqoqo to test my own product's API? Yes, you can evaluate any product that an agent can interact with, including custom APIs, CLIs, SDKs, MCP servers, and documentation. You define tasks that represent real use cases for your product.
4. Who is responsible for the cost of AI model usage on Oqoqo? Oqoqo uses a bring-your-own-keys (BYOK) model. Users provide their own API keys or subscriptions for the AI models (e.g., OpenAI, Anthropic), while Oqoqo's platform fee covers the managed cloud infrastructure and experiment orchestration.
5. What do I see after an experiment runs? You see a pass/fail result for each task requirement, the full step-by-step trajectory (commands, tool calls, errors), and aggregate experiment metrics including pass rates, token consumption, duration, and a count of identified frictions.