AgentScore

Product Introduction

1. Definition: AgentScore by Latitude is a production-grade AI agent performance analytics and scoring platform. It falls into the technical categories of AI observability, LLM (Large Language Model) evaluation, and performance monitoring.
2. Core Value Proposition: It exists to provide AI developers and product teams with a single, evidence-based quality score for their production AI agents, moving beyond isolated metrics to a holistic understanding of performance across five critical dimensions: outcome, reliability, cost, speed, and safety. Its primary value is in transforming raw production telemetry into actionable insights for continuous agent improvement.

Main Features

1. Five-Dimensional Agent Score: The core feature is a composite score derived from real production sessions. It evaluates:

   • Outcome: Measures task completion success, moving beyond plausible text generation to assess if the user's intent was fulfilled.
   • Reliability: Tracks operational failures like unhandled errors, failed tool/function calls, and terminal session crashes.
   • Cost: Analyzes token and tool usage efficiency, identifying redundant calls, repeated context, and other forms of computational waste.
   • Speed: Measures latency and identifies avoidable delays, such as slow sequential calls that could be parallelized or unnecessary retries.
   • Safety: Flags confirmed harmful actions, destructive behaviors, and policy violations observed in real user interactions.
   • How it works: The platform ingests production traces (via OpenTelemetry or direct import), applies a gated methodology requiring minimum traffic (1,000+ sessions), coverage, and statistical confidence, and outputs a score with a 95% confidence interval for a rolling 7-28 day window.

2. Evidence-Based Root Cause Analysis: The platform directly links score fluctuations to the underlying production sessions. Users can drill down from a dimensional dip (e.g., low "Reliability") to inspect the specific sessions containing errors, enabling precise debugging.

   • How it works: It aggregates failures by pattern, allows deep inspection of session transcripts and trace data, and facilitates turning recurring failures into targeted evaluations for testing fixes.

3. Production-to-Improvement Workflow Integration: AgentScore is designed to close the loop between observation and action. It supports turning identified failures into tracked evaluations and integrates with coding agents via the Model Context Protocol (MCP) to provide evidence directly into the development cycle.

   • How it works: After identifying a failure pattern, teams can create targeted evals, implement fixes (e.g., prompt engineering, code changes), and use the platform to verify the impact of the new release on the Agent Score.

Problems Solved

1. Pain Point: The "black box" problem in LLM and AI agent development. Teams lack a standardized, holistic, and trustworthy way to measure the true quality and performance of agents in live production environments, leading to guesswork and reactive firefighting.
2. Target Audience: AI Engineers, ML Engineers, and Developer teams building and maintaining production AI agents and LLM applications; Product Managers and Technical Leaders responsible for AI product quality and ROI.
3. Use Cases:
   • Continuous Performance Monitoring: Daily tracking of an AI customer support agent's effectiveness and cost-efficiency.
   • Post-Deployment Regression Detection: Immediately identifying if a new model version or prompt change degraded reliability or safety scores.
   • Cost Optimization: Pinpointing specific agent workflows that generate excessive token usage or redundant API calls.
   • Prioritization of Development Efforts: Using the multi-dimensional score to objectively decide whether to focus engineering resources on improving speed, outcome success rate, or error handling.

Unique Advantages

1. Differentiation: Unlike basic logging dashboards or synthetic testing tools, AgentScore is grounded exclusively in real user production evidence. It avoids the "good at one thing" pitfall by evaluating five interdependent dimensions simultaneously, ensuring a balanced view of agent health that mirrors real-user experience.
2. Key Innovation: Its gated, evidence-first methodology. The system refuses to show a score until statistically significant evidence is collected (1,000+ eligible sessions across all dimensions with confidence intervals). This prevents misleading metrics based on low traffic and builds trust in the score's validity. The transparent, open methodology is a significant innovation in the often-opaque field of AI evaluation.

Frequently Asked Questions (FAQ)

1. What is AgentScore and how is it calculated? AgentScore is a holistic performance metric for AI agents, calculated daily from production session data across five dimensions: Outcome, Reliability, Cost, Speed, and Safety. It uses a minimum of 1,000 sessions, applies statistical confidence gates, and reports a score with a 95% confidence interval over a rolling 7-28 day window.
2. How does Latitude AgentScore differ from traditional application performance monitoring (APM)? While APM tools track system metrics like latency and errors, AgentScore is purpose-built for AI agent behavior. It evaluates higher-order concepts like task outcome success and safety, and specifically analyzes LLM-specific inefficiencies like token waste and avoidable sequential calls, which standard APM cannot.
3. What data do I need to connect to get an Agent Score? You need to send detailed traces of your AI agent's production sessions. This is typically done by integrating the OpenTelemetry SDK (OpenTelemetry) or by importing existing trace data from compatible sources. The traces must contain the full interaction chain, including LLM calls, tool executions, and user inputs.
4. Can I use AgentScore if my agent is still in development or has low traffic? The platform will process your traces, but a score will only appear once your agent meets the eligibility criteria: a minimum of 1,000 eligible production sessions with sufficient coverage and confidence across all five dimensions. This ensures the score is meaningful and actionable.
5. How does AgentScore help me actually improve my AI agent? It directly links performance issues to the exact production sessions where they occurred. You can drill down from a low dimension score to see the failing sessions, identify patterns, create targeted evaluations to test fixes, and monitor how your next deployment affects the score, creating a data-driven improvement loop.