Pheebs

Product Introduction

1. Definition: Pheebs is an open-source AI telemetry and observability tool designed for software development teams. Technically, it operates as a lightweight agent that integrates via hooks into popular AI-powered coding environments (Claude Code, Cursor, and Codex) to capture and analyze developer-AI interaction data.
2. Core Value Proposition: Pheebs exists to provide data-driven visibility into how engineering teams actually utilize AI coding assistants, moving beyond anecdotal evidence. Its primary value is quantifying AI proficiency, identifying wasteful model spend, and measuring the adoption of effective AI workflows to improve team efficiency and reduce unnecessary AI API costs.

Main Features

1. Privacy-First Event Capture: Pheebs uses hook-based instrumentation to capture granular interaction signals without storing sensitive data. It records event types (e.g., session_started, tool_use_completed, prompt_submitted), metadata (durations, model names, intent labels), and counts, but explicitly excludes source code, file paths, and prompt text from persistent storage. This is implemented via custom hooks for each supported IDE/agent.
2. AI Proficiency Model & Analytics: The tool translates raw telemetry into a structured proficiency model across six competencies: Models, Artifacts, MCP, Evals, Context Management, and Orchestration. It generates actionable metrics like Verification Coverage, Pushback Rate, Refinement-to-Repair Ratio, Wholesale-Accept Rate, and Model-Fit Rate, providing a quantifiable baseline for team and individual performance.
3. Flexible Deployment & Data Pipeline: Pheebs supports a dual-pipeline architecture. All data is first written to a local JSONL log file (~/.pheebs/logs). With a configured token, events can be forwarded to a backend via a simple HTTP contract (POST /ingest). This allows for fully self-hosted deployment (where you control all data) or a managed service (Eversynced's backend with reporting dashboards). It also includes an OpenTelemetry (OTLP) passthrough for native metrics from Claude Code and Codex.

Problems Solved

1. Pain Point: Uncontrolled and unmeasured AI model spend. Teams lack visibility into whether they are over-provisioning expensive models (like Claude Opus) for simple tasks or under-utilizing AI capabilities, leading to wasted budget or suboptimal output.
2. Target Audience: Engineering leaders (CTOs, VPs of Engineering, Engineering Managers), DevOps/Platform teams, and individual developers in organizations adopting AI coding assistants. It is particularly relevant for teams using Claude Code, Cursor, or Codex in their daily workflow.
3. Use Cases:
   • Cost Optimization Audit: Identifying specific instances and patterns of "model misfit" to recapture thousands in overspend on AI API calls.
   • AI Workflow Enablement: Measuring the adoption and habitual use of best practices like running tests after AI edits, using context management effectively, or leveraging custom skills/MCP servers.
   • Team Coaching & Training: Using the "Practice Heatmap" and individual engineer views to identify structural gaps (e.g., no one uses evals) versus individual coaching opportunities (e.g., low pushback rate).
   • Proving ROI on AI Investment: Moving from vague claims of "increased productivity" to concrete metrics on verification rates, rework reduction, and efficient model usage.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros:
     • Privacy by Design: Captures interaction shape not content, addressing major security and IP concerns for enterprise adoption.
     • Actionable Framework: The predefined proficiency model (6 competencies, 5 signals) provides an immediate, opinionated lens for analysis, unlike raw telemetry dumps.
     • Developer-Centric Integration: Non-invasive hook-based installation requires no code changes to the primary codebase.
     • Deployment Flexibility: The open-source core and simple backend contract enable both full ownership (self-hosted) and convenience (managed service).
   • Cons:
     • Agent Lock-In: Currently only supports Claude Code, Cursor, and Codex. Developers using GitHub Copilot, Tabnine, or other AI assistants are not covered.
     • Interpretation Overhead: The value is heavily dependent on leadership's ability to interpret the proficiency metrics and translate them into process changes or training.
     • Initial Setup: Requires installation and configuration on each developer's machine, which can be a hurdle for large, decentralized teams without strong dev tooling practices.

2. Key Alternatives & Differentiation:

   • Generic Application Performance Monitoring (APM) / OpenTelemetry: Tools like Datadog, New Relic, or self-hosted Grafana stacks can collect metrics. Differentiation: Pheebs provides a pre-built, domain-specific schema and analytics model for AI coding interactions. Configuring an equivalent "proficiency model" in a generic APM would require significant custom instrumentation and dashboard building.
   • IDE Plugin Analytics: Some AI coding agents may have basic built-in analytics. Differentiation: Pheebs is agent-agnostic, allowing for cross-tool comparison (e.g., Cursor vs. Claude Code usage). It also focuses on higher-order behavioral analysis (proficiency, cost) rather than just simple usage counts.
   • Manual Process Audits / Surveys: The traditional method of assessing tool adoption. Differentiation: Pheebs provides continuous, objective, and granular data, eliminating recall bias and self-reporting inaccuracies. It reveals what people actually do, not what they say they do.

Frequently Asked Questions (FAQ)

1. Does Pheebs read or store my code or prompts? No. Pheebs is designed with a privacy-first architecture. It explicitly does not store source code, file paths, directory structures, or prompt text. A prompt is recorded only as a character count and, if the optional classifier is enabled, an intent label (e.g., "task"). The raw text is never written to disk or sent to the backend unless explicitly configured for the one-time classification request.
2. How does Pheebs calculate potential cost savings on model spend? Pheebs calculates the "Model-Fit Rate" by analyzing each session's recorded model against the assessed "size" of the work (derived from prompt intents and scope). The "savings opportunity" is an estimate based on the gap between the model used and a more appropriate, less expensive model, priced at the provider's list-price API rates. It shows the upper bound of potentially recoverable spend.
3. Can I use Pheebs completely offline or self-hosted? Yes. The core Pheebs client is open-source (Apache-2.0). You can run it in a local-only mode where data is solely written to the JSONL log file on the developer's machine. For centralized analysis, you can deploy the provided reference backend on your own infrastructure, ensuring all telemetry data never leaves your network.
4. What is the difference between the "Managed" service and the open-source tool? The open-source tool is the data collection client and a basic backend contract. The "Managed" service (Eversynced's AI Enablement Assessment) includes Eversynced hosting the backend, providing the pre-built reporting dashboards shown in the documentation, running the prompt intent classifier, and delivering a professional analysis and debrief based on your team's 30-day data.
5. How does Pheebs handle developer identification and privacy? Developer identity is resolved to a token ID (if a Pheebs backend token is set) or a truncated hash of the developer's git email address. It does not look up or store GitHub handles, real names, or full email addresses, maintaining anonymity within the telemetry stream.