dif.sh

Product Introduction

1. Definition: dif.sh is an open-source developer tool and SDK for implementing feature flags, A/B tests, holdouts, and staged rollouts. It is technically categorized as a feature management and experimentation platform designed to be version-controlled and infrastructure-as-code (IaC) native.
2. Core Value Proposition: It exists to move the logic, configuration, and history of feature flags and experiments out of external SaaS dashboards and into a project's Git repository as Markdown and YAML files. This enables PR-based workflows, eliminates vendor lock-in, ensures the experiment definition is always in sync with the code it controls, and provides a portable context file for AI coding agents.

Main Features

1. Markdown-First Experiment Definition: Each feature flag or A/B test is a single Markdown file (your-experiment.md) stored in dif/experiments/active/. The file uses YAML frontmatter for machine-readable configuration (ID, status, hypothesis, audience, variants) and Markdown sections for human-readable documentation (Brief, Rationale, Decision). This creates a single source of truth reviewed in a pull request.
2. Build-Time Conflict Resolution & Client Generation: The dif build command is a core compiler. It parses all active experiment files, validates the schema, resolves the exclusion group graph to prevent user bucketing conflicts, and fails the build if clashes are detected. It then generates a strongly-typed, tree-shakeable client SDK (dif/generated/client.ts) that is imported directly into the application, ensuring runtime decisions are local with no network latency.
3. Git-Centric Audit Log and Institutional Memory: Concluded experiments are moved to dif/experiments/concluded/ by dif conclude, which also drafts a Decision block in the file. Crucially, it appends a one-line summary to a surface log file (e.g., dif/surfaces/checkout.md). This creates a running, versioned history of learnings for each part of the application, which the dif new command reads to inform the hypothesis of new drafts.
4. Runtime-Agnostic Audience Targeting: Audience segmentation is declared in the experiment file's frontmatter (e.g., include: - returning_visitor: true). The actual attribute values (like country or plan) are provided at runtime from the application's user context via the SDK's dif.track() method or initialization. No sensitive customer data or lists are ever stored in the repository.
5. Coding Agent Context (context.json): Every build regenerates a dif/context.json file containing a structured summary of all active experiments, variants, and recent conclusions. This file is designed to be read by AI coding assistants at the start of a development session, providing them with immediate context about ongoing feature work and past learnings.

Problems Solved

1. Pain Point: Disconnect between code and feature management. Traditional SaaS platforms create a separate system of record, leading to drift where the live experiment configuration doesn't match the code expecting it, or where the "why" behind a flag is lost in a separate tool's notes.
2. Pain Point: Opaque and non-portable experiment history. Learning from past A/B tests is often trapped in dashboards or spreadsheets, not integrated into the development workflow, making it easy for teams to repeat mistakes.
3. Target Audience: Engineering teams (particularly in startups and mid-size companies) who value DevOps practices, infrastructure-as-code, and PR-based review cycles. Product managers and data scientists who need rigorous, well-documented experimentation integrated into the development lifecycle.
4. Use Cases: Safely rolling out a new checkout UI with a feature flag. Running a statistically valid A/B test on pricing page copy. Implementing a staged rollout (percentage-based release) of a high-risk backend service. Creating a long-term holdout group to measure system-wide impact. Ensuring AI coding agents have full context of active experiments when modifying related code.

Unique Advantages

1. Differentiation: Unlike SaaS platforms (LaunchDarkly, Optimizely), dif.sh has no external control plane, dashboard, or vendor runtime. The entire state is in your repo. Unlike simple open-source flag SDKs (Unleash), it bakes in experimentation-specific concepts (hypotheses, metrics, exclusion groups), a built-in audit log, and a compiler for safety, moving complexity from runtime to build time.
2. Key Innovation: The integration of the experiment lifecycle—from draft (dif new) to conclusion and learning capture (dif conclude)—directly into the file system and Git workflow. The surface log files transform ephemeral experiment results into a persistent, versioned institutional memory that directly seeds future work.

Frequently Asked Questions (FAQ)

1. How does dif.sh handle A/B testing analytics and compute statistical significance? The core open-source dif.sh CLI and SDK handle experiment definition, user bucketing, and event tracking. For analytics, you can use dif.track() to send events to a connected Dif Cloud account (a paid SaaS add-on) which automatically computes lift and confidence intervals. Alternatively, you can use dif init --events custom to own the event handlers and forward data to your own analytics pipeline (e.g., Segment, Amplitude, a data warehouse).
2. Is dif.sh suitable for large-scale, enterprise applications? Its architecture is designed for scale at the build/compile stage, not runtime. The generated client is minimal and fast. The primary considerations for large enterprises would be the need to manage the dif/ directory across many microservices and to potentially build internal tooling around the file-based workflow, as it lacks a centralized management GUI.
3. Can I use dif.sh with a coding agent like Claude Code or Cursor? Yes, this is a primary design goal. The AI agent reads the generated dif/context.json file at project open, giving it immediate awareness of all active feature flags, their status, owners, and recent conclusions from surface logs. This allows the agent to reason about code changes in the context of live experiments.
4. What happens if two developers create experiments with conflicting exclusion groups? The dif build command will fail. It constructs a graph of all active experiments and their exclusion_group assignments. If it detects that a single user could be bucketed into two experiments within the same group, it throws a compilation error, preventing the conflicting configuration from being deployed. This conflict check is enforced in CI/CD.
5. How do I manage secrets or environment-specific configuration? dif.sh's configuration is intentionally static and committed. For environment-specific overrides (e.g., disabling all experiments in a staging environment), you would use your application's existing environment variable system to conditionally skip the dif client initialization or override the status check at runtime within your application code.