Scrawl

Product Introduction

1. Definition: Scrawl is a context-aware screen recording and annotation tool designed specifically for AI-assisted development workflows. It is a native desktop application (macOS/Windows) that captures screen activity, audio narration, mouse clicks, and on-screen drawings, packaging them into a single, intelligent artifact for AI coding agents.
2. Core Value Proposition: It exists to solve the communication gap between a developer and an AI coding assistant (like Cursor, Claude Code, or GitHub Copilot). Instead of writing lengthy prompts to describe a visual bug or UI flow, Scrawl allows developers to show the problem directly through a multimodal capture that includes precisely timed frames, transcribed speech, and visual annotations.

Main Features

1. Intelligent, Event-Driven Frame Capture: Scrawl does not record a continuous video. It uses a heuristic-based system to take discrete screenshots (frames) only when a relevant event occurs: a mouse click, a drawing stroke, a spoken trigger word ("look" or "now"), or a detectable change in the screen content. This creates a lightweight, information-dense "contact sheet" instead of a large video file.
2. On-Device Speech-to-Text & Contextual Alignment: All audio narration is transcribed locally on the user's computer (macOS), ensuring privacy and low latency. Crucially, each transcribed sentence is temporally aligned and attached to the specific frame that was active when it was spoken, providing perfect context for the AI agent.
3. Dynamic Screen Cropping & iPhone Integration: To reduce visual noise, Scrawl automatically crops each captured frame to the active window or region where the user interacted (clicked or drew). It natively supports iPhone Mirroring on macOS, allowing recording of iOS simulator or a physical iPhone screen in a window. For deeper iOS integration, ScrawlKit enables in-app recording directly on the device, capturing every tap and swipe.
4. Agent-Focused Output & "Closer Look" Command: The final output is a single, compact data structure (pasted as text/markdown) containing frame references, transcript, and metadata. Integrated AI agents can use a dedicated scrawl command to request a high-density grid of frames around any timestamp or spoken phrase, enabling detailed forensic analysis of a specific moment.

Problems Solved

1. Pain Point: Inefficient and ambiguous communication with AI coding agents. Describing complex visual bugs, UI state, or multi-step workflows via text prompts is time-consuming and prone to misinterpretation.
2. Target Audience: Software engineers, QA testers, product managers, and technical educators who regularly use AI pair programmers. It is particularly valuable for front-end developers (React, Vue, etc.), mobile app developers (iOS), and anyone debugging visual or interactive elements.
3. Use Cases: Demonstrating a visual regression bug, explaining a complex refactoring task across multiple files, walking through a user onboarding flow, reporting an issue with an iOS app's gesture handling, or creating a detailed tutorial for a specific code interaction.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros: Unparalleled context delivery to AI agents, drastically improving prompt understanding and solution accuracy. Privacy-focused, local-first processing of audio and screenshots. Extremely efficient output format (not bulky video). Deep integration with macOS/iOS ecosystem. Open-source (MIT license) for transparency and customization.
   • Cons: Currently platform-restricted (macOS 15+ on Apple Silicon, Windows preview). Requires an AI agent that can parse its specific output format for full benefit. The "preview, unsigned" Windows version may trigger security warnings. The event-driven capture might miss subtle, non-interactive visual changes if not triggered by a defined event.

2. Key Alternatives & Differentiation:

   • Standard Screen Recorders (Loom, QuickTime): Produce large video files that AI agents cannot natively parse or search. Lack integrated transcription aligned to specific screen events, automatic cropping, and the structured data output for AI consumption. Scrawl is purpose-built for agent interaction, not human video sharing.
   • Screenshot + Manual Prompting: The manual alternative involves taking multiple screenshots, uploading them, and writing a detailed prompt. Scrawl automates and structures this entire workflow into a single, rapid action (⌥⇧S), with intelligent framing and transcription baked in.
   • Other AI Context Tools: While some AI IDEs have built-in "@codebase" or screenshot features, Scrawl is a standalone, agent-agnostic tool that provides a richer, multimodal context (voice, drawings, sequential frames) and works across any application or development environment.

Frequently Asked Questions (FAQ)

1. Is Scrawl free and open source? Yes, Scrawl is completely free to use and is released under the permissive MIT open-source license. The source code is available on GitHub for review, contribution, or self-hosting.
2. How does Scrawl handle privacy with screen recording and audio? Scrawl is designed with a privacy-first architecture. All processing—including screen capture, audio recording, and speech transcription—occurs locally on your machine. No screen data or audio is sent to external servers unless you explicitly paste the output into a cloud-based AI agent.
3. Can I use Scrawl with any AI coding assistant? Yes, the core functionality (recording and generating a capture) works independently. You can paste the text output into any AI chat interface. For the advanced "closer look" feature, the AI agent needs to integrate the Scrawl skill/command, which is currently supported in agent frameworks like Cursor's Composer.
4. What are the system requirements for Scrawl on Mac? Scrawl for Mac requires a computer with Apple silicon (M-series chip) running macOS 15 (Sequoia) or later. This is due to its reliance on newer macOS APIs for advanced screen capture and iPhone integration.
5. How do I record my iPhone screen with Scrawl? On macOS 15+, you can use the built-in iPhone Mirroring feature to display your iPhone's screen in a window on your Mac. You can then start a Scrawl recording and interact with that window as you would any other application, with your voice and drawings overlaid.