iphone-use

Product Introduction

1. Definition: iphone-use is an open-source, self-hosted automation framework and Model Context Protocol (MCP) server that provides AI agents and developers with programmatic control over a real, physical iPhone. It functions as a bridge between high-level AI instructions and low-level device interaction via Apple's WebDriverAgent (WDA).
2. Core Value Proposition: It exists to enable reliable, deterministic, and honest automation of iOS applications—including sensitive apps like banking and payment platforms that block traditional screen capture—by providing AI agents with a truthful, state-aware interface to a real iPhone's screen and input system.

Main Features

1. Honest, State-Aware Agent Interface: The system provides the iPhone screen as structured text and a snapshot token. When an agent issues a command (tap, type, scroll), the response explicitly states the outcome: "applied," "not sent," or "unknown," along with whether a retry is safe. This prevents agents from operating on stale or incorrect assumptions, a critical failure point in UI automation.
2. Multi-Access Protocol Support: It exposes device control through three primary interfaces: a RESTful HTTP API for scripts and custom integrations; a comprehensive MCP server with 21 tools for seamless use with Claude Code, Claude Desktop, and other MCP clients; and a human-accessible web UI and native iOS app for manual remote control and monitoring.
3. Model-Free, Replayable Flows: Users can record a multi-step task (a "flow") once. The system then stores it as a reviewed, version-controlled script in a public registry. These flows can be replayed deterministically without invoking an LLM, eliminating token costs and latency for repetitive tasks. The daemon automatically suggests saving new tasks as flows and syncs the public registry.
4. Resilience to Security Screens: A key technical capability is handling apps that employ secureTextEntry or UIPreventScreenCapture. Instead of failing or returning blank data, iphone-use returns a wireframe representation of the screen, allowing the agent to understand the UI layout and proceed logically without violating app security.

Problems Solved

1. Pain Point: The inability to automate iOS applications that lack a public API, especially fintech, banking, and enterprise apps with strict anti-botting and screen capture protections.
2. Pain Point: The unreliability and "silent failure" modes in typical UI automation, where an agent cannot distinguish between a successful action, a failed action, or a changed UI state.
3. Target Audience: AI Agent Developers and Researchers building agents that require interaction with mobile-only services; QA Automation Engineers needing robust testing for complex iOS apps; Power Users and RPA (Robotic Process Automation) practitioners seeking to automate personal mobile workflows.
4. Use Cases: Automating mobile banking transfers or statement checks; completing app-based two-factor authentication (2FA) within an agent workflow; automated social media posting or management from a phone; regression testing for iOS apps with sensitive input fields; creating personal automation scripts for repetitive mobile tasks.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros:
     • Truthful Feedback: The "honest outcome" model is a fundamental architectural advantage for reliable agentic systems.
     • Bypasses Screen Capture Blocks: The wireframe feature uniquely tackles a major hurdle in mobile automation.
     • Cost-Efficient via Flows: The flow system converts expensive LLM operations into cheap, repeatable scripts.
     • Self-Hosted & Open Source: Offers full control, privacy, and no dependency on external SaaS services.
   • Cons:
     • Complex Local Setup: Requires a dedicated Mac, a developer Apple ID, a physically connected iPhone, and technical steps to configure WebDriverAgent. This creates a significant barrier to entry.
     • Network Security Model: The primary authentication protects the Mac daemon, but WebDriverAgent's ports on the iPhone are unauthenticated, necessitating use on a trusted, secure network.
     • Mac-Centric: The control server runs exclusively on macOS, limiting deployment flexibility.

2. Key Alternatives & Differentiation:

   • Appium: The industry-standard open-source mobile automation framework. While powerful, it's designed for QA testing, not agentic interaction. It lacks the native "honest outcome" reporting, MCP integration, and flow replay systems. iphone-use is a specialized layer on top of WDA (which Appium also uses) optimized for AI agents.
   • Commercial RPA Platforms (e.g., UiPath, Automation Anywhere): Offer iOS automation via agents or VMs, but are closed-source, expensive, and part of a large enterprise suite. iphone-use is a lightweight, focused, and free alternative for developers and tech-savvy users.
   • Cloud-based Device Farms (e.g., BrowserStack, Sauce Labs): Provide access to real iOS devices via the cloud but are cost-prohibitive for continuous, long-running agent tasks and may not allow the low-level control or custom WDA setup required for secure apps. iphone-use is for private, always-available device access.

Frequently Asked Questions (FAQ)

1. Can iphone-use automate banking apps that block screenshots? Yes. This is a core feature. When an app activates screen capture protection, iphone-use returns a wireframe representation of the UI elements, allowing the automation logic to proceed based on layout rather than pixel data.
2. What are the system requirements to run iphone-use? You need a Mac running macOS 15 or later, with Xcode installed and signed into a developer account (a free Apple Developer Personal Team suffices). You also need an iPhone with Developer Mode enabled, connected via USB, and the libimobiledevice package installed via Homebrew.
3. How does the "flow" system save on AI costs? Once a task is performed manually or by an agent, it can be saved as a "flow"—a recorded sequence of actions. Future executions replay this script directly through the WebDriverAgent interface without querying an LLM for each step, eliminating all associated token costs and latency.
4. Is it safe to use with my personal iPhone and accounts? The software is open-source for auditability and runs locally. However, the security model requires your iPhone's WebDriverAgent ports to be exposed on your local network. The recommendation is to use it only on a trusted, private network (like a home network) to mitigate potential local access risks.
5. Can I use iphone-use with AI platforms other than Claude? Yes. While it includes a tailored MCP server for Claude, the primary HTTP API (/agent/* endpoints) is platform-agnostic. Any AI agent, script, or application that can make HTTP requests can be configured to use iphone-use as its "hands" for iPhone control.