Open Computer Use

Product Introduction

1. Definition: Open Computer Use is an open-source Model Context Protocol (MCP) server that provides a standardized, cross-platform desktop automation service. It functions as a middleware layer, translating high-level commands into native system-level interactions via accessibility APIs.
2. Core Value Proposition: It exists to democratize and standardize the "Computer Use" paradigm pioneered by OpenAI's Codex, enabling any AI agent or MCP-compatible client (like Claude Code, Gemini CLI, or opencode) to programmatically inspect, control, and automate graphical user interfaces (GUI) on macOS, Linux, and Windows desktops.

Main Features

1. Cross-Platform Desktop Automation via MCP: The service exposes a consistent set of tools (like list_apps, get_app_state, click, type, scroll) through the MCP standard. It abstracts the underlying platform-specific implementations: Apple Accessibility API (AXAPI) on macOS, AT-SPI on Linux, and UI Automation/UIA on Windows. This allows a single client command to work across all supported operating systems.
2. Visual Inspection and State Management: The get_app_state tool captures the hierarchical structure of a target application's UI, including window titles, button labels, text fields, and their positional metadata. This state is returned as structured JSON, enabling AI agents to reason about the screen and plan subsequent actions, maintaining context through element indexing.
3. Multi-Client Integration and Easy Deployment: It includes one-command installers for major AI agent environments (install-codex-mcp, install-claude-mcp, install-gemini-mcp, install-opencode-mcp). The core server is distributed as a global npm package (npm i -g open-computer-use), ensuring a simple, unified deployment process regardless of the end-user's chosen AI client stack.

Problems Solved

1. Pain Point: Fragmentation and complexity in building AI agents that require real-world desktop interaction. Traditionally, this required writing and maintaining separate, brittle automation scripts for each OS (e.g., AppleScript, PowerShell, xdotool).
2. Target Audience: AI Agent Developers, MCP Client Builders, QA Automation Engineers, and Power Users building custom automation workflows. Specifically, developers integrating AI capabilities into tools like Codex App, Claude Code, or custom Node.js/Python agents that need GUI interaction.
3. Use Cases: Automating repetitive cross-platform software tasks (data entry, app testing), enabling AI coding assistants to manipulate IDE settings or version control GUIs, building research agents that can gather data from desktop applications, and creating accessibility tools powered by natural language commands.

Unique Advantages

1. Differentiation: Unlike monolithic automation frameworks (e.g., Selenium for web, or dedicated RPA tools), Open Computer Use is a lightweight, protocol-based service focused solely on providing automation to AI agents. It is not a standalone IDE or recorder. Compared to the official (and closed) Codex Computer Use, it is open-source, extensible, and supports Linux officially.
2. Key Innovation: Its core innovation is the repurposing of system accessibility frameworks—designed for assistive technology—into a high-fidelity, real-time control plane for AI agents. The "Cursor Motion" subsystem for macOS, based on public research, provides a precise, non-intrusive visual cursor, which is critical for reliable clicking and dragging operations that users can observe and trust.

Frequently Asked Questions (FAQ)

1. How does Open Computer Use ensure security and privacy during desktop automation? The service operates entirely locally on the user's machine; no screen data or control signals are transmitted to external servers. On macOS, it requires explicit user grants for Accessibility and Screen Recording permissions, following the same security model as other legitimate automation and assistive tools.
2. Can Open Computer Use automate applications running in a web browser? While it can interact with the browser window itself (e.g., focusing, resizing), for deep web automation, the developers recommend the companion project open-browser-use, which is optimized for controlling web pages via DevTools Protocol, offering more precise element selection and web-specific actions.
3. What are the system requirements for running Open Computer Use? It requires Node.js (for the npm installation), a supported OS (macOS 10.13+, Windows 10+, or a Linux distribution with AT-SPI and X11/Wayland support), and on macOS, the necessary privacy permissions. The automation capabilities are contingent on the target applications themselves supporting the underlying platform's accessibility APIs.
4. How does the state management and element indexing work to prevent errors? When get_app_state is called, it returns a snapshot of the UI hierarchy with indexed elements. Subsequent actions (like click) can reference these indices. The call --calls-file feature allows running a sequence of tools in a single process, maintaining the context of the initial state snapshot to reduce errors from a dynamically changing UI between separate calls.