Openclick

Product Introduction

1. Definition: Openclick is an experimental, open-source Command Line Interface (CLI) tool designed for agentic macOS UI automation. It functions as a specialized automation wrapper that translates natural language prompts into executable UI actions by leveraging Large Language Models (LLMs) and the native macOS Accessibility (AX) framework.

2. Core Value Proposition: Openclick exists to bridge the gap between high-level human intent and low-level desktop interaction. By utilizing an "LLM-as-a-planner" architecture, it eliminates the need for manual scripting in languages like AppleScript or Swift. It provides a flexible, prompt-based alternative to traditional robotic process automation (RPA), specifically optimized for the macOS ecosystem and local-first privacy.

Main Features

1. Prompt-to-Action Plan Generation: Openclick utilizes an LLM to interpret complex natural language instructions and decompose them into a structured sequence of UI events. These actions include mouse clicks, keystrokes, and Accessibility (AX) attribute reads. The model functions as a reasoning engine that maps user intent to the specific UI hierarchy of active macOS applications.

2. macOS Accessibility API Runner: The core execution engine interacts directly with the macOS Accessibility tree. This allows the tool to identify and manipulate UI elements (buttons, text fields, menus) across diverse applications such as Safari, Notes, Slack, and Terminal. Unlike vision-based automation, Openclick relies on the structured data provided by the AX framework, ensuring higher precision in element targeting.

3. Bring Your Own Model (BYOM) Architecture: Openclick is model-agnostic, supporting a wide range of inference endpoints. Users can connect to cloud providers like OpenAI, Anthropic, and DeepSeek, or utilize local inference via Ollama or LM Studio. This flexibility allows users to swap models based on task complexity, cost, or privacy requirements using the --model flag.

4. Human-in-the-Loop Safety Controls: To mitigate the risks associated with LLM hallucinations, Openclick implements a "Plan-Confirm-Run" workflow. By default, the tool operates in --dry-run mode, printing the proposed action sequence for user review. Every call to the macOS Accessibility API is logged in real-time, allowing users to edit or abort the plan before any changes are made to the system state.

5. Local-First Privacy and Security: As a CLI-based tool with no required accounts or telemetry, Openclick ensures that prompts and screen metadata remain on the local machine. When paired with local LLMs (via Ollama or LM Studio), the entire automation pipeline—from prompt processing to UI execution—operates within the user's local environment, making it suitable for handling sensitive meeting notes or personal data.

Problems Solved

1. Pain Point: Traditional macOS automation methods, such as AppleScript, Automator, or Shortcuts, require significant technical expertise and are often brittle when app interfaces change. Openclick solves this by using natural language as the interface, allowing for "fuzzy" intent matching that adapts to different UI states without hardcoded scripts.

2. Target Audience: The primary users include software developers seeking to automate repetitive workflows, power users who manage complex multi-app tasks, and accessibility researchers. It is particularly valuable for users who prefer CLI-driven environments and those looking for an open-source alternative to proprietary AI desktop agents.

3. Use Cases:

• Automated Communications: Summarizing meeting notes and drafting contextual email replies in Mail.app.
• File and System Management: Reorganizing complex directory structures (e.g., Downloads folder) based on file metadata and content.
• Research and Data Extraction: Searching travel sites for specific flight criteria and saving structured data to the clipboard or local files.
• Calendar Orchestration: Intelligent time-blocking and conflict resolution within the macOS Calendar app based on high-level scheduling preferences.

Unique Advantages

1. Differentiation: Unlike closed-source AI agents or heavy RPA software, Openclick is a "thin shell" that adheres to the Unix philosophy. It provides a lightweight, MIT-licensed foundation that is easily inspectable and forkable. It distinguishes itself from pixel-based agents by using the structured macOS Accessibility tree, which is more robust and less resource-intensive than continuous screen recording/vision processing.

2. Key Innovation: The project's innovation lies in its direct integration of the LLM planning phase with the macOS AXTree. This allows the model to "see" the UI through the same lens as assistive technologies, providing a semantic understanding of application states that vision-only models often lack.

Frequently Asked Questions (FAQ)

1. Is Openclick compatible with all macOS applications? Openclick can interact with any application that properly exposes its UI elements through the macOS Accessibility tree. While native apps like Safari, Mail, and Notes have excellent support, applications with non-standard UI frameworks (like certain Electron apps or games) may have "poor AX support" and might not be fully controllable.

2. Does Openclick require an internet connection to function? While Openclick can connect to cloud-based LLMs like OpenAI or Anthropic, it is fully compatible with local inference engines such as Ollama and LM Studio. When configured with a local model, Openclick can perform macOS UI automation entirely offline, ensuring maximum data privacy.

3. How does Openclick prevent accidental actions on my Mac? Safety is handled through a mandatory review process. Before executing any plan, Openclick prints the entire sequence of proposed actions to the terminal. The user must manually confirm the plan before execution begins. Additionally, the --dry-run mode allows users to test prompts and see the model's logic without triggering any actual UI interactions.

4. What are the technical requirements to install Openclick? Openclick requires macOS and Node.js version 20 or higher. Because it drives the UI via the Accessibility APIs, users must manually grant "Accessibility" permissions to the terminal (e.g., iTerm2 or Terminal.app) or the Openclick binary within the macOS System Settings.