sideOS.ai

Product Introduction

1. Definition: sideOS.ai is a native macOS AI agent application that operates from the MacBook's notch. It functions as a context-aware, voice-first desktop automation and productivity tool. Technically, it is a system-level accessibility client and automation orchestrator that leverages on-device speech recognition, screen reading via the macOS Accessibility API, and integration with large language models (LLMs) to execute user commands directly within the operating system and applications.
2. Core Value Proposition: sideOS.ai exists to bridge the gap between conversational AI and tangible desktop action. Its primary value is enabling users to control their Mac through natural language voice commands, automating complex multi-step workflows that traditionally require manual clicking, typing, and app switching. Key proposition keywords include: voice-controlled Mac automation, on-device AI assistant, screen-reading AI agent, local AI for macOS, and privacy-first desktop automation.

Main Features

1. Voice-First Agent with Screen Context: The core feature is a push-to-talk agent activated by holding Control + Option. It uses macOS's native SFSpeechRecognizer or SpeechAnalyzer engines for zero-cost, offline dictation and command capture. Upon release, it reads the current screen state in ~50ms by parsing the live accessibility tree (AX API) to understand UI elements, then uses a configured LLM (cloud or local) to plan actions. It executes tasks via a toolbox of system actions like open_app, click_element, type_text, run_applescript, and press_keys.
2. Visual Region Select & Analysis: Users can drag a selection box while holding the activation keys to capture a specific screen region. sideOS.ai sends both the cropped pixels at native resolution and the relevant portion of the accessibility tree for that region to the model. This hybrid approach ensures accurate analysis of both structured UI elements and visual content like charts, graphs, or code errors.
3. Background Task Engine: Long or complex jobs can be offloaded to a dedicated background agent. This loop runs separately from the main interface and can execute chained operations like web_search, read_file, write_file, and render_pdf. Tasks requiring approval (e.g., run_shell) trigger a prompt. All output is saved to a dedicated Documents/sideOS Tasks folder, and the user is notified upon completion.
4. Integrated Meeting Assistant: The tool detects windows from Zoom, Google Meet, or Microsoft Teams and offers one-tap recording. Audio is transcribed locally using Whisper.cpp, with no cloud upload. It performs speaker diarization, generates summaries, and extracts action items which can be one-click converted into reminders with meeting context attached.
5. Tool Integration via MCP (Model Context Protocol): sideOS.ai can connect to external services like Gmail, Calendar, Notion, Slack, and GitHub by integrating MCP servers. These connections add service-specific tools (e.g., "send email," "create calendar event") to the agent's existing toolbox, enabling cross-application workflows. User secrets are stored in the macOS Keychain.
6. Local-Only Operation Mode: For maximum privacy, users can bypass cloud APIs entirely. The agent can be configured to run with local MLX models (like Qwen2.5 4B or 8B) for reasoning and a separate vision model for screen understanding. In this mode, dictation, transcription, model inference, and data storage all occur on-device with no network traffic.

Problems Solved

1. Pain Point: Context Switching and Manual Grunt Work. The constant need to switch between applications, manually copy-paste data, describe on-screen content to a chatbot, and then manually execute its suggestions destroys focus and productivity.
2. Target Audience: Knowledge Workers and Power Users on macOS. This includes software engineers, product managers, executives, researchers, writers, and data analysts who perform repetitive digital tasks, manage communications across multiple apps, and participate in frequent meetings requiring follow-up.
3. Use Cases:
   • Cross-App Workflow Automation: "Reply to this email politely declining, then create a calendar event for a follow-up in two weeks."
   • Data Synthesis & Reporting: "Research Q3 competitor pricing online, analyze my local notes, and write a summary report saved as a PDF."
   • Meeting Management: Automatically record, transcribe, summarize calls, and convert action items into tracked reminders.
   • In-Context Assistance: Circle a chart on screen and ask, "What's driving this spike?" or point at an error log and say, "Explain this and suggest a fix."
   • Hands-Free Documentation: Use the offline dictation mode to type notes, code comments, or emails without touching the keyboard.

Unique Advantages

1. Strengths & Limitations (Pros & Cons):

   • Pros:
     • Unmatched Native Integration: Deep hooks into macOS via Accessibility API and AppleScript allow for reliable, low-latency UI interaction that browser-based agents cannot match.
     • Privacy-Centric Architecture: Core features (dictation, meeting transcription, local model mode) run offline. API keys are stored in Keychain, and screen data is only processed per-request.
     • One-Time Purchase Model: A lifetime licence contrasts sharply with the SaaS subscription model prevalent in productivity software.
     • Hybrid Input Processing: Combining the accessibility tree (for speed/structure) with selective screenshots (for visuals) provides more accurate context than vision-only models.
   • Cons:
     • Platform Lock-in: Exclusively for macOS, with no Windows or Linux support.
     • Dependency on App Accessibility: Performance can vary with poorly implemented or non-native applications that have incomplete accessibility trees.
     • Cloud Model Cost Uncertainty: While the app itself is a fixed cost, heavy usage with premium cloud LLMs (Claude 3.5 Sonnet, GPT-4) can lead to variable, ongoing API costs.
     • Learning Curve: Understanding the scope and phrasing of effective commands for complex automation requires some user adaptation.

2. Key Alternatives & Differentiation:

   • Cursor IDE / Windsurf (Agentive Coding): These are AI-powered code editors focused exclusively on software development within the IDE. sideOS.ai is a system-wide assistant that works in any app (Mail, Safari, Finder) and automates general desktop tasks, not just coding.
   • Macro Tools (Keyboard Maestro, Alfred): These are powerful but require manual, rule-based programming of workflows (macros). sideOS.ai uses natural language understanding and AI planning to execute ad-hoc, multi-step commands without pre-configuration.
   • Cloud AI Assistants (ChatGPT Desktop, Claude Desktop): These are primarily chat interfaces that may have "screen reader" features but typically operate by taking screenshots, lacking deep OS integration for direct control. sideOS.ai is a true agent with a built-in toolbox to act (click, type, open) rather than just advise. Its one-time fee also differs from their subscription-based premium tiers.

Frequently Asked Questions (FAQ)

1. Does sideOS.ai work without an internet connection? Yes, in multiple ways. Dictation and speech recognition use macOS's on-device engines. The entire AI agent can run using locally installed MLX models (like Qwen2.5), enabling complete offline operation for both understanding and task execution. Meeting transcription also uses a local Whisper model.

2. How does sideOS.ai's privacy compare to other AI assistants? sideOS.ai is architected for local-first privacy. Your audio, screen data, and meeting recordings are processed on your Mac. When using cloud models (OpenAI, Anthropic), only the minimal necessary text context from your screen is sent via your own API key. The company states no data is used for training, and anonymous usage statistics can be disabled.

3. What is the real cost of using sideOS.ai? The app is a one-time purchase (currently $89.90). The primary ongoing cost is for the LLM API calls if you use cloud providers like OpenAI or Anthropic. The developer estimates ~2-5¢ per typical command. Using local models or the free tier of certain APIs can reduce this to zero.

4. Can sideOS.ai automate tasks in any application? It works best with standard macOS applications that have robust accessibility support (Safari, Mail, Notes, etc.) and those scriptable via AppleScript. Performance may be limited in custom or cross-platform apps (e.g., some Electron apps) with poor accessibility tree implementation, where it may fall back to less precise screenshot analysis.

5. How does the MCP (Model Context Protocol) integration work? MCP allows sideOS.ai to connect to external data sources and tools. You can connect pre-configured servers for services like Gmail or Notion, or paste a custom server configuration (similar to Claude Desktop). This extends the agent's toolbox, allowing it to perform actions like "check my calendar" or "create a Notion page" directly from a voice command.