Pawvis

Product Introduction

1. Definition: Pawvis is a macOS-native, open-source accessibility and productivity application that transforms a standard webcam into a touch-free, hand-tracking input device. It functions as a software-based alternative to physical mice and touchpads, leveraging on-device machine learning for real-time gesture control.
2. Core Value Proposition: Pawvis exists to provide a completely hardware-free, private, and customizable pointing interface for macOS. Its primary value is enabling hands-free computer control, ergonomic wrist relief, and accessible computing by using your hand as an air mouse, all processed with 100% on-device privacy and requiring no extra hardware.

Main Features

1. Vision-Based Hand Tracking: Pawvis uses Apple's native Vision framework to perform real-time hand pose estimation, tracking 21 skeletal joints of the user's hand through the built-in or external webcam. It intelligently activates cursor control only when a specific "open hand" pose (all four fingers extended) is detected, preventing accidental cursor movement during normal hand activity.
2. Gesture-Based Mouse Emulation: The core interaction model replicates a multi-button mouse through intuitive hand gestures. Cursor movement is mapped to palm translation. Left-click is triggered by dipping the index finger. Right-click is performed by dipping the pinky finger. Scrolling is activated by folding the middle and ring fingers while moving the hand vertically or horizontally. Drag-and-hold is initiated by maintaining a finger dip while moving.
3. Custom Gesture Training & Automation: Beyond basic mouse functions, Pawvis includes a machine learning-based custom gesture trainer. Users can record unique hand motions (e.g., finger wiggles, specific poses) which the app learns to recognize. These gestures can be bound to system-wide or per-app keyboard shortcuts, application launches, window management commands (like moving or resizing), or any Automator/scriptable action, creating a powerful gesture-based macro system.
4. On-Device Voice Control (Beta): An optional, privacy-focused voice command layer uses Apple's on-device Speech Recognition engine. Users set a configurable wake word ("Pawvis" by default) to issue commands for navigation (e.g., "go to github.com"), text input ("type good morning"), app control ("open Notes"), and system actions. Complex commands can be executed via on-device Apple Intelligence (macOS 26+) for step-by-step visual automation or delegated to external AI agents like Claude Code or Codex for advanced computer-use tasks.

Problems Solved

1. Pain Point: Repetitive Strain Injury (RSI) and ergonomic discomfort from prolonged mouse and keyboard use. Pawvis allows users to alternate input methods, using an open-hand posture to browse and scroll, providing relief for wrists and hands.
2. Target Audience: Professionals with accessibility needs seeking alternative input methods; presenters and educators needing remote control of their Mac from across the room; makers, cooks, and lab technicians with dirty or occupied hands who need to reference digital information; developers and power users interested in customizable, gesture-based automation.
3. Use Cases: Hands-free browsing and document navigation while eating or handling objects; controlling slide decks during presentations without a clicker; following a digital recipe or instruction manual in the kitchen or workshop without touching the device; creating a customizable, gesture-driven shortcut panel for specific creative or development workflows.

Unique Advantages

1. Differentiation: Unlike macOS's built-in Head Pointer (which requires head movement) or Voice Control (which relies on voice grids), Pawvis offers direct, 1:1 hand-to-cursor mapping with deliberate click gestures. Compared to commercial hand-tracking hardware (e.g., Leap Motion), it requires zero additional hardware, using the existing webcam, and is free and open-source (MIT licensed).
2. Key Innovation: The integration of multiple on-device Apple frameworks (Vision, Speech) into a single, cohesive, and private input system. The ability to train custom gestures for system automation without cloud dependency is a significant technical differentiator. The optional agent hand-off to Claude Code/Codex represents a novel fusion of local gesture/voice control with powerful AI-driven computer automation.

Frequently Asked Questions (FAQ)

1. Does Pawvis work with Zoom or other video conferencing apps? Yes, on most modern Macs, macOS allows camera sharing between applications. Pawvis and Zoom can typically use the webcam simultaneously. If an app takes exclusive control, Pawvis will pause tracking until the camera is released.
2. What are the system requirements and performance impact of Pawvis? Pawvis requires macOS 14 or later and works on both Intel and Apple Silicon Macs, though it runs more efficiently on Apple Silicon due to the Neural Engine. It performs continuous Vision framework hand-pose inference, which consumes CPU/GPU resources; users should disable tracking via the menu bar when not in use to conserve battery.
3. Is my data private with Pawvis? By default, all processing is on-device. Hand tracking, speech recognition, and visual command analysis occur locally using Apple's frameworks, with camera frames processed in memory and discarded. The only exception is the optional AI agent hand-off feature (Claude Code/Codex), which sends voice commands to an external service and is disabled by default with a clear warning.
4. Can I use Pawvis in low light or with an external camera? Tracking reliability depends on camera image quality. It requires adequate lighting on the user's hand. It supports any camera visible to macOS, including external USB webcams and Continuity Camera from an iPhone, selectable within the app's settings.
5. How do I install and set up Pawvis? Download the signed and notarized Pawvis.zip from GitHub, unzip it, and drag the app to your Applications folder. On first run, it will request necessary permissions: Camera for tracking, Accessibility to control the cursor and click, Microphone for voice control, and optionally Screen Recording for advanced voice command features.