The Spectrum Dispatch News

technology

New macOS CLI Lets AI Agents Point at UI Elements with Click‑Through Arrows

The open‑source tool draws a transient, click‑through arrow on screen to guide humans toward buttons or dialogs that AI agents cannot interact with directly.

New macOS CLI Lets AI Agents Point at UI Elements with Click‑Through Arrows

bigarrow is a small macOS command‑line interface and an accompanying agent skill released under the MIT license. According to the source text, it draws a click‑through arrow that never steals the keyboard focus and disappears automatically after a set time or when the agent that created it exits. The binary is a single Swift executable with no daemon, no menu‑bar icon, no account, no telemetry and no AI components built in.

New macOS CLI Lets AI Agents Point at UI Elements with Click‑Through Arrows

The primary purpose is to give AI agents a way to indicate where a human should act when the agent cannot perform the interaction itself. For example, the command bigarrow point --element "Allow" --app "System Settings" --text "Franz, click Allow" produces a large, friendly arrow with a text label that points at the Allow button in a System Settings dialog. The arrow is click‑through everywhere except on its own sign and shaft; clicking the sign removes the arrow, while clicking the target or anywhere near the arrow’s head passes the click through to the underlying application.

bigarrow works across all displays, Spaces, full‑screen apps and Stage Manager. It can be positioned by UI element label (--element), by screen coordinates (--at X,Y), by a window or tab title (--window App[:title]), or by a Peekaboo snapshot. The tool automatically brings the target app, window or tab to the front unless the --no‑raise flag is used. Multiple arrows can be displayed simultaneously, and their labels are arranged to avoid overlapping.

Visual appearance is highly configurable. Users can choose arrow shape (bend, straight, zigzag, spiral), style (arrow, ring, box), size (S, M, L), corner rounding (round, sharp), and colour (named colours or hex values). Border options include a default white border with drop shadow, a thin black edge outside the white border, or a plain black outline. A clickable close button (--close‑button) can be added, and the arrow can be set to follow a moving element, stop on a click (--until‑click), or speak its label aloud via --say.

For documentation, the arrow can be rendered directly to a PNG with --png. In debugging mode, --dry-run --json reports the coordinates where the arrow would appear without drawing it. The tool reports exit codes: 0 for success, 2 for bad input, 3 when the target cannot be found, and 4 when a required permission (such as Accessibility for element‑based targeting) is missing; bigarrow doctor shows which app holds the permission and details about connected displays.

Performance measurements on a CI runner indicate that a pulsing arrow consumes about 1.4 % of CPU, handled by Core Animation in the render server. The --element feature works inside Electron apps and, in Chrome, only when Chrome is launched with --force-renderer-accessibility or when VoiceOver is enabled; Chrome’s own toolbar remains accessible without those flags.

Installation is possible via Homebrew (brew install franzenzenhofer/tap/bigarrow) or by building from source with Swift (swift build -c release, requiring Xcode 16+ and macOS 14+). The bigarrow install-skill command adds the capability to Claude Code and Codex agent environments.

Key facts

Sources

← All posts