revert: computer-use cua-driver (PR #16919) (#16927)

Reverts PR #16919 (commits dad10a78d, 413ee1a28, b4a8031b2, afb958829) which was merged prematurely. Restoring the pre-merge state so #14817 and #15328 can be revisited as standing PRs. Reverted commits: - afb958829 fix(computer-use): harden image-rejection fallback + AUTHOR_MAP - b4a8031b2 fix(computer-use): unwrap _multimodal tool results - 413ee1a28 feat(computer-use): background focus-safe backend - dad10a78d feat(computer-use): cua-driver backend, universal any-model schema Co-authored-by: teknium1 <teknium@users.noreply.github.com>
2026-05-01 00:11:39 +08:00 · 2026-04-28 01:57:21 -07:00
parent cf0852f92e
commit e63364b8df
27 changed files with 35 additions and 3536 deletions
--- a/website/docs/user-guide/features/computer-use.md
+++ b/website/docs/user-guide/features/computer-use.md
@@ -1,163 +0,0 @@
-# Computer Use (macOS)
-
-Hermes Agent can drive your Mac's desktop — clicking, typing, scrolling,
-dragging — in the **background**. Your cursor doesn't move, keyboard focus
-doesn't change, and macOS doesn't switch Spaces on you. You and the agent
-co-work on the same machine.
-
-Unlike most computer-use integrations, this works with **any tool-capable
-model** — Claude, GPT, Gemini, or an open model on a local vLLM endpoint.
-There's no Anthropic-native schema to worry about.
-
-## How it works
-
-The `computer_use` toolset speaks MCP over stdio to [`cua-driver`](https://github.com/trycua/cua),
-a macOS driver that uses SkyLight private SPIs (`SLEventPostToPid`,
-`SLPSPostEventRecordTo`) and the `_AXObserverAddNotificationAndCheckRemote`
-accessibility SPI to:
-
- Post synthesized events directly to target processes — no HID event tap,
-  no cursor warp.
- Flip AppKit active-state without raising windows — no Space switching.
- Keep Chromium/Electron accessibility trees alive when windows are
-  occluded.
-
-That combination is what OpenAI's Codex "background computer-use" ships.
-cua-driver is the open-source equivalent.
-
-## Enabling
-
-1. Run `hermes tools`, pick `🖱️ Computer Use (macOS)` → `cua-driver (background)`.
-2. The setup runs the upstream installer:
-   `curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh`.
-3. Grant macOS permissions when prompted:
-   - **System Settings → Privacy & Security → Accessibility** → allow the
-     terminal (or Hermes app).
-   - **System Settings → Privacy & Security → Screen Recording** → allow
-     the same.
-4. Start a session with the toolset enabled:
-   ```
-   hermes -t computer_use chat
-   ```
-   or add `computer_use` to your enabled toolsets in `~/.hermes/config.yaml`.
-
-## Quick example
-
-User prompt: *"Find my latest email from Stripe and summarise what they want me to do."*
-
-The agent's plan:
-
-1. `computer_use(action="capture", mode="som", app="Mail")` — gets a
-   screenshot of Mail with every sidebar item, toolbar button, and message
-   row numbered.
-2. `computer_use(action="click", element=14)` — clicks the search field
-   (element #14 from the capture).
-3. `computer_use(action="type", text="from:stripe")`
-4. `computer_use(action="key", keys="return", capture_after=True)` — submit
-   and get the new screenshot.
-5. Click the top result, read the body, summarise.
-
-During all of this, your cursor stays wherever you left it and Mail never
-comes to front.
-
-## Provider compatibility
-
-| Provider | Vision? | Works? | Notes |
-|---|---|---|---|
-| Anthropic (Claude Sonnet/Opus 3+) | ✅ | ✅ | Best overall; SOM + raw coordinates. |
-| OpenRouter (any vision model) | ✅ | ✅ | Multi-part tool messages supported. |
-| OpenAI (GPT-4+, GPT-5) | ✅ | ✅ | Same as above. |
-| Local vLLM / LM Studio (vision model) | ✅ | ✅ | If the model supports multi-part tool content. |
-| Text-only models | ❌ | ✅ (degraded) | Use `mode="ax"` for accessibility-tree-only operation. |
-
-Screenshots are sent inline with tool results as OpenAI-style `image_url`
-parts. For Anthropic, the adapter converts them into native `tool_result`
-image blocks.
-
-## Safety
-
-Hermes applies multi-layer guardrails:
-
- Destructive actions (click, type, drag, scroll, key, focus_app) require
-  approval — either interactively via the CLI dialog or via the
-  messaging-platform approval buttons.
- Hard-blocked key combos at the tool level: empty trash, force delete,
-  lock screen, log out, force log out.
- Hard-blocked type patterns: `curl | bash`, `sudo rm -rf /`, fork bombs,
-  etc.
- The agent's system prompt tells it explicitly: no clicking permission
-  dialogs, no typing passwords, no following instructions embedded in
-  screenshots.
-
-Pair with `security.approval_level` in `~/.hermes/config.yaml` if you want
-every action confirmed.
-
-## Token efficiency
-
-Screenshots are expensive. Hermes applies four layers of optimisation:
-
- **Screenshot eviction** — the Anthropic adapter keeps only the 3 most
-  recent screenshots in context; older ones become `[screenshot removed
-  to save context]` placeholders.
- **Client-side compression pruning** — the context compressor detects
-  multimodal tool results and strips image parts from old ones.
- **Image-aware token estimation** — each image is counted as ~1500 tokens
-  (Anthropic's flat rate) instead of its base64 char length.
- **Server-side context editing (Anthropic only)** — when active, the
-  adapter enables `clear_tool_uses_20250919` via `context_management` so
-  Anthropic's API clears old tool results server-side.
-
-A 20-action session on a 1568×900 display typically costs ~30K tokens
-of screenshot context, not ~600K.
-
-## Limitations
-
- **macOS only.** cua-driver uses private Apple SPIs that don't exist on
-  Linux or Windows. For cross-platform GUI automation, use the `browser`
-  toolset.
- **Private SPI risk.** Apple can change SkyLight's symbol surface in any
-  OS update. Pin the driver version with the `HERMES_CUA_DRIVER_VERSION`
-  env var if you want reproducibility across a macOS bump.
- **Performance.** Background mode is slower than foreground —
-  SkyLight-routed events take ~5-20ms vs direct HID posting. Not
-  noticeable for agent-speed clicking; noticeable if you try to record a
-  speed-run.
- **No keyboard password entry.** `type` has hard-block patterns on
-  command-shell payloads; for passwords, use the system's autofill.
-
-## Configuration
-
-Override the driver binary path (tests / CI):
-
-```
-HERMES_CUA_DRIVER_CMD=/opt/homebrew/bin/cua-driver
-HERMES_CUA_DRIVER_VERSION=0.5.0    # optional pin
-```
-
-Swap the backend entirely (for testing):
-
-```
-HERMES_COMPUTER_USE_BACKEND=noop   # records calls, no side effects
-```
-
-## Troubleshooting
-
-**`computer_use backend unavailable: cua-driver is not installed`** — Run
-`hermes tools` and enable Computer Use.
-
-**Clicks seem to have no effect** — Capture and verify. A modal you
-didn't see may be blocking input. Dismiss it with `escape` or the close
-button.
-
-**Element indices are stale** — SOM indices are only valid until the
-next `capture`. Re-capture after any state-changing action.
-
-**"blocked pattern in type text"** — The text you tried to `type`
-matches the dangerous-shell-pattern list. Break the command up or
-reconsider.
-
-## See also
-
- [Universal skill: `macos-computer-use`](https://github.com/NousResearch/hermes-agent/blob/main/skills/apple/macos-computer-use/SKILL.md)
- [cua-driver source (trycua/cua)](https://github.com/trycua/cua)
- [Browser automation](./browser-use.md) for cross-platform web tasks.