Files
hermes-agent/plugins/google_meet/tools.py
Teknium df3c9593f8 feat(plugins): google_meet \u2014 join, transcribe, speak, follow up (#16364)
* feat(plugins): google_meet — bundled plugin for join+transcribe Meet calls

v1 shipping transcribe-only. Spawns headless Chromium via Playwright,
joins an explicit https://meet.google.com/ URL, enables live captions,
and scrapes them into a transcript file the agent can read across turns.
The agent then has the meeting content in context and can do followup
work (send recap, file issues, schedule followups) with its regular tools.

Surface:
  - Tools: meet_join, meet_status, meet_transcript, meet_leave, meet_say
    (meet_say is a v1 stub — returns not-implemented; v2 will wire
    realtime duplex audio via OpenAI Realtime / Gemini Live +
    BlackHole / PulseAudio null-sink.)
  - CLI: hermes meet setup | auth | join | status | transcript | stop
  - Lifecycle: on_session_end auto-leaves any still-running bot.

Safety:
  - URL regex rejects anything that isn't https://meet.google.com/...
  - No calendar scanning, no auto-dial, no auto-consent announcement.
  - Single active meeting per install; a second meet_join leaves the first.
  - Platform-gated to Linux + macOS (Windows audio routing for v2 untested).
  - Opt-in: standalone plugin, user must add 'google_meet' to
    plugins.enabled in config.yaml.

Zero core changes. Plugin uses existing register_tool /
register_cli_command / register_hook surfaces. 21 new unit tests cover the
URL safety gate, transcript dedup + status round-trip, process-manager
refusals/start/stop paths, tool-handler JSON shape under each branch,
session-end cleanup, and platform-gated register().

* feat(plugins/google_meet): v2 realtime audio + v3 remote node host

v2 \u2014 agent speaks in-meeting
  audio_bridge.py: PulseAudio null-sink (Linux) + BlackHole probe (macOS).
    On Linux we load pactl module-null-sink + module-virtual-source, track
    module ids for teardown; Chrome gets PULSE_SOURCE=<virt src> env so its
    fake mic reads what we write to the sink. macOS just probes BlackHole
    2ch and returns its device name \u2014 the plugin refuses to switch the
    user's default audio input (that would surprise them).
  realtime/openai_client.py: sync WebSocket client for the OpenAI Realtime
    API. RealtimeSession.speak(text) sends conversation.item.create +
    response.create, accumulates response.audio.delta PCM bytes, appends
    them to a file. RealtimeSpeaker runs a JSONL-queue loop consuming
    meet_say calls. 'websockets' is an optional dep imported lazily.
  meet_bot.py: when HERMES_MEET_MODE=realtime, provisions AudioBridge,
    starts RealtimeSession + speaker thread, spawns paplay to pump PCM
    into the null-sink, then cleans everything up on SIGTERM. If any
    realtime setup step fails, falls back cleanly to transcribe mode
    with an error flagged in status.json.
  process_manager.enqueue_say(): writes a JSONL line to say_queue.jsonl;
    refuses when no active meeting or active meeting is transcribe-only.
  tools.meet_say: real implementation; requires active mode='realtime'.
  meet_join: adds mode='transcribe'|'realtime' param.

v3 \u2014 remote node host
  node/protocol.py: JSON envelope (type, id, token, payload) + validate.
  node/registry.py: $HERMES_HOME/workspace/meetings/nodes.json, with
    resolve() auto-selecting the sole registered node when name is None.
  node/server.py: NodeServer \u2014 websockets.serve, bearer-token auth,
    dispatches start_bot/stop/status/transcript/say/ping onto the local
    process_manager. Token auto-generated + persisted on first run.
  node/client.py: NodeClient \u2014 short-lived sync WS per RPC, raises
    RuntimeError on error envelopes, clean API matching the server.
  node/cli.py: 'hermes meet node {run,list,approve,remove,status,ping}'
    subtree; wired into the main meet CLI by cli.py so 'hermes meet node'
    Just Works.
  tools.py: every meet_* tool accepts node='<name>'|'auto'; when set,
    routes through NodeClient to the remote bot instead of running
    locally. Unknown node \u2192 clear 'no registered meet node matches ...'
    error.
  cli.py: 'hermes meet join --node my-mac --mode realtime' and
    'hermes meet say "..." --node my-mac' route to the node; 'hermes
    meet node approve <name> <url> <token>' registers one.

Tests
  21 v1 tests updated (meet_say is no longer a stub; active-record now
    carries mode).
  20 new audio_bridge + realtime tests.
  42 new node tests (protocol/registry/server/client/cli).
  17 new v1/v2/v3 integration tests at the plugin level covering
    enqueue_say edge cases, env var passthrough, mode validation, node
    routing (known/unknown/auto/ambiguous), and argparse wiring for
    `hermes meet say` + `hermes meet node` + --mode/--node flags.
  Total: 100 plugin tests + 58 plugin-system tests = 158 passing.

E2E verified on Linux with fresh HERMES_HOME: plugin loads, 5 tools
register, on_session_end hook wires, 'hermes meet' CLI tree wires
including the node subtree, NodeRegistry round-trips, meet_join routes
correctly to NodeClient under node='my-mac' with mode='realtime',
enqueue_say accepts realtime/rejects transcribe, argparse parses every
new flag cleanly.

Zero changes to core. All new code lives under plugins/google_meet/.

* feat(plugins/google_meet): auto-install, admission detect, mac PCM pump, barge-in, richer status

Ready-for-live-test follow-up on PR #16364. Five additions that matter for
the first live run on a real Meet, in priority order:

1. hermes meet install [--realtime] [--yes]
   pip install playwright websockets + python -m playwright install chromium
   --realtime: installs platform audio deps (pulseaudio-utils on Linux via
   sudo apt, blackhole-2ch + ffmpeg on macOS via brew). Prompts before
   sudo/brew unless --yes. Refuses on Windows. Refuses to auto-flip the
   macOS default input — user still selects BlackHole in System Settings
   (deliberate; surprise audio rerouting is worse than a manual step).

2. Admission detection
   _detect_admission(page): Leave-button visible OR caption region
   attached OR participants list present → we're in-call.
   _detect_denied(page): 'You can\'t join this video call' / 'You were
   removed' / 'No one responded to your request' → bail out.
   HERMES_MEET_LOBBY_TIMEOUT (default 300s) caps how long we sit in
   the lobby before giving up. in_call stays False until admitted.
   Status surfaces leaveReason: duration_expired | lobby_timeout |
   denied | page_closed.

3. macOS PCM pump
   ffmpeg reads speaker.pcm (24kHz s16le mono) and writes to the
   BlackHole AVFoundation output via -f audiotoolbox
   -audio_device_index <N>. _mac_audio_device_index() probes
   ffmpeg -f avfoundation -list_devices true to resolve 'BlackHole 2ch'
   → numeric index. Falls back to index 0 on probe failure. Linux
   paplay pump unchanged.

4. Richer status dict
   _BotState now tracks realtime, realtimeReady, realtimeDevice,
   audioBytesOut, lastAudioOutAt, lastBargeInAt, joinAttemptedAt,
   leaveReason. RealtimeSession.audio_bytes_out / last_audio_out_at
   counters fold into the status file once a second so meet_status()
   can show the agent's voice activity in near-real-time.

5. Barge-in
   RealtimeSession.cancel_response() sends type='response.cancel' over
   the same WS (lock-guarded so it's safe to call from the caption
   thread while speak() is reading frames). Handles response.cancelled
   as a terminal frame type. _looks_like_human_speaker() gates triggers
   so the bot's own name, 'You', 'Unknown', and blanks don't self-cancel.
   Called from the caption drain loop: when a new caption arrives
   attributed to a real participant while rt.session exists, we fire
   cancel_response() and stamp lastBargeInAt.

Tests: 20 new unit tests across _BotState telemetry, barge-in gating,
admission/denied probe error handling, cancel_response with and without
a connected WS, and `hermes meet install` CLI wiring (flag parsing +
end-to-end subprocess.run verification + Linux-already-installed fast
path). Total 171 passing across all google_meet test files + the
plugin-system regression suite.

E2E verified on Linux: plugin loads, all 5 tools register,
`hermes meet install --realtime --yes` parses, fresh-bot status.json
has every new telemetry key, cancel_response on a disconnected session
returns False without raising, barge-in helper gates the bot's own
name correctly.

Still out of scope (for a future PR, not blocking live test):
mic → Realtime duplex (the agent listening to meeting audio via
WebRTC), node-host TLS/pairing UX, Windows audio, Meet create+Twilio.

Docs updated: SKILL.md now lists the installer subcommand, lobby
timeout, barge-in caveat, and the full status-dict reference table.
README.md quick-start uses hermes meet install.
2026-04-27 06:22:25 -07:00

349 lines
12 KiB
Python

"""Agent-facing tools for the google_meet plugin.
Tools:
meet_join — join a Google Meet URL (spawns Playwright bot locally
OR on a remote node host via node=<name>)
meet_status — report bot liveness + transcript progress
meet_transcript — read the current transcript (optional last-N)
meet_leave — signal the bot to leave cleanly
meet_say — (v2) speak text through the realtime audio bridge.
Requires the active meeting to have been joined with
mode='realtime'.
"""
from __future__ import annotations
import json
from typing import Any, Dict, Optional
from plugins.google_meet import process_manager as pm
# ---------------------------------------------------------------------------
# Runtime gate
# ---------------------------------------------------------------------------
def check_meet_requirements() -> bool:
"""Return True when the plugin can actually run LOCALLY.
Gates on:
* Python ``playwright`` package importable
* the plugin being on a supported platform (Linux or macOS)
Note: remote-node operation (``node=<name>``) only needs the
``websockets`` dep on the gateway side — Chromium lives on the node.
But the plugin-level gate keeps the v1 semantics; individual tool
handlers relax the requirement when a node is addressed.
"""
import platform as _p
if _p.system().lower() not in ("linux", "darwin"):
return False
try:
import playwright # noqa: F401
except ImportError:
return False
return True
# ---------------------------------------------------------------------------
# Node client helper
# ---------------------------------------------------------------------------
def _resolve_node_client(node: Optional[str]):
"""Return (NodeClient, node_name) for *node*, or (None, None) to run local.
Raises RuntimeError with a readable message if the node is named but
unresolvable, so the handler can surface a clear error to the agent.
"""
if node is None or node == "":
return None, None
from plugins.google_meet.node.registry import NodeRegistry
from plugins.google_meet.node.client import NodeClient
reg = NodeRegistry()
entry = reg.resolve(node if node != "auto" else None)
if entry is None:
raise RuntimeError(
f"no registered meet node matches {node!r}"
"run `hermes meet node approve <name> <url> <token>` first"
)
client = NodeClient(url=entry["url"], token=entry["token"])
return client, entry.get("name")
# ---------------------------------------------------------------------------
# Schemas
# ---------------------------------------------------------------------------
MEET_JOIN_SCHEMA: Dict[str, Any] = {
"name": "meet_join",
"description": (
"Join a Google Meet call and start scraping live captions into a "
"transcript file. Only meet.google.com URLs are accepted; no calendar "
"scanning, no auto-dial. Spawns a headless Chromium subprocess that "
"runs in parallel with the agent loop — returns immediately. Poll "
"with meet_status and read captions with meet_transcript. Reminder "
"to the agent: you should announce yourself in the meeting (there is "
"no automatic consent announcement)."
),
"parameters": {
"type": "object",
"properties": {
"url": {
"type": "string",
"description": (
"Full https://meet.google.com/... URL. Required."
),
},
"mode": {
"type": "string",
"enum": ["transcribe", "realtime"],
"description": (
"transcribe (default): listen-only, scrape captions. "
"realtime: also enable agent speech via meet_say "
"(requires OpenAI Realtime key + platform audio bridge)."
),
},
"guest_name": {
"type": "string",
"description": (
"Display name to use when joining as guest. Defaults to "
"'Hermes Agent'."
),
},
"duration": {
"type": "string",
"description": (
"Optional max duration before auto-leave (e.g. '30m', "
"'2h', '90s'). Omit to stay until meet_leave is called."
),
},
"headed": {
"type": "boolean",
"description": (
"Run Chromium headed instead of headless (debug only). "
"Default false."
),
},
"node": {
"type": "string",
"description": (
"Name of a registered remote node to run the bot on "
"(useful when the gateway runs on a headless Linux box "
"but the user's Chrome with a signed-in Google profile "
"lives on their Mac). Pass 'auto' to use the single "
"registered node. Default: run locally. Nodes are "
"approved via `hermes meet node approve`."
),
},
},
"required": ["url"],
"additionalProperties": False,
},
}
MEET_STATUS_SCHEMA: Dict[str, Any] = {
"name": "meet_status",
"description": (
"Report the current Meet session state — whether the bot is alive, "
"has joined, is sitting in the lobby, number of transcript lines "
"captured, and last-caption timestamp."
),
"parameters": {
"type": "object",
"properties": {
"node": {"type": "string"},
},
"additionalProperties": False,
},
}
MEET_TRANSCRIPT_SCHEMA: Dict[str, Any] = {
"name": "meet_transcript",
"description": (
"Read the scraped transcript for the active Meet session. Returns "
"full transcript unless 'last' is set, in which case returns the last "
"N lines only."
),
"parameters": {
"type": "object",
"properties": {
"last": {
"type": "integer",
"description": (
"Optional: return only the last N caption lines. Useful "
"for polling during a meeting without re-reading the "
"whole transcript."
),
"minimum": 1,
},
"node": {"type": "string"},
},
"additionalProperties": False,
},
}
MEET_LEAVE_SCHEMA: Dict[str, Any] = {
"name": "meet_leave",
"description": (
"Leave the active Meet call cleanly, stop caption scraping, and "
"finalize the transcript file. Safe to call when no meeting is "
"active — returns ok=false with a reason."
),
"parameters": {
"type": "object",
"properties": {
"node": {"type": "string"},
},
"additionalProperties": False,
},
}
MEET_SAY_SCHEMA: Dict[str, Any] = {
"name": "meet_say",
"description": (
"Speak text into the active Meet call. Requires the active meeting "
"to have been joined with mode='realtime'. The text is queued to "
"the bot's OpenAI Realtime session; the generated audio is streamed "
"into Chrome's fake microphone via a virtual audio device "
"(PulseAudio null-sink on Linux, BlackHole on macOS). Returns "
"immediately — the actual speech lags by a couple of seconds."
),
"parameters": {
"type": "object",
"properties": {
"text": {"type": "string", "description": "Text to speak."},
"node": {"type": "string"},
},
"required": ["text"],
"additionalProperties": False,
},
}
# ---------------------------------------------------------------------------
# Handlers
# ---------------------------------------------------------------------------
def _json(obj: Any) -> str:
return json.dumps(obj, ensure_ascii=False)
def _err(msg: str, **extra) -> str:
return _json({"success": False, "error": msg, **extra})
def handle_meet_join(args: Dict[str, Any], **_kw) -> str:
url = (args.get("url") or "").strip()
if not url:
return _err("url is required")
mode = (args.get("mode") or "transcribe").strip().lower()
if mode not in ("transcribe", "realtime"):
return _err(f"mode must be 'transcribe' or 'realtime' (got {mode!r})")
node = args.get("node")
try:
client, node_name = _resolve_node_client(node)
except RuntimeError as e:
return _err(str(e))
if client is not None:
# Remote path — delegate to the node host.
try:
res = client.start_bot(
url=url,
guest_name=str(args.get("guest_name") or "Hermes Agent"),
duration=str(args.get("duration")) if args.get("duration") else None,
headed=bool(args.get("headed", False)),
mode=mode,
)
return _json({"success": bool(res.get("ok")), "node": node_name, **res})
except Exception as e:
return _err(f"remote node start_bot failed: {e}", node=node_name)
# Local path — same as v1, with v2 params.
if not check_meet_requirements():
return _err(
"google_meet plugin prerequisites missing — install with "
"`pip install playwright && python -m playwright install "
"chromium`. Plugin is supported on Linux and macOS only."
)
res = pm.start(
url=url,
headed=bool(args.get("headed", False)),
guest_name=str(args.get("guest_name") or "Hermes Agent"),
duration=str(args.get("duration")) if args.get("duration") else None,
mode=mode,
)
return _json({"success": bool(res.get("ok")), **res})
def handle_meet_status(args: Dict[str, Any], **_kw) -> str:
try:
client, node_name = _resolve_node_client(args.get("node"))
except RuntimeError as e:
return _err(str(e))
if client is not None:
try:
res = client.status()
return _json({"success": bool(res.get("ok")), "node": node_name, **res})
except Exception as e:
return _err(f"remote node status failed: {e}", node=node_name)
res = pm.status()
return _json({"success": bool(res.get("ok")), **res})
def handle_meet_transcript(args: Dict[str, Any], **_kw) -> str:
last = args.get("last")
try:
last_i = int(last) if last is not None else None
if last_i is not None and last_i < 1:
last_i = None
except (TypeError, ValueError):
last_i = None
try:
client, node_name = _resolve_node_client(args.get("node"))
except RuntimeError as e:
return _err(str(e))
if client is not None:
try:
res = client.transcript(last=last_i)
return _json({"success": bool(res.get("ok")), "node": node_name, **res})
except Exception as e:
return _err(f"remote node transcript failed: {e}", node=node_name)
res = pm.transcript(last=last_i)
return _json({"success": bool(res.get("ok")), **res})
def handle_meet_leave(args: Dict[str, Any], **_kw) -> str:
try:
client, node_name = _resolve_node_client(args.get("node"))
except RuntimeError as e:
return _err(str(e))
if client is not None:
try:
res = client.stop()
return _json({"success": bool(res.get("ok")), "node": node_name, **res})
except Exception as e:
return _err(f"remote node stop failed: {e}", node=node_name)
res = pm.stop(reason="agent called meet_leave")
return _json({"success": bool(res.get("ok")), **res})
def handle_meet_say(args: Dict[str, Any], **_kw) -> str:
text = (args.get("text") or "").strip()
if not text:
return _err("text is required")
try:
client, node_name = _resolve_node_client(args.get("node"))
except RuntimeError as e:
return _err(str(e))
if client is not None:
try:
res = client.say(text)
return _json({"success": bool(res.get("ok")), "node": node_name, **res})
except Exception as e:
return _err(f"remote node say failed: {e}", node=node_name)
res = pm.enqueue_say(text)
return _json({"success": bool(res.get("ok")), **res})