mirror of
https://github.com/NousResearch/hermes-agent.git
synced 2026-04-28 06:51:16 +08:00
* feat(plugins): google_meet — bundled plugin for join+transcribe Meet calls v1 shipping transcribe-only. Spawns headless Chromium via Playwright, joins an explicit https://meet.google.com/ URL, enables live captions, and scrapes them into a transcript file the agent can read across turns. The agent then has the meeting content in context and can do followup work (send recap, file issues, schedule followups) with its regular tools. Surface: - Tools: meet_join, meet_status, meet_transcript, meet_leave, meet_say (meet_say is a v1 stub — returns not-implemented; v2 will wire realtime duplex audio via OpenAI Realtime / Gemini Live + BlackHole / PulseAudio null-sink.) - CLI: hermes meet setup | auth | join | status | transcript | stop - Lifecycle: on_session_end auto-leaves any still-running bot. Safety: - URL regex rejects anything that isn't https://meet.google.com/... - No calendar scanning, no auto-dial, no auto-consent announcement. - Single active meeting per install; a second meet_join leaves the first. - Platform-gated to Linux + macOS (Windows audio routing for v2 untested). - Opt-in: standalone plugin, user must add 'google_meet' to plugins.enabled in config.yaml. Zero core changes. Plugin uses existing register_tool / register_cli_command / register_hook surfaces. 21 new unit tests cover the URL safety gate, transcript dedup + status round-trip, process-manager refusals/start/stop paths, tool-handler JSON shape under each branch, session-end cleanup, and platform-gated register(). * feat(plugins/google_meet): v2 realtime audio + v3 remote node host v2 \u2014 agent speaks in-meeting audio_bridge.py: PulseAudio null-sink (Linux) + BlackHole probe (macOS). On Linux we load pactl module-null-sink + module-virtual-source, track module ids for teardown; Chrome gets PULSE_SOURCE=<virt src> env so its fake mic reads what we write to the sink. macOS just probes BlackHole 2ch and returns its device name \u2014 the plugin refuses to switch the user's default audio input (that would surprise them). realtime/openai_client.py: sync WebSocket client for the OpenAI Realtime API. RealtimeSession.speak(text) sends conversation.item.create + response.create, accumulates response.audio.delta PCM bytes, appends them to a file. RealtimeSpeaker runs a JSONL-queue loop consuming meet_say calls. 'websockets' is an optional dep imported lazily. meet_bot.py: when HERMES_MEET_MODE=realtime, provisions AudioBridge, starts RealtimeSession + speaker thread, spawns paplay to pump PCM into the null-sink, then cleans everything up on SIGTERM. If any realtime setup step fails, falls back cleanly to transcribe mode with an error flagged in status.json. process_manager.enqueue_say(): writes a JSONL line to say_queue.jsonl; refuses when no active meeting or active meeting is transcribe-only. tools.meet_say: real implementation; requires active mode='realtime'. meet_join: adds mode='transcribe'|'realtime' param. v3 \u2014 remote node host node/protocol.py: JSON envelope (type, id, token, payload) + validate. node/registry.py: $HERMES_HOME/workspace/meetings/nodes.json, with resolve() auto-selecting the sole registered node when name is None. node/server.py: NodeServer \u2014 websockets.serve, bearer-token auth, dispatches start_bot/stop/status/transcript/say/ping onto the local process_manager. Token auto-generated + persisted on first run. node/client.py: NodeClient \u2014 short-lived sync WS per RPC, raises RuntimeError on error envelopes, clean API matching the server. node/cli.py: 'hermes meet node {run,list,approve,remove,status,ping}' subtree; wired into the main meet CLI by cli.py so 'hermes meet node' Just Works. tools.py: every meet_* tool accepts node='<name>'|'auto'; when set, routes through NodeClient to the remote bot instead of running locally. Unknown node \u2192 clear 'no registered meet node matches ...' error. cli.py: 'hermes meet join --node my-mac --mode realtime' and 'hermes meet say "..." --node my-mac' route to the node; 'hermes meet node approve <name> <url> <token>' registers one. Tests 21 v1 tests updated (meet_say is no longer a stub; active-record now carries mode). 20 new audio_bridge + realtime tests. 42 new node tests (protocol/registry/server/client/cli). 17 new v1/v2/v3 integration tests at the plugin level covering enqueue_say edge cases, env var passthrough, mode validation, node routing (known/unknown/auto/ambiguous), and argparse wiring for `hermes meet say` + `hermes meet node` + --mode/--node flags. Total: 100 plugin tests + 58 plugin-system tests = 158 passing. E2E verified on Linux with fresh HERMES_HOME: plugin loads, 5 tools register, on_session_end hook wires, 'hermes meet' CLI tree wires including the node subtree, NodeRegistry round-trips, meet_join routes correctly to NodeClient under node='my-mac' with mode='realtime', enqueue_say accepts realtime/rejects transcribe, argparse parses every new flag cleanly. Zero changes to core. All new code lives under plugins/google_meet/. * feat(plugins/google_meet): auto-install, admission detect, mac PCM pump, barge-in, richer status Ready-for-live-test follow-up on PR #16364. Five additions that matter for the first live run on a real Meet, in priority order: 1. hermes meet install [--realtime] [--yes] pip install playwright websockets + python -m playwright install chromium --realtime: installs platform audio deps (pulseaudio-utils on Linux via sudo apt, blackhole-2ch + ffmpeg on macOS via brew). Prompts before sudo/brew unless --yes. Refuses on Windows. Refuses to auto-flip the macOS default input — user still selects BlackHole in System Settings (deliberate; surprise audio rerouting is worse than a manual step). 2. Admission detection _detect_admission(page): Leave-button visible OR caption region attached OR participants list present → we're in-call. _detect_denied(page): 'You can\'t join this video call' / 'You were removed' / 'No one responded to your request' → bail out. HERMES_MEET_LOBBY_TIMEOUT (default 300s) caps how long we sit in the lobby before giving up. in_call stays False until admitted. Status surfaces leaveReason: duration_expired | lobby_timeout | denied | page_closed. 3. macOS PCM pump ffmpeg reads speaker.pcm (24kHz s16le mono) and writes to the BlackHole AVFoundation output via -f audiotoolbox -audio_device_index <N>. _mac_audio_device_index() probes ffmpeg -f avfoundation -list_devices true to resolve 'BlackHole 2ch' → numeric index. Falls back to index 0 on probe failure. Linux paplay pump unchanged. 4. Richer status dict _BotState now tracks realtime, realtimeReady, realtimeDevice, audioBytesOut, lastAudioOutAt, lastBargeInAt, joinAttemptedAt, leaveReason. RealtimeSession.audio_bytes_out / last_audio_out_at counters fold into the status file once a second so meet_status() can show the agent's voice activity in near-real-time. 5. Barge-in RealtimeSession.cancel_response() sends type='response.cancel' over the same WS (lock-guarded so it's safe to call from the caption thread while speak() is reading frames). Handles response.cancelled as a terminal frame type. _looks_like_human_speaker() gates triggers so the bot's own name, 'You', 'Unknown', and blanks don't self-cancel. Called from the caption drain loop: when a new caption arrives attributed to a real participant while rt.session exists, we fire cancel_response() and stamp lastBargeInAt. Tests: 20 new unit tests across _BotState telemetry, barge-in gating, admission/denied probe error handling, cancel_response with and without a connected WS, and `hermes meet install` CLI wiring (flag parsing + end-to-end subprocess.run verification + Linux-already-installed fast path). Total 171 passing across all google_meet test files + the plugin-system regression suite. E2E verified on Linux: plugin loads, all 5 tools register, `hermes meet install --realtime --yes` parses, fresh-bot status.json has every new telemetry key, cancel_response on a disconnected session returns False without raising, barge-in helper gates the bot's own name correctly. Still out of scope (for a future PR, not blocking live test): mic → Realtime duplex (the agent listening to meeting audio via WebRTC), node-host TLS/pairing UX, Windows audio, Meet create+Twilio. Docs updated: SKILL.md now lists the installer subcommand, lobby timeout, barge-in caveat, and the full status-dict reference table. README.md quick-start uses hermes meet install.
294 lines
9.5 KiB
Python
294 lines
9.5 KiB
Python
"""Tests for plugins.google_meet.realtime.openai_client (v2).
|
|
|
|
Uses a scripted fake WebSocket — no network, no API key required.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import base64
|
|
import json
|
|
import sys
|
|
import threading
|
|
import types
|
|
from pathlib import Path
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _isolate_home(tmp_path, monkeypatch):
|
|
hermes_home = tmp_path / ".hermes"
|
|
hermes_home.mkdir()
|
|
monkeypatch.setenv("HERMES_HOME", str(hermes_home))
|
|
yield hermes_home
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Fake WebSocket
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class _FakeWS:
|
|
"""Scripted WS: send() records frames, recv() pops a queue."""
|
|
|
|
def __init__(self, recv_frames: list):
|
|
self.sent: list[dict] = []
|
|
self._recv_q: list = list(recv_frames)
|
|
self.closed = False
|
|
|
|
def send(self, payload):
|
|
# Always accept str payloads — client encodes JSON with json.dumps.
|
|
if isinstance(payload, (bytes, bytearray)):
|
|
payload = payload.decode()
|
|
self.sent.append(json.loads(payload))
|
|
|
|
def recv(self, timeout=None): # noqa: ARG002
|
|
if not self._recv_q:
|
|
raise RuntimeError("fake ws: no more frames")
|
|
frame = self._recv_q.pop(0)
|
|
if isinstance(frame, dict):
|
|
return json.dumps(frame)
|
|
return frame
|
|
|
|
def close(self):
|
|
self.closed = True
|
|
|
|
|
|
def _install_fake_websockets(monkeypatch, fake_ws):
|
|
"""Install a fake ``websockets.sync.client`` module in sys.modules."""
|
|
mod_websockets = types.ModuleType("websockets")
|
|
mod_sync = types.ModuleType("websockets.sync")
|
|
mod_sync_client = types.ModuleType("websockets.sync.client")
|
|
|
|
captured = {"url": None, "headers": None, "kwargs": None}
|
|
|
|
def _connect(url, **kwargs):
|
|
captured["url"] = url
|
|
captured["kwargs"] = kwargs
|
|
captured["headers"] = (
|
|
kwargs.get("additional_headers") or kwargs.get("extra_headers")
|
|
)
|
|
return fake_ws
|
|
|
|
mod_sync_client.connect = _connect
|
|
mod_sync.client = mod_sync_client
|
|
mod_websockets.sync = mod_sync
|
|
|
|
monkeypatch.setitem(sys.modules, "websockets", mod_websockets)
|
|
monkeypatch.setitem(sys.modules, "websockets.sync", mod_sync)
|
|
monkeypatch.setitem(sys.modules, "websockets.sync.client", mod_sync_client)
|
|
return captured
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# connect()
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_connect_sends_session_update_with_voice_and_instructions(monkeypatch):
|
|
from plugins.google_meet.realtime.openai_client import RealtimeSession
|
|
|
|
ws = _FakeWS(recv_frames=[])
|
|
captured = _install_fake_websockets(monkeypatch, ws)
|
|
|
|
sess = RealtimeSession(
|
|
api_key="sk-test",
|
|
model="gpt-realtime",
|
|
voice="verse",
|
|
instructions="Be brief.",
|
|
)
|
|
sess.connect()
|
|
|
|
# Auth + beta headers set.
|
|
assert captured["url"].startswith("wss://api.openai.com/v1/realtime")
|
|
assert "model=gpt-realtime" in captured["url"]
|
|
headers = captured["headers"] or []
|
|
hdict = dict(headers)
|
|
assert hdict.get("Authorization") == "Bearer sk-test"
|
|
assert hdict.get("OpenAI-Beta") == "realtime=v1"
|
|
|
|
# First frame sent must be session.update with the right shape.
|
|
assert len(ws.sent) == 1
|
|
update = ws.sent[0]
|
|
assert update["type"] == "session.update"
|
|
s = update["session"]
|
|
assert s["voice"] == "verse"
|
|
assert s["instructions"] == "Be brief."
|
|
assert set(s["modalities"]) == {"audio", "text"}
|
|
assert s["output_audio_format"] == "pcm16"
|
|
assert s["input_audio_format"] == "pcm16"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# speak()
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_speak_sends_create_and_response_and_writes_audio(monkeypatch, tmp_path):
|
|
from plugins.google_meet.realtime.openai_client import RealtimeSession
|
|
|
|
audio_bytes = b"\x01\x02\x03\x04PCM!"
|
|
b64 = base64.b64encode(audio_bytes).decode()
|
|
|
|
recv_frames = [
|
|
{"type": "response.created"},
|
|
{"type": "response.audio.delta", "delta": b64},
|
|
{"type": "response.audio.delta", "delta": base64.b64encode(b"more").decode()},
|
|
{"type": "response.done"},
|
|
]
|
|
ws = _FakeWS(recv_frames=recv_frames)
|
|
_install_fake_websockets(monkeypatch, ws)
|
|
|
|
sink = tmp_path / "out.pcm"
|
|
sess = RealtimeSession(api_key="sk-test", audio_sink_path=sink)
|
|
sess.connect()
|
|
result = sess.speak("Hello everyone.")
|
|
|
|
# Frames sent after session.update: conversation.item.create then response.create.
|
|
types_sent = [f["type"] for f in ws.sent]
|
|
assert types_sent == ["session.update", "conversation.item.create", "response.create"]
|
|
|
|
item = ws.sent[1]["item"]
|
|
assert item["role"] == "user"
|
|
assert item["content"][0]["type"] == "input_text"
|
|
assert item["content"][0]["text"] == "Hello everyone."
|
|
|
|
resp = ws.sent[2]["response"]
|
|
assert resp["modalities"] == ["audio"]
|
|
|
|
# Audio file got decoded + appended bytes.
|
|
data = sink.read_bytes()
|
|
assert data == audio_bytes + b"more"
|
|
assert result["ok"] is True
|
|
assert result["bytes_written"] == len(audio_bytes) + len(b"more")
|
|
assert result["duration_ms"] >= 0.0
|
|
|
|
|
|
def test_speak_raises_on_error_frame(monkeypatch, tmp_path):
|
|
from plugins.google_meet.realtime.openai_client import RealtimeSession
|
|
|
|
ws = _FakeWS(recv_frames=[
|
|
{"type": "response.created"},
|
|
{"type": "error", "error": {"message": "bad juju"}},
|
|
])
|
|
_install_fake_websockets(monkeypatch, ws)
|
|
|
|
sess = RealtimeSession(api_key="sk-test", audio_sink_path=tmp_path / "o.pcm")
|
|
sess.connect()
|
|
with pytest.raises(RuntimeError, match="bad juju"):
|
|
sess.speak("hi")
|
|
|
|
|
|
def test_speak_without_connect_raises(monkeypatch):
|
|
from plugins.google_meet.realtime.openai_client import RealtimeSession
|
|
|
|
sess = RealtimeSession(api_key="sk-test")
|
|
with pytest.raises(RuntimeError, match="connect"):
|
|
sess.speak("hi")
|
|
|
|
|
|
def test_close_is_idempotent_and_closes_ws(monkeypatch):
|
|
from plugins.google_meet.realtime.openai_client import RealtimeSession
|
|
|
|
ws = _FakeWS(recv_frames=[])
|
|
_install_fake_websockets(monkeypatch, ws)
|
|
|
|
sess = RealtimeSession(api_key="sk-test")
|
|
sess.connect()
|
|
sess.close()
|
|
assert ws.closed is True
|
|
# Second close is a no-op.
|
|
sess.close()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# websockets dependency missing
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_connect_raises_clean_error_when_websockets_missing(monkeypatch):
|
|
from plugins.google_meet.realtime.openai_client import RealtimeSession
|
|
|
|
# Make `import websockets.sync.client` fail.
|
|
monkeypatch.setitem(sys.modules, "websockets", None)
|
|
monkeypatch.setitem(sys.modules, "websockets.sync", None)
|
|
monkeypatch.setitem(sys.modules, "websockets.sync.client", None)
|
|
|
|
sess = RealtimeSession(api_key="sk-test")
|
|
with pytest.raises(RuntimeError, match="pip install websockets"):
|
|
sess.connect()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# RealtimeSpeaker
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class _StubSession:
|
|
def __init__(self):
|
|
self.spoken: list[str] = []
|
|
|
|
def speak(self, text, timeout=30.0): # noqa: ARG002
|
|
self.spoken.append(text)
|
|
return {"ok": True, "bytes_written": len(text), "duration_ms": 1.0}
|
|
|
|
|
|
def test_speaker_run_until_stopped_processes_queue(tmp_path):
|
|
from plugins.google_meet.realtime.openai_client import RealtimeSpeaker
|
|
|
|
queue = tmp_path / "queue.jsonl"
|
|
processed = tmp_path / "processed.jsonl"
|
|
queue.write_text(
|
|
json.dumps({"id": "a", "text": "hello one"}) + "\n"
|
|
+ json.dumps({"id": "b", "text": "hello two"}) + "\n"
|
|
)
|
|
|
|
stub = _StubSession()
|
|
speaker = RealtimeSpeaker(stub, queue_path=queue, processed_path=processed)
|
|
|
|
# Stop once the queue is empty.
|
|
def _stop():
|
|
return queue.exists() and queue.read_text().strip() == ""
|
|
|
|
speaker.run_until_stopped(_stop, poll_interval=0.01)
|
|
|
|
assert stub.spoken == ["hello one", "hello two"]
|
|
|
|
# Processed file has both entries, in order.
|
|
lines = [json.loads(l) for l in processed.read_text().splitlines() if l.strip()]
|
|
assert [l["id"] for l in lines] == ["a", "b"]
|
|
assert all(l["result"]["ok"] for l in lines)
|
|
|
|
# Queue is empty (possibly empty string) after processing.
|
|
assert queue.read_text().strip() == ""
|
|
|
|
|
|
def test_speaker_exits_immediately_when_stop_fn_true(tmp_path):
|
|
from plugins.google_meet.realtime.openai_client import RealtimeSpeaker
|
|
|
|
queue = tmp_path / "q.jsonl"
|
|
queue.write_text(json.dumps({"id": "x", "text": "never spoken"}) + "\n")
|
|
|
|
stub = _StubSession()
|
|
speaker = RealtimeSpeaker(stub, queue_path=queue)
|
|
speaker.run_until_stopped(lambda: True, poll_interval=0.01)
|
|
assert stub.spoken == []
|
|
|
|
|
|
def test_speaker_drops_line_without_processed_path_when_none(tmp_path):
|
|
from plugins.google_meet.realtime.openai_client import RealtimeSpeaker
|
|
|
|
queue = tmp_path / "q.jsonl"
|
|
queue.write_text(json.dumps({"id": "only", "text": "once"}) + "\n")
|
|
|
|
stub = _StubSession()
|
|
speaker = RealtimeSpeaker(stub, queue_path=queue, processed_path=None)
|
|
|
|
def _stop():
|
|
return queue.read_text().strip() == ""
|
|
|
|
speaker.run_until_stopped(_stop, poll_interval=0.01)
|
|
assert stub.spoken == ["once"]
|
|
assert queue.read_text().strip() == ""
|