Add node-inspect-debugger and python-debugpy skills

Two new skills under skills/software-development/ for real breakpoint-driven debugging from the terminal: - node-inspect-debugger: node --inspect / --inspect-brk, node inspect REPL, CDP scripting via chrome-remote-interface, attaching to running Node processes (SIGUSR1), ui-tui-specific recipes, Vitest under debugger, CPU profiles + heap snapshots. - python-debugpy: pdb quick reference, breakpoint() workflow, pytest --pdb (with xdist caveat for scripts/run_tests.sh), post-mortem, debugpy for remote/attach, remote-pdb as the agent-friendly alternative to DAP, recipes for tui_gateway/_SlashWorker/subprocess debugging.
2026-04-28 06:51:16 +08:00 · 2026-04-26 17:10:11 -05:00
parent 82f842277e
commit bbf96008dc
2 changed files with 692 additions and 0 deletions
--- a/skills/software-development/node-inspect-debugger/SKILL.md
+++ b/skills/software-development/node-inspect-debugger/SKILL.md
@@ -0,0 +1,318 @@
+---
+name: node-inspect-debugger
+description: Use when debugging Node.js code (ui-tui, tui_gateway child processes, any Node script/test) with real breakpoints, stepping, scope inspection, and expression evaluation. Drives `node --inspect` via the Chrome DevTools Protocol from the terminal — no browser required.
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+metadata:
+  hermes:
+    tags: [debugging, nodejs, node-inspect, cdp, breakpoints, ui-tui]
+    related_skills: [systematic-debugging, python-debugpy, debugging-hermes-tui-commands]
+---
+
+# Node.js Inspect Debugger
+
+## Overview
+
+When `console.log` isn't enough, drive Node's built-in V8 inspector programmatically from the terminal. You get real breakpoints, step in/over/out, call-stack walking, local/closure scope dumps, and arbitrary expression evaluation in the paused frame.
+
+Two tools, pick one:
+
+- **`node inspect`** — built-in, zero install, CLI REPL. Best for quick poking.
+- **`ndb` / CDP via `chrome-remote-interface`** — scriptable from Node/Python; best when you want to automate many breakpoints, collect state across runs, or debug non-interactively from an agent loop.
+
+**Prefer `node inspect` first.** It's always available and the REPL is fast.
+
+## When to Use
+
+- A Node test fails and you need to see intermediate state
+- ui-tui crashes or behaves wrong and you want to inspect React/Ink state pre-render
+- tui_gateway child processes (`_SlashWorker`, PTY bridge workers) misbehave
+- You need to inspect a value in a closure that `console.log` can't reach without patching
+- Perf: attach to a running process to capture a CPU profile or heap snapshot
+
+**Don't use for:** things `console.log` solves in under a minute. Breakpoint-driven debugging is heavier; use it when the payoff is real.
+
+## Quick Reference: `node inspect` REPL
+
+Launch paused on first line:
+
+```bash
+node inspect path/to/script.js
+# or with tsx
+node --inspect-brk $(which tsx) path/to/script.ts
+```
+
+The `debug>` prompt accepts:
+
+| Command | Action |
+|---|---|
+| `c` or `cont` | continue |
+| `n` or `next` | step over |
+| `s` or `step` | step into |
+| `o` or `out` | step out |
+| `pause` | pause running code |
+| `sb('file.js', 42)` | set breakpoint at file.js line 42 |
+| `sb(42)` | set breakpoint at line 42 of current file |
+| `sb('functionName')` | break when function is called |
+| `cb('file.js', 42)` | clear breakpoint |
+| `breakpoints` | list all breakpoints |
+| `bt` | backtrace (call stack) |
+| `list(5)` | show 5 lines of source around current position |
+| `watch('expr')` | evaluate expr on every pause |
+| `watchers` | show watched expressions |
+| `repl` | drop into REPL in current scope (Ctrl+C to exit REPL) |
+| `exec expr` | evaluate expression once |
+| `restart` | restart script |
+| `kill` | kill the script |
+| `.exit` | quit debugger |
+
+**In the `repl` sub-mode:** type any JS expression, including access to locals/closure variables. `Ctrl+C` exits back to `debug>`.
+
+## Attaching to a Running Process
+
+When the process is already running (e.g. a long-lived dev server or the TUI gateway):
+
+```bash
+# 1. Send SIGUSR1 to enable the inspector on an existing process
+kill -SIGUSR1 <pid>
+# Node prints: Debugger listening on ws://127.0.0.1:9229/<uuid>
+
+# 2. Attach the debugger CLI
+node inspect -p <pid>
+# or by URL
+node inspect ws://127.0.0.1:9229/<uuid>
+```
+
+To start a process with the inspector from the beginning:
+
+```bash
+node --inspect script.js           # listen on 127.0.0.1:9229, keep running
+node --inspect-brk script.js       # listen AND pause on first line
+node --inspect=0.0.0.0:9230 script.js   # custom host:port
+```
+
+For TypeScript via tsx:
+
+```bash
+node --inspect-brk --import tsx script.ts
+# or older tsx
+node --inspect-brk -r tsx/cjs script.ts
+```
+
+## Programmatic CDP (scripting from terminal)
+
+When you want to automate — set many breakpoints, capture scope state, script a repro — use `chrome-remote-interface`:
+
+```bash
+npm i -g chrome-remote-interface        # or project-local
+# Start your target:
+node --inspect-brk=9229 target.js &
+```
+
+Driver script (save as `/tmp/cdp-debug.js`):
+
+```javascript
+const CDP = require('chrome-remote-interface');
+
+(async () => {
+  const client = await CDP({ port: 9229 });
+  const { Debugger, Runtime } = client;
+
+  Debugger.paused(async ({ callFrames, reason }) => {
+    const top = callFrames[0];
+    console.log(`PAUSED: ${reason} @ ${top.url}:${top.location.lineNumber + 1}`);
+
+    // Walk scopes for locals
+    for (const scope of top.scopeChain) {
+      if (scope.type === 'local' || scope.type === 'closure') {
+        const { result } = await Runtime.getProperties({
+          objectId: scope.object.objectId,
+          ownProperties: true,
+        });
+        for (const p of result) {
+          console.log(`  ${scope.type}.${p.name} =`, p.value?.value ?? p.value?.description);
+        }
+      }
+    }
+
+    // Evaluate an expression in the paused frame
+    const { result } = await Debugger.evaluateOnCallFrame({
+      callFrameId: top.callFrameId,
+      expression: 'typeof state !== "undefined" ? JSON.stringify(state) : "n/a"',
+    });
+    console.log('state =', result.value ?? result.description);
+
+    await Debugger.resume();
+  });
+
+  await Runtime.enable();
+  await Debugger.enable();
+
+  // Set a breakpoint by URL regex + line
+  await Debugger.setBreakpointByUrl({
+    urlRegex: '.*app\\.tsx$',
+    lineNumber: 119,       // 0-indexed
+    columnNumber: 0,
+  });
+
+  await Runtime.runIfWaitingForDebugger();
+})();
+```
+
+Run it:
+
+```bash
+node /tmp/cdp-debug.js
+```
+
+Hermes-specific note: `chrome-remote-interface` is NOT in `ui-tui/package.json`. Install it to a throwaway location if you don't want to dirty the project:
+
+```bash
+mkdir -p /tmp/cdp-tools && cd /tmp/cdp-tools && npm i chrome-remote-interface
+NODE_PATH=/tmp/cdp-tools/node_modules node /tmp/cdp-debug.js
+```
+
+## Debugging Hermes ui-tui
+
+The TUI is built Ink + tsx. Two common scenarios:
+
+### Debugging a single Ink component under dev
+
+`ui-tui/package.json` has `npm run dev` (tsx --watch). Add `--inspect-brk` by running tsx directly:
+
+```bash
+cd /home/bb/hermes-agent/ui-tui
+npm run build    # produce dist/ once so transpile isn't needed on first load
+node --inspect-brk dist/entry.js
+# In another terminal:
+node inspect -p <node pid>
+```
+
+Then inside `debug>`:
+
+```
+sb('dist/app.js', 220)     # or wherever the suspect render is
+cont
+```
+
+When it pauses, `repl` → inspect `props`, state refs, `useInput` handler values, etc.
+
+### Debugging a running `hermes --tui`
+
+The TUI spawns Node from the Python CLI. Easiest path:
+
+```bash
+# 1. Launch TUI
+hermes --tui &
+TUI_PID=$(pgrep -f 'ui-tui/dist/entry' | head -1)
+
+# 2. Enable inspector on that Node PID
+kill -SIGUSR1 "$TUI_PID"
+
+# 3. Find the WS URL
+curl -s http://127.0.0.1:9229/json/list | jq -r '.[0].webSocketDebuggerUrl'
+
+# 4. Attach
+node inspect ws://127.0.0.1:9229/<uuid>
+```
+
+Interacting with the TUI (typing in its window) continues to advance execution; your debugger can pause it on a breakpoint at any `sb(...)`.
+
+### Debugging `_SlashWorker` / PTY child processes
+
+Those are Python, not Node — use the `python-debugpy` skill for them. Only Node portions (Ink UI, tui_gateway client, tsx-run tests under `ui-tui/`) use this skill.
+
+## Running Vitest Tests Under the Debugger
+
+```bash
+cd /home/bb/hermes-agent/ui-tui
+# Run a single test file paused on entry
+node --inspect-brk ./node_modules/vitest/vitest.mjs run --no-file-parallelism src/app/foo.test.tsx
+```
+
+In another terminal: `node inspect -p <pid>`, then `sb('src/app/foo.tsx', 42)`, `cont`.
+
+Use `--no-file-parallelism` (vitest) or `--runInBand` (jest) so only one worker exists — debugging a pool is painful.
+
+## Heap Snapshots & CPU Profiles (Non-interactive)
+
+From the CDP driver above, swap Debugger for `HeapProfiler` / `Profiler`:
+
+```javascript
+// CPU profile for 5 seconds
+await client.Profiler.enable();
+await client.Profiler.start();
+await new Promise(r => setTimeout(r, 5000));
+const { profile } = await client.Profiler.stop();
+require('fs').writeFileSync('/tmp/cpu.cpuprofile', JSON.stringify(profile));
+// Open /tmp/cpu.cpuprofile in Chrome DevTools → Performance tab
+```
+
+```javascript
+// Heap snapshot
+await client.HeapProfiler.enable();
+const chunks = [];
+client.HeapProfiler.addHeapSnapshotChunk(({ chunk }) => chunks.push(chunk));
+await client.HeapProfiler.takeHeapSnapshot({ reportProgress: false });
+require('fs').writeFileSync('/tmp/heap.heapsnapshot', chunks.join(''));
+```
+
+## Common Pitfalls
+
+1. **Wrong line numbers in TS source.** Breakpoints hit the emitted JS, not the `.ts`. Either (a) break in the built `dist/*.js`, or (b) enable sourcemaps (`node --enable-source-maps`) and use `sb('src/app.tsx', N)` — but only with CDP clients that follow sourcemaps. `node inspect` CLI does not.
+
+2. **`--inspect` vs `--inspect-brk`.** `--inspect` starts the inspector but doesn't pause; your script races past your first breakpoint if you attach too late. Use `--inspect-brk` when you need to set breakpoints before any code runs.
+
+3. **Port collisions.** Default is `9229`. If multiple Node processes are inspecting, pass `--inspect=0` (random port) and read the actual URL from `/json/list`:
+   ```bash
+   curl -s http://127.0.0.1:9229/json/list   # lists all inspectable targets on the host
+   ```
+
+4. **Child processes.** `--inspect` on a parent does NOT inspect its children. Use `NODE_OPTIONS='--inspect-brk' node parent.js` to propagate to every child; be aware they all need unique ports (Node auto-increments when `NODE_OPTIONS='--inspect'` is inherited).
+
+5. **Background kills.** If you `Ctrl+C` out of `node inspect` while the target is paused, the target stays paused. Either `cont` first, or `kill` the target explicitly.
+
+6. **Running `node inspect` through an agent terminal.** It's a PTY-friendly REPL. In Hermes, launch it with `terminal(pty=true)` or `background=true` + `process(action='submit', data='...')`. Non-PTY foreground mode will work for one-shot commands but not for interactive stepping.
+
+7. **Security.** `--inspect=0.0.0.0:9229` exposes arbitrary code execution. Always bind to `127.0.0.1` (the default) unless you have an isolated network.
+
+## Verification Checklist
+
+After setting up a debug session, verify:
+
+- [ ] `curl -s http://127.0.0.1:9229/json/list` returns exactly the target you expect
+- [ ] First breakpoint actually hits (if it doesn't, you likely missed `--inspect-brk` or attached after execution completed)
+- [ ] Source listing at pause shows the right file (mismatch = sourcemap issue, see pitfall 1)
+- [ ] `exec process.pid` in `repl` returns the PID you meant to attach to
+
+## One-Shot Recipes
+
+**"Why is this variable undefined at line X?"**
+```bash
+node --inspect-brk script.js &
+node inspect -p $!
+# debug>
+sb('script.js', X)
+cont
+# paused. Now:
+repl
+> myVariable
+> Object.keys(this)
+```
+
+**"What's the call path into this function?"**
+```
+debug> sb('suspectFn')
+debug> cont
+# paused on entry
+debug> bt
+```
+
+**"This async chain hangs — where?"**
+```
+# Start with --inspect (no -brk), let it run to the hang, then:
+debug> pause
+debug> bt
+# Now you see the stuck frame
+```
--- a/skills/software-development/python-debugpy/SKILL.md
+++ b/skills/software-development/python-debugpy/SKILL.md
@@ -0,0 +1,374 @@
+---
+name: python-debugpy
+description: Use when debugging Python code (run_agent.py, cli.py, tui_gateway, tests, scripts) with real breakpoints, stepping, scope inspection, and post-mortem analysis. Covers `pdb` for interactive REPL debugging and `debugpy` for remote/headless DAP-driven sessions.
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+metadata:
+  hermes:
+    tags: [debugging, python, pdb, debugpy, breakpoints, dap, post-mortem]
+    related_skills: [systematic-debugging, node-inspect-debugger, debugging-hermes-tui-commands]
+---
+
+# Python Debugger (pdb + debugpy)
+
+## Overview
+
+Three tools, picked by situation:
+
+| Tool | When |
+|---|---|
+| **`breakpoint()` + pdb** | Local, interactive, simplest. Add `breakpoint()` in the source, run normally, get a REPL at that line. |
+| **`python -m pdb`** | Launch an existing script under pdb with no source edits. Useful for quick poking. |
+| **`debugpy`** | Remote / headless / "attach to already-running process." Talks DAP, scriptable from terminal, works for long-lived processes (gateway, daemon, PTY children). |
+
+**Start with `breakpoint()`.** It's the cheapest thing that works.
+
+## When to Use
+
+- A test fails and the traceback doesn't reveal why a value is wrong
+- You need to step through a function and watch a collection mutate
+- A long-running process (hermes gateway, tui_gateway) misbehaves and you can't restart it
+- Post-mortem: an exception fired in prod-ish code and you want to inspect locals at the crash site
+- A subprocess / child (Python `_SlashWorker`, PTY bridge worker) is the actual bug site
+
+**Don't use for:** things `print()` / `logging.debug` solve in under a minute, or things `pytest -vv --tb=long --showlocals` already reveals.
+
+## pdb Quick Reference
+
+Inside any pdb prompt (`(Pdb)`):
+
+| Command | Action |
+|---|---|
+| `h` / `h cmd` | help |
+| `n` | next line (step over) |
+| `s` | step into |
+| `r` | return from current function |
+| `c` | continue |
+| `unt N` | continue until line N |
+| `j N` | jump to line N (same function only) |
+| `l` / `ll` | list source around current line / full function |
+| `w` | where (stack trace) |
+| `u` / `d` | move up / down in the stack |
+| `a` | print args of the current function |
+| `p expr` / `pp expr` | print / pretty-print expression |
+| `display expr` | auto-print expr on every stop |
+| `b file:line` | set breakpoint |
+| `b func` | break on function entry |
+| `b file:line, cond` | conditional breakpoint |
+| `cl N` | clear breakpoint N |
+| `tbreak file:line` | one-shot breakpoint |
+| `!stmt` | execute arbitrary Python (assignments included) |
+| `interact` | drop into full Python REPL in current scope (Ctrl+D to exit) |
+| `q` | quit |
+
+The `interact` command is the most powerful — you can import anything, inspect complex objects, even call methods that mutate state. Locals are read-only by default; use `!x = 42` from the `(Pdb)` prompt to mutate.
+
+## Recipe 1: Local breakpoint
+
+Easiest. Edit the file:
+
+```python
+def compute(x, y):
+    result = some_helper(x)
+    breakpoint()           # <-- drops into pdb here
+    return result + y
+```
+
+Run the code normally. You land at the `breakpoint()` line with full access to locals.
+
+**Don't forget to remove `breakpoint()` before committing.** Use `git diff` or a pre-commit grep:
+```bash
+rg -n 'breakpoint\(\)' --type py
+```
+
+## Recipe 2: Launch a script under pdb (no source edits)
+
+```bash
+python -m pdb path/to/script.py arg1 arg2
+# Lands at first line of script
+(Pdb) b path/to/script.py:42
+(Pdb) c
+```
+
+## Recipe 3: Debug a pytest test
+
+The hermes test runner and pytest both support this:
+
+```bash
+# Drop to pdb on failure (or on any raised exception):
+scripts/run_tests.sh tests/path/to/test_file.py::test_name --pdb
+
+# Drop to pdb at the START of the test:
+scripts/run_tests.sh tests/path/to/test_file.py::test_name --trace
+
+# Show locals in tracebacks without pdb:
+scripts/run_tests.sh tests/path/to/test_file.py --showlocals --tb=long
+```
+
+Note: `scripts/run_tests.sh` uses xdist (`-n 4`) by default, and pdb does NOT work under xdist. Add `-p no:xdist` or run a single test with `-n 0`:
+
+```bash
+scripts/run_tests.sh tests/foo_test.py::test_bar --pdb -p no:xdist
+# or
+source .venv/bin/activate
+python -m pytest tests/foo_test.py::test_bar --pdb
+```
+
+This bypasses the hermetic-env guarantees — fine for debugging, but re-run under the wrapper to confirm before pushing.
+
+## Recipe 4: Post-mortem on any exception
+
+```python
+import pdb, sys
+try:
+    run_the_thing()
+except Exception:
+    pdb.post_mortem(sys.exc_info()[2])
+```
+
+Or wrap a whole script:
+
+```bash
+python -m pdb -c continue script.py
+# When it crashes, pdb catches it and you're in the frame of the exception
+```
+
+Or set a global hook in a repl/jupyter:
+
+```python
+import sys
+def excepthook(etype, value, tb):
+    import pdb; pdb.post_mortem(tb)
+sys.excepthook = excepthook
+```
+
+## Recipe 5: Remote debug with debugpy (attach to running process)
+
+For long-lived processes: Hermes gateway, tui_gateway, a daemon, a process that's already misbehaving and can't be restarted clean.
+
+### Setup
+
+```bash
+source /home/bb/hermes-agent/.venv/bin/activate
+pip install debugpy
+```
+
+### Pattern A: Source-edit — process waits for debugger at launch
+
+Add near the top of the entry point (or inside the function you want to debug):
+
+```python
+import debugpy
+debugpy.listen(("127.0.0.1", 5678))
+print("debugpy listening on 5678, waiting for client...", flush=True)
+debugpy.wait_for_client()
+debugpy.breakpoint()       # optional: pause immediately once attached
+```
+
+Start the process; it blocks on `wait_for_client()`.
+
+### Pattern B: No source edit — launch with `-m debugpy`
+
+```bash
+python -m debugpy --listen 127.0.0.1:5678 --wait-for-client your_script.py arg1
+```
+
+Equivalent for module entry:
+
+```bash
+python -m debugpy --listen 127.0.0.1:5678 --wait-for-client -m your.module
+```
+
+### Pattern C: Attach to an already-running process
+
+Needs the PID and debugpy preinstalled in the target's environment:
+
+```bash
+python -m debugpy --listen 127.0.0.1:5678 --pid <pid>
+# debugpy injects itself into the process. Then attach a client as below.
+```
+
+Some kernels/security configs block the ptrace-based injection (`/proc/sys/kernel/yama/ptrace_scope`). Fix with:
+```bash
+echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
+```
+
+### Connecting a client from the terminal
+
+The easiest terminal-side DAP client is VS Code CLI or a small script. From inside Hermes you have two practical options:
+
+**Option 1: `debugpy`'s own CLI REPL** — not an official feature, but a tiny DAP client script:
+
+```python
+# /tmp/dap_client.py
+import socket, json, itertools, time, sys
+
+HOST, PORT = "127.0.0.1", 5678
+s = socket.create_connection((HOST, PORT))
+seq = itertools.count(1)
+
+def send(msg):
+    msg["seq"] = next(seq)
+    body = json.dumps(msg).encode()
+    s.sendall(f"Content-Length: {len(body)}\r\n\r\n".encode() + body)
+
+def recv():
+    header = b""
+    while b"\r\n\r\n" not in header:
+        header += s.recv(1)
+    length = int(header.decode().split("Content-Length:")[1].split("\r\n")[0].strip())
+    body = b""
+    while len(body) < length:
+        body += s.recv(length - len(body))
+    return json.loads(body)
+
+send({"type": "request", "command": "initialize", "arguments": {"adapterID": "python"}})
+print(recv())
+send({"type": "request", "command": "attach", "arguments": {}})
+print(recv())
+send({"type": "request", "command": "setBreakpoints",
+      "arguments": {"source": {"path": sys.argv[1]},
+                    "breakpoints": [{"line": int(sys.argv[2])}]}})
+print(recv())
+send({"type": "request", "command": "configurationDone"})
+# ... loop reading events and sending continue/stepIn/etc.
+```
+
+This is fine for one-off automation but painful as an interactive UX.
+
+**Option 2: Attach from VS Code / Cursor / Zed** — if the user has one open, they can add a `launch.json`:
+
+```json
+{
+  "name": "Attach to Hermes",
+  "type": "debugpy",
+  "request": "attach",
+  "connect": { "host": "127.0.0.1", "port": 5678 },
+  "justMyCode": false,
+  "pathMappings": [
+    { "localRoot": "${workspaceFolder}", "remoteRoot": "/home/bb/hermes-agent" }
+  ]
+}
+```
+
+**Option 3: Ditch DAP, use `remote-pdb`** — usually what you actually want from a terminal agent:
+
+```bash
+pip install remote-pdb
+```
+
+In your code:
+```python
+from remote_pdb import set_trace
+set_trace(host="127.0.0.1", port=4444)   # blocks until connection
+```
+
+Then from the terminal:
+```bash
+nc 127.0.0.1 4444
+# You get a (Pdb) prompt exactly as if debugging locally.
+```
+
+`remote-pdb` is the cleanest agent-friendly choice when `debugpy`'s DAP protocol is overkill. Use `debugpy` only when you actually need IDE integration.
+
+## Debugging Hermes-specific Processes
+
+### Tests
+See Recipe 3. Always add `-p no:xdist` or run single tests without xdist.
+
+### `run_agent.py` / CLI — one-shot
+Easiest: add `breakpoint()` near the suspect line, then run `hermes` normally. Control returns to your terminal at the pause point.
+
+### `tui_gateway` subprocess (spawned by `hermes --tui`)
+The gateway runs as a child of the Node TUI. Options:
+
+**A. Source-edit the gateway:**
+```python
+# tui_gateway/server.py near the top of serve()
+import debugpy
+debugpy.listen(("127.0.0.1", 5678))
+debugpy.wait_for_client()
+```
+Start `hermes --tui`. The TUI will appear frozen (its backend is waiting). Attach a client; execution resumes when you `continue`.
+
+**B. Use `remote-pdb` at a specific handler:**
+```python
+from remote_pdb import set_trace
+set_trace(host="127.0.0.1", port=4444)   # in the RPC handler you want to trap
+```
+Trigger the matching slash command from the TUI, then `nc 127.0.0.1 4444` in another terminal.
+
+### `_SlashWorker` subprocess
+Same pattern — `remote-pdb` with `set_trace()` inside the worker's `exec` path. The worker is persistent across slash commands, so the first trigger blocks until you connect; subsequent slash commands pass through normally unless you re-arm.
+
+### Gateway (`gateway/run.py`)
+Long-lived. Use `remote-pdb` at a handler, or `debugpy` with `--wait-for-client` if you're restarting the gateway anyway.
+
+## Common Pitfalls
+
+1. **pdb under pytest-xdist silently does nothing.** You won't see the prompt, the test just hangs. Always use `-p no:xdist` or `-n 0`.
+
+2. **`breakpoint()` in CI / non-TTY contexts hangs the process.** Safe locally; never commit it. Add a pre-commit grep as a safety net.
+
+3. **`PYTHONBREAKPOINT=0`** disables all `breakpoint()` calls. Check the env if your breakpoint isn't hitting:
+   ```bash
+   echo $PYTHONBREAKPOINT
+   ```
+
+4. **`debugpy.listen` blocks only if you also call `wait_for_client()`.** Without it, execution continues and your first breakpoint may fire before the client is attached.
+
+5. **Attach to PID fails on hardened kernels.** `ptrace_scope=1` (Ubuntu default) allows only same-user ptrace of child processes. Workaround: `echo 0 > /proc/sys/kernel/yama/ptrace_scope` (needs root) or launch under `debugpy` from the start.
+
+6. **Threads.** `pdb` only debugs the current thread. For multithreaded code, use `debugpy` (thread-aware DAP) or set `threading.settrace()` per thread.
+
+7. **asyncio.** `pdb` works in coroutines but `await` inside pdb requires Python 3.13+ or `await` from `interact` mode on older versions. For 3.11/3.12, use `asyncio.run_coroutine_threadsafe` tricks or `!stmt`-based awaits via `asyncio.ensure_future`.
+
+8. **`scripts/run_tests.sh` strips credentials and sets `HOME=<tmpdir>`.** If your bug depends on user config or real API keys, it won't reproduce under the wrapper. Debug with raw `pytest` first to repro, then re-confirm under the wrapper.
+
+9. **Forking / multiprocessing.** pdb does not follow forks. Each child needs its own `breakpoint()` or `set_trace()`. For Hermes subagents, debug one process at a time.
+
+## Verification Checklist
+
+- [ ] After `pip install debugpy`, confirm: `python -c "import debugpy; print(debugpy.__version__)"`
+- [ ] For remote debug, confirm the port is actually listening: `ss -tlnp | grep 5678`
+- [ ] First breakpoint actually hits (if it doesn't, you likely have `PYTHONBREAKPOINT=0`, you're under xdist, or execution finished before attach)
+- [ ] `where` / `w` shows the expected call stack
+- [ ] Post-debug cleanup: no stray `breakpoint()` / `set_trace()` in committed code
+  ```bash
+  rg -n 'breakpoint\(\)|set_trace\(|debugpy\.listen' --type py
+  ```
+
+## One-Shot Recipes
+
+**"Why is this dict missing a key?"**
+```python
+# add above the KeyError site
+breakpoint()
+# then in pdb:
+(Pdb) pp d
+(Pdb) pp list(d.keys())
+(Pdb) w                # how did we get here
+```
+
+**"This test passes in isolation but fails in the suite."**
+```bash
+scripts/run_tests.sh tests/the_test.py --pdb -p no:xdist
+# But if it only fails WITH other tests:
+source .venv/bin/activate
+python -m pytest tests/ -x --pdb -p no:xdist
+# Now it pdb-traps at the exact failing test after state accumulated.
+```
+
+**"My async handler deadlocks."**
+```python
+# Add at handler entry
+import remote_pdb; remote_pdb.set_trace(host="127.0.0.1", port=4444)
+```
+Trigger the handler. `nc 127.0.0.1 4444`, then `w` to see the suspended frame, `!import asyncio; asyncio.all_tasks()` to see what else is pending.
+
+**"Post-mortem on a crash in an Ink child process / subprocess."**
+```bash
+PYTHONFAULTHANDLER=1 python -m pdb -c continue path/to/entrypoint.py
+# On crash, pdb lands at the frame of the exception with full locals
+```