Capture Architecture

Reference for hippo’s capture-reliability stack: how events land, what the system promises, and what fires when something breaks. For per-source detail (what each source captures and where it lands), see sources.md. For first-aid recipes when something goes wrong, see operator-runbook.md. For review-blocker rules every contributor needs to internalize, see anti-patterns.md.

TL;DR

Every capture path writes two things in the same SQLite transaction: the event row and a source_health row. A background watchdog reads source_health once a minute, asserts sixteen named invariants, and writes alarms to capture_alarms on violations. A separate probe job sends synthetic events through each path every five minutes and records round-trip latency. Operators see all of this through hippo doctor and hippo alarms.

The four layers

+--------------------+       +-------------------+      +-------------+
|  capture path      |       |  source_health    |      | capture_    |
|  (per source)      | ----> |  (one row/source) | <----| alarms      |
|  writes event +    |       |                   |      |             |
|  health in same Tx |       |                   |      +-------------+
+--------------------+       +-------------------+              ^
                                       ^                        |
                                       |                        |
                              +-----------------+      +-----------------+
                              |  watchdog       |      |  doctor / CLI   |
                              |  every 60s,     |----> |  reads alarms,  |
                              |  asserts I-1..  |      |  shows status   |
                              |  I-12           |      +-----------------+
                              +-----------------+
                                       ^
                                       |
                              +-----------------+
                              |  probe          |
                              |  every 5m,      |
                              |  synthetic      |
                              |  round-trip     |
                              +-----------------+

1. Capture path — the per-source code that writes events. Shell hook → daemon socket. FSEvents watcher → daemon. Native messaging → daemon. Each path writes to its source’s events table AND to source_health in the same SQLite transaction. (See anti-patterns.md AP-1: writing health from inside the user’s interactive prompt is forbidden — health writes happen in the daemon’s flush_events, never in shell/hippo.zsh.)
2. source_health table — one row per source, holds the latest “did the event land?” signal: last_event_ts, consecutive_failures, events_last_1h, probe_ok, probe_last_run_ts, probe_lag_ms. Single SQL ground truth.
3. Watchdog (com.hippo.watchdog, every 60 s) — asserts sixteen invariants (most against source_health; I-14 against the knowledge-node vector store, I-16 against the knowledge-node/agentic-session join), writes capture_alarms rows on violations. Rate-limited per invariant (one alarm per invariant per hour). Implemented in crates/hippo-daemon/src/watchdog.rs.
4. Probe (com.hippo.probe, every 5 minutes) — sends synthetic events through each capture path, measures end-to-end latency, records probe_lag_ms in source_health. Probe rows carry probe_tag IS NOT NULL and are filtered out of every user-facing query (RAG, MCP tools, hippo events). See crates/hippo-daemon/src/probe.rs. (See anti-patterns.md AP-6: probe rows must never appear in user-facing queries.)

Operator interface: hippo doctor for a snapshot, hippo alarms for unacknowledged violations, hippo probe to run a one-off synthetic check.

Tables

source_health

One row per source. Updated in the same transaction as event writes; the watchdog reads it; the probe job updates probe_* columns.

capture_alarms

Append-only ledger of invariant violations. The watchdog writes; hippo alarms ack flips the acknowledgment flag.

Invariants (I-1..I-16)

Asserted by the watchdog every 60 s. Each has a formal predicate in crates/hippo-daemon/src/watchdog.rs. Violations create or refresh a capture_alarms row; the doctor surfaces them with [!!] severity.

Probes

Synthetic round-trip verification, every 5 minutes per source.

• Mechanism. A hippo probe --source <name> invocation generates a synthetic event tagged with a per-run UUID in probe_tag, then waits for it to appear in the source’s events table. Browser probes use the same per-run tag to bypass the normal browser URL/time-bucket dedup window. End-to-end latency is recorded in source_health.probe_lag_ms.
• Where they live. Probe rows have probe_tag IS NOT NULL. Every user-facing query (RAG retrieval, MCP search_events / search_knowledge / get_entities, hippo events, hippo ask) filters them out at the daemon-side query path. A Semgrep rule blocks new query call-sites that omit the filter. (See anti-patterns.md AP-6.)
• Per-source probe_ok definition. For shell: pgrep -x zsh non-empty AND hippo.zsh sourced AND HID idle < 5 min. For browser: Firefox running AND extension heartbeat fresh. For claude-session: at least one JSONL under ~/.claude/projects with recent mtime. The watchdog computes these on every cycle.
• Manual probe. hippo probe --source <name> runs one cycle on demand. Useful when bringing a source back up after a configuration change.

Backstops

The system promises observability, not correctness. If something breaks, the goal is for the user to see it within minutes, not 21 days (the duration of an actual past silent browser-capture outage that motivated this architecture).

• hippo doctor — interactive, < 2 s wall-clock, ten checks, exit code = fail count. --explain adds CAUSE/FIX/DOC per failure.
• hippo alarms list — unacknowledged alarms; exits 1 if any.
• macOS notification — opt-in via [watchdog] notify_macos = true. Rate-limited to one per invariant per hour.
• OTel — every counter is a Prometheus metric when the otel feature is built (default-on). See otel/README.md ↗ for the local Grafana stack.

See also

• docs/lifecycle.md — end-to-end trace of how each event type becomes a knowledge node, with diagnostic SQL recipes.
• sources.md — what each source captures, where it lands, what fires.
• anti-patterns.md — AP-1..AP-12: review blockers.
• operator-runbook.md — doctor recipes, alarm responses, recovery flows.
• test-matrix.md — failure-mode-to-test mapping and the contract for adding new tests.
• docs/archive/capture-reliability-overhaul/ — historical design records (P0–P3 overhaul, post-mortems).