Capture Architecture

Reference for hippo’s capture-reliability stack: how events land, what the system promises, and what fires when something breaks. For per-source detail (what each source captures and where it lands), see sources.md. For first-aid recipes when something goes wrong, see operator-runbook.md. For review-blocker rules every contributor needs to internalize, see anti-patterns.md.

TL;DR

Every capture path writes two things in the same SQLite transaction: the event row and a source_health row. A background watchdog reads source_health once a minute, asserts the named invariants I-1 through I-21, and writes alarms to capture_alarms on violations. A separate probe job sends synthetic events through each path every five minutes and records round-trip latency. Operators see all of this through hippo doctor and hippo alarms.

The four layers

+--------------------+       +-------------------+      +-------------+
|  capture path      |       |  source_health    |      | capture_    |
|  (per source)      | ----> |  (one row/source) | <----| alarms      |
|  writes event +    |       |                   |      |             |
|  health in same Tx |       |                   |      +-------------+
+--------------------+       +-------------------+              ^
                                       ^                        |
                                       |                        |
                              +-----------------+      +-----------------+
                              |  watchdog       |      |  doctor / CLI   |
                              |  every 60s,     |----> |  reads alarms,  |
                              |  asserts I-1..  |      |  shows status   |
                              |  I-12           |      +-----------------+
                              +-----------------+
                                       ^
                                       |
                              +-----------------+
                              |  probe          |
                              |  every 5m,      |
                              |  synthetic      |
                              |  round-trip     |
                              +-----------------+
  1. Capture path — the per-source code that writes events. Shell hook → daemon socket. FSEvents watcher → daemon. Native messaging → daemon. Each path writes to its source’s events table AND to source_health in the same SQLite transaction. (See anti-patterns.md AP-1: writing health from inside the user’s interactive prompt is forbidden — health writes happen in the daemon’s flush_events, never in shell/hippo.zsh.)
  2. source_health table — one row per source, holds the latest “did the event land?” signal: last_event_ts, consecutive_failures, events_last_1h, probe_ok, probe_last_run_ts, probe_lag_ms. Single SQL ground truth.
  3. Watchdog (com.hippo.watchdog, every 60 s): asserts the invariants I-1 through I-21 (most against source_health; I-14 against the knowledge-node vector store, I-16 against the knowledge-node/agentic-session join), writes capture_alarms rows on violations. Rate-limited per invariant (one alarm per invariant per hour). Implemented in crates/hippo-daemon/src/watchdog.rs.
  4. Probe (com.hippo.probe, every 5 minutes) — sends synthetic events through each capture path, measures end-to-end latency, records probe_lag_ms in source_health. Probe rows carry probe_tag IS NOT NULL and are filtered out of every user-facing query (RAG, MCP tools, hippo events). See crates/hippo-daemon/src/probe.rs. (See anti-patterns.md AP-6: probe rows must never appear in user-facing queries.)

Operator interface: hippo doctor for a snapshot, hippo alarms for unacknowledged violations, hippo probe to run a one-off synthetic check.

Claude session watcher

The Claude watcher polls shutdown signals while ingestion or SQLite writes are pending. One sequential maintenance worker handles heartbeat, settling, and file discovery independently of ingestion. Missed notifications are recovered through reconciliation. A timed-out ingestion worker must finish before the same file can be processed again. Startup database work is also cancellable at its async wait. Runtime teardown waits at most one second for blocking workers; unfinished ingestion is recovered from persisted offsets on the next start. See crates/hippo-daemon/src/watch_claude_sessions.rs for the implementation and shared-database contention regressions.

Write transactions and SQLITE_BUSY

hippo_core::storage::open_db sets the connection’s transaction behavior to IMMEDIATE, so every transaction opened with rusqlite’s transaction API (transaction(), unchecked_transaction()) on an open_db connection runs BEGIN IMMEDIATE and takes the write lock at the start. Raw BEGIN; statements in migration batches and connections opened outside open_db are not covered. busy_timeout=5000 then applies. A deferred transaction that reads and then writes skips the busy wait at the read-to-write upgrade and fails at once with SQLITE_BUSY or SQLITE_BUSY_SNAPSHOT whenever any other connection writes, however briefly. A SQLITE_BUSY that survives the 5 s wait (and the single 100 ms retry in with_busy_retry) therefore means a real 5 s lock, and the ERROR log line for it stays. Do not run a long read-only scan inside a Connection::transaction() on an open_db connection, because it would hold the write lock. Use plain statements or a read-only connection.

Tables

source_health

One row per source. Updated in the same transaction as event writes; the watchdog reads it; the probe job updates probe_* columns.

ColumnTypeMeaning
sourceTEXT PKSource name (one row per source; non-exhaustive): shell, claude-tool, agentic-session-claude, agentic-session-opencode, agentic-session-codex, agentic-session-cursor, browser, claude-session-watcher, workflow, watchdog, brain-preflight. (There is no probe row — the probe job writes probe_* columns onto each real source’s row, not a separate probe heartbeat.)
last_event_tsINTEGEREpoch ms of the most recent production event for this source. Synthetic events with a non-NULL probe_tag do not advance it.
consecutive_failuresINTEGERBumped on each failure; reset on success. Backstop for I-1, I-4 freshness alarms.
events_last_1h / _24hINTEGERProduction event counts excluding probes. The daemon increments these on real writes and corrects them every 5 minutes from events, agentic_sessions, and browser_events. That correction also replaces shell, Claude-tool, and browser timestamps previously derived from probes with the latest production timestamp, or NULL when none exists.
probe_okINTEGERLast probe-job result: 1 = healthy, 0 = unhealthy. Source-specific definition (see “Probes” below).
probe_last_run_tsINTEGERWhen the probe last completed for this source.
probe_lag_msINTEGEREnd-to-end latency of the most recent successful probe.
updated_atINTEGERAlways bumped on any column update.

Shell, Claude-tool, and browser capture liveness uses the later of the production timestamp and the latest probe timestamp when that probe succeeded. Doctor labels this combined signal as capture; the source-health lag gauge also accepts heartbeats. These liveness signals do not represent production activity. Probe failure and freshness remain independently checked by I-8.

capture_alarms

Append-only ledger of invariant violations. The watchdog writes; hippo alarms ack flips the acknowledgment flag.

ColumnMeaning
idPK
invariant_idOne of I-1 … I-21
raised_atFirst detection time (epoch ms)
details_jsonInvariant-specific diagnostic context — affected source, since_ms, and per-invariant details
acked_atNULL until hippo alarms ack <id>
ack_noteOptional note supplied at acknowledgment
resolved_atSet once the invariant has stayed clean for 2 consecutive ticks
clean_ticksConsecutive-clean tick count driving the auto-resolve loop

Invariants

Asserted by the watchdog every 60 s. Each has a formal predicate in crates/hippo-daemon/src/watchdog.rs. Violations create or refresh a capture_alarms row; the doctor surfaces them with [!!] severity.

IDAssertionThresholdSuppressed whenBackstop
I-1 Shell livenessIf a real (non-probe) shell command has landed within the last 10 minutes, shell/probe events must land within one probe cadence plus jitter grace.7 minprobe_ok != 1; no real (probe_tag IS NULL) shell event within the last 10 minutes (ordinary idleness: sleep, stepping away, working elsewhere).Watchdog alarm + doctor [!!] shell capture.
I-2 Claude-session end-to-endFor every Claude JSONL with mtime < 5 min, a matching claude_sessions row must exist.5 minNo live JSONL.Watchdog alarm naming each missing session_id.
I-3 Claude-tool concurrencyIf a live JSONL has received a tool_use line within 5 min, at least one matching events.source_kind='claude-tool' row must exist in that window.5 minNo live JSONL with recent tool_use.Structured log only by default; opt-in alarm via [watchdog] claude_tool_alarm = true.
I-4 Browser round-tripIf extension heartbeat is recent (within 5 min cadence + grace) and Firefox is running, browser events must land within one probe cadence plus jitter grace.7 minExtension heartbeat absent or stale; Firefox not running (doctor suppresses).Watchdog alarm + doctor [!!] browser capture with connectivity state.
I-5 Drop visibilityEvery event dropped (socket accept + crash, buffer overflow) increments a persistent counter. Zero tolerance for invisible drops.every drop—OTel counter hippo.daemon.events.dropped (paired with hippo.daemon.events.ingested); see crates/hippo-daemon/src/metrics.rs.
I-6 Buffer non-saturationSustained drop rate over any 5 min sliding window ≤ 0.1% of total event traffic.0.1% / 5 min—Watchdog alarm + doctor [!!] drop-rate.
I-7 Watchdog livenessThe watchdog itself writes to source_health WHERE source='watchdog' at least every 60 s.180 s stale—Doctor only (a dead watchdog can’t alarm about itself).
I-8 Probe freshnessFor each source with probe_last_run_ts IS NOT NULL: probe_ok = 1 OR probe_last_run_ts > now − 15 min.15 minshell only: probe stale (not outright failing) AND no real shell event within 10 minutes either. Most likely the machine was asleep, not that shell-probe delivery broke. Never suppresses an outright probe failure (probe_ok = 0).Watchdog alarm + doctor [!!] <source> probe.
I-9 Fallback file ageIf any JSONL fallback file under ~/.local/share/hippo/ is > 24 h old AND the daemon socket is responsive, recovery is broken.24 hDaemon down (fallback drain happens at startup).Doctor [!!] fallback files.
I-10 Capture/enrichment decouplingBrain being down (HTTP 5xx/timeout) MUST NOT prevent source_health updates for capture sources. Architectural — verified via canary in CI, not at runtime.——Architectural enforcement; if violated, every other invariant becomes unreliable.
I-11 Opencode-session coverageIf agentic-session-opencode.consecutive_failures > 3, the poller is actively broken. Proxy predicate; full freshness check lives in hippo doctor which suppresses on idle opencode-DB mtime.proxyBench pause window.Watchdog alarm + doctor [!!] agentic-session-opencode events.
I-12 Brain preflight stuckIf brain-preflight.consecutive_failures > 12 (≈ 1 minute at the brain’s 5 s poll), the inference backend has been unreachable for long enough to be a real outage. Motivating incident: silent [lmstudio] → [inference] config-section drift made the brain point at port 1234 forever with no alarm.~1 minOperator enrichment pause (hippo enrichment pause, or an unreadable state file). The brain also resets brain-preflight.consecutive_failures on pause transitions, so only fresh preflight results count after resume. This holds when the brain ran during the pause; if the brain was down for the whole pause, a count recorded before the pause still applies.Watchdog alarm. Doctor surfaces it via the existing Brain inference backend: unreachable line.
I-13 Codex-session coverageIf agentic-session-codex.consecutive_failures > 3, the Codex rollout poller is actively broken. Proxy predicate; full freshness check lives in hippo doctor.proxyBench pause window.Watchdog alarm + doctor [!!] agentic-session-codex events.
I-14 Embedding orphan backlogCount of knowledge_nodes older than reaper.orphan_stale_secs with no row in the knowledge_vectors shadow table must stay ≤ reaper.alarm_threshold. A sustained backlog means the embedding orphan-reaper is down or wedged.25 orphans (configurable)Shadow table absent — fresh install, nothing embedded yet.Watchdog alarm.
I-15 Cursor-session coverageIf agentic-session-cursor.consecutive_failures > 3, the Cursor poller is actively broken. Proxy predicate; full freshness in hippo doctor.proxyBench pause window.Watchdog alarm + doctor [!!] agentic-session-cursor events.
I-16 Agentic node dedupNo single agentic_sessions segment may carry more than one knowledge node (any type) with identical (content, embed_text, node_type). A non-zero count means an enricher is minting duplicate nodes instead of replacing/reusing them (the historical KB-duplication class — see AP-13). Covers all node types now that every writer is guarded (agentic = write-time replacement; workflow/browser = write-time content dedup).0 dup-groups (configurable via [watchdog] dup_node_alarm_threshold)knowledge_node_agentic_sessions absent — fresh install.Watchdog alarm. Remediate with brain/scripts/dedup-knowledge-nodes.py.
I-19 Stale auto-memory projectionNo active, non-probe memory_documents row may stay projection_status = 'pending' past the stale threshold. Staleness is measured from the later of the row’s updated_at and the last operator resume time.30 minBench pause window; suppressed while enrichment is paused by the operator.Watchdog alarm + doctor auto-memory projection line.
I-21 Backup freshnessA backup must have succeeded within 36 h, and no shipped artifact may wait more than 24 h for upload (Drive FS xattr com.google.drivefs.item-id#S).36 h / 24 h[backup] not configured (no destination and recipient).Watchdog alarm + doctor [!!] backup / [WW] backup. See ../backup.md.

Probes

Synthetic round-trip verification, every 5 minutes per source.

Backstops

The system promises observability, not correctness. If something breaks, the goal is for the user to see it within minutes, not 21 days (the duration of an actual past silent browser-capture outage that motivated this architecture).

See also