Test Matrix for Capture Reliability

This matrix is the companion to architecture.md (where the I-1..I-15 invariants live) and anti-patterns.md. It exists to make one question answerable at a glance: for every failure avenue we know about, is there a test that would have caught it?

Every failure mode in the 2026-04-22 sev1 investigation (issues #49–#53, #58, plus the two hotfixes #54/#55) and every invariant from architecture.md must appear as a row. When a row is marked blocked-on-*, the test file and a #[ignore] skeleton must still exist, so that enabling the infrastructure is a one-line change rather than “remember to write the test later”.

Conventions

• existing — test was already on main before this PR
• existing (#NN) — test was added by issue/PR #NN (cross-reference only)
• new (this PR) — test added by the PR introducing this matrix
• added-by-#NN-fix — fix PR for #NN owns the regression test; cross-referenced here to avoid duplication
• blocked-on-P0.X — test skeleton exists with #[ignore]; fires when the named roadmap task lands
• source-change-required — test cannot be written without a source change that is outside the scope of this PR; noted here so the gap is explicit

“Invariant” refers to I-1..I-15 defined in architecture.md.

Failure modes

Phase 1 (Bug A) test coverage — F-25

The tests below cover the watcher data-loss fix shipped in T-A.1–T-A.7 (2026-04-27). Row F-25 in the table above represents the failure mode; the entries here give individual test names and file paths for traceability.

Running Phase 1 tests:

# Rust — migration, hash, upsert, enqueue gate, sweep, backfill
cargo test -p hippo-core storage::tests::test_migrate_v11
cargo test -p hippo-daemon claude_session::tests::test_hash_
cargo test -p hippo-daemon claude_session::tests::test_upsert_
cargo test -p hippo-daemon claude_session::tests::test_decide_enqueue_
cargo test -p hippo-daemon claude_session::tests::test_insert_segments_skips_enqueue_for_empty_segment
cargo test -p hippo-daemon watch_claude_sessions::tests::test_sweep_
cargo test -p hippo-daemon backfill::tests::test_backfill_

# Python — brain hash propagation
uv run --project brain pytest brain/tests/test_claude_sessions.py::TestContentHashPropagation -v

Invariant coverage cross-check

Any invariant without at least one new (this PR) or existing row is by definition gated on a P0/P1/P2 task. If you see an invariant listed in architecture.md that is not in the table above, that is a gap — open an issue and add a row.

Test coverage gaps

These are the failure modes that cannot be tested against main today:

• F-6 NM manifest validation — hippo doctor never reads the NM manifest. The doctor check would need ~20 lines in commands.rs (read JSON, resolve path, assert executable, assert allowed_extensions contains hippo-browser@local). Test skeleton exists; remove #[ignore] once the source check lands.
• F-8 Fallback age in doctor — storage::list_fallback_files returns paths sorted by name; doctor only prints a count. An age check requires either (a) reading each file’s mtime in doctor, or (b) a new storage::list_stale_fallback_files(dir, cutoff_ms) helper. Test skeleton exists; source change tracked by follow-up issue.
• F-10..F-14 — All require source_health table and the watchdog/probe subsystems from architecture.md. Skeletons live in crates/hippo-daemon/tests/capture_invariants.rs.
• F-15 — #53 is about the plumbing from “hippo CI failure” → upsert_cluster; the lessons.py logic itself has solid unit coverage (brain/tests/test_lessons.py). Our xfail test asserts the end-to-end pipeline. It will stay xfail until the plumbing ships.

Running the tests

# Rust — the redaction negative cases, doctor unit tests, NM/fallback skeletons
cargo test -p hippo-core redaction::
cargo test -p hippo-daemon commands::tests::test_check_claude_session_hook
cargo test -p hippo-daemon --test nm_manifest_doctor
cargo test -p hippo-daemon --test nm_restart_integration
cargo test -p hippo-daemon --test fallback_age_doctor
cargo test -p hippo-daemon --test capture_invariants
cargo test -p hippo-daemon --test schema_handshake_negative

# Every `#[ignore]` skeleton — re-enable when its P0/P1/P2 dependency lands
cargo test -p hippo-daemon -- --ignored --nocapture

# Shell
bash tests/shell/test-claude-session-hook-extended.sh
bash tests/shell/test-hook-pid-ppid.sh

# Brain (xfail stays green; becomes pass on #53 fix)
uv run --project brain pytest brain/tests/test_lessons_graduation_hippo.py -v

# Static analysis (when wired into CI)
semgrep --config .semgrep.yml crates/ brain/

How to extend

When you add a new capture path (say, iMessage ingestion) or discover a new failure mode, you MUST:

1. Add a row to the Failure modes table above. Include:
   • A one-sentence description of the failure.
   • The trigger (issue number, commit, design-doc reference).
   • The chosen test type and file path.
   • An invariant reference, if any.
2. Write the test. If the test depends on infrastructure that does not exist yet, commit the test file with a #[ignore = "blocked on <task-id>"] attribute (Rust) or @pytest.mark.skip(reason=...) / xfail (Python) so the file compiles and the skeleton is visible.
3. If the test cannot be written at all without changing source, add a row to Test coverage gaps explaining why, and open a follow-up issue for the source change. Do not silently drop the failure mode.
4. Update the Invariant coverage cross-check if the new row fills a gap.

The matrix is the source of truth for “what do we test?” If a failure recurs and its row’s test did not fire, that is a bug in the test, not additional justification to skip writing a test next time.