TypeSafe experiments for Hippo
The synthetic screen supports comparing Jev reranking and advisory claim verification with Hippo’s current implementation. It does not establish a production accuracy or latency improvement. The answerability question failed to distinguish missing requested facts; relevance must not become an answerability gate.
cases.json ↗ contains the initial authored fixtures.
holdout.json ↗ contains follow-up stress cases authored after
inspecting failures, so it is not an independent holdout. Labels are excluded
from inference requests. run.py ↗ supports offline replay and explicit
live requests using the Python standard library.
Generated outputs and the full research report are stored outside the repository:
archive="${XDG_DATA_HOME:-$HOME/.local/share}/hippo-bench/decisions/archive/typesafe-2026-09-18"
The archive contains README.md, exact request/response records in results.json
and holdout-results.json, their *-summary.json files, and original fixtures
and runner snapshots. The parent archive’s relocation-2026-09-22.json records
SHA256 checksums for the verified relocation. Existing archive files are not
rewritten by replay.
Replay the initial and follow-up results without inference:
python3 docs/research/typesafe-2026-09-18/run.py
python3 docs/research/typesafe-2026-09-18/run.py \
--cases docs/research/typesafe-2026-09-18/holdout.json \
--results "$archive/holdout-results.json"
--self-check validates metrics, ordering and label exclusion without saved
results. Live requests require an existing TYPESAFE_API_KEY, --live, and a
new external --results path. The runner refuses existing output files and
paths inside a Git repository. The external default respects XDG_DATA_HOME;
an unset or empty value uses ~/.local/share.
Use the decision sidecar for comparisons against Hippo’s local model and declarative rules. No production selection changed.