Isolated decision sidecars
The opt-in runtime, expanded corpus workflow, and classification commands are documented in Jev decisions. This page describes the original isolated sidecar runner and its historical comparison contract.
Run the offline rules benchmark:
mise run bench:decisions
The runner prints its artifact directory under
$XDG_DATA_HOME/hippo-bench/decisions/ (default
~/.local/share/hippo-bench/decisions/). Each arm runs in a separate process.
Generated runs, source snapshots and recorded results must remain outside Git.
Archived experiments live under decisions/archive/; only authored corpora,
reusable scripts and concise documentation belong in the repository.
No decision worker opens hippo.db, claims enrichment work, changes models, pauses the
brain, or publishes a result into Hippo. Production inference selection stays
unchanged. No new dependencies are required.
The benchmark covers two semantic judgments:
- Reranking: reuse Hippo’s existing
build_rerank_messagesandparse_rankingfor the local LLM. Jev asks one relevance Score per candidate in one request. Every arm sees the same query and candidate text, including Hippo’s existing 300-character field budgets. Original candidate identities are retained across reversed repeats. - Claim verification: classify a supplied claim against supplied evidence
as
supports,contradicts, orunsupported. This is an experimental verifier using the configured local LLM as comparator. Hippo has no existing production verifier to replay. Claims must already be split and associated with their evidence.
Neither judgment generates summaries or answers. Relevance is not proof that the requested fact is known. The earlier answerability experiment did not justify an answerability gate.
Run all five arms on the bundled synthetic corpus:
# TYPESAFE_API_KEY must already be exported in this shell.
mise run bench:decisions -- \
--arms llm,jev,rules,rules_jev,rules_llm \
--llm-url http://127.0.0.1:42069/v1 \
--llm-model YOUR_CURRENT_QUERY_MODEL \
--jev-model jev-1.13.0 \
--repeats 2 --reverse
An explicit inference URL and model are required for LLM arms. The runner does
not resolve or change the production configuration. Supplying production’s
inference URL shares that server’s compute and caches. A separate inference
server provides compute isolation. Arm execution is sequential by default;
--parallel runs workers concurrently and records that scheduling choice.
Use sequential runs when comparing latency without inter-arm contention.
Jev arms send the selected corpus text to TypeSafe. Capture alone is local;
decision runs containing jev or rules_jev make those requests.
The default run is rules-only and makes no inference requests.
All inference requests are single attempts with a configurable --timeout
(60 seconds by default), temperature zero for the local LLM, and pooled HTTP
connections. There are no hidden retries. This compares decision prompts and
outputs, not the complete production service. Production’s transport retry,
connection, and 300-second timeout behavior are not being timed here.
Failed calls remain in the artifacts and denominators. Run exit status is 3
for completed runs with case errors and 2 for setup or worker failures.
Ctrl-C and SIGTERM terminate and reap owned workers, retain partial artifacts,
and exit with status 130. Missing records remain visible in the summary.
The default fixture contains the 46 ranking and verification cases from the September research, plus three controls for exact-text, empty-evidence, and exact-identifier rules. The research cases are synthetic; the follow-up cases were authored after inspecting initial failures. These are regression checks, not an independent production holdout.
The rules live in
decision_rules.json ↗.
They are a declarative decision table evaluated in Python:
{
"id": "empty-evidence",
"task": "verification",
"priority": 100,
"when": [{"fact": "empty_source", "op": "eq", "value": true}],
"output": "unsupported"
}
Conditions within a rule use AND semantics. Multiple rules express alternatives. The highest priority wins. Conflicting outputs at that priority, or no match, produce an abstention. The engine validates fact names, operators, priorities, and outputs before starting workers. Rules cannot execute code or mutate facts. This is a decision table, not a forward-chaining Drools implementation.
Ranking facts are token_overlap, query_tokens, and exact_query.
Verification facts are empty_source and identical_text. Operators are
eq, ne, ge, and gt. Ranking outputs are relevance levels 0 through 3;
verification outputs are the three verdicts above. Exact matches are a limited
rules baseline; lexical overlap cannot establish semantic support.
Use --rules PATH to compare a changed rule set. Every result retains facts,
matched rule IDs, winning rule IDs, and conflicts. The rules arm exposes its
raw decisions and abstentions. rules_jev and rules_llm run the engine first
and invoke inference on abstention. For ranking, they also invoke inference
unless the best rule score exceeds the second by --rules-margin (default 2).
Inference receives the original complete candidate set; rules do not prune it.
These are separately executed paths, so their measured time and token usage
include the inference calls they actually make.
For production input capture, use the capture configuration and limits. Capture does not enable reranking; its synchronous file write adds local filesystem work to the query.
Replay the frozen captures locally:
mise run bench:decisions -- \
--cases ~/.local/share/hippo-bench/decisions/captures \
--arms llm,rules,rules_llm \
--llm-url http://127.0.0.1:42069/v1 \
--llm-model YOUR_CURRENT_QUERY_MODEL
Custom corpora can be JSON arrays, JSONL, or directories of captured JSON files.
Ranking cases have id, task: "ranking", query, and candidates (strings
or objects with summary, embed_text, and commands_raw). Optional grades
are integers 0 through 3 in candidate order. Verification cases have id,
task: "verification", source, claim, and optional expected. Capture
files place inference fields under state. Labels are separated before any
inference request. Unlabeled captures report agreement, latency, errors, and
coverage; they do not report accuracy.
Inspect or scrape a run while it is running or after it finishes:
mise run bench:decisions:summary -- /path/to/run
mise run bench:decisions:metrics -- /path/to/run --port 9836
curl http://127.0.0.1:9836/metrics
The metrics sidecar binds to loopback. Its metrics use
hippo_decision_shadow_* names and service_namespace="hippo-bench".
Inference workers disable production OpenTelemetry exporters. Point a
Prometheus scrape job at the metrics sidecar explicitly; starting a benchmark
does not alter observability configuration or install a background service.
Each artifact directory contains:
manifest.json,inputs.jsonl,labels.json, andrules.json: frozen corpus, labels, rule set, hashes, model IDs, request policy, and scheduling.<arm>.jsonl: exact requests and responses, input and request hashes, returned model version, token usage, rule traces, HTTP status, errors, monotonic request and end-to-end decision durations, and worker CPU/RSS.<arm>.logandcompletion.json: worker failures and exit status.summary.json: planned and observed counts, missing records, coverage, p50/p95 latency, token totals, paired agreement, and labeled quality.
The summary’s accuracy and ndcg measure successful inference decisions over
all planned scoreable cases. Errors, missing worker records, and abstentions
count as zero. effective_ranking_accuracy and effective_ndcg additionally
score original-order fallback on failed calls. Jev’s fallback is the proposed
shadow behavior, not a production integration. Ranking cases with all-zero
grades have no meaningful ideal ranking and are excluded from ranking quality
denominators, but remain in operational counts. Coverage uses planned cases,
so crashed or missing workers cannot look complete. Paired agreement uses
matching input hashes and reports the number of pairs with successful outputs;
agreement with the current LLM is not ground truth.
Token totals include only provider-reported usage. usage_missing counts
requests with unknown token usage; unknown tokens are not estimated from
character counts. CPU and memory are client-process measurements, excluding
the inference server. Peak RSS is the process lifetime high-water mark.
Latency excludes process startup and artifact serialization. The raw records
retain enough information to inspect order sensitivity and model probabilities.
Conflict experiments compare the existing analyze_conflicts implementation
with semantic judgments over identical knowledge-hit fields:
# Offline, current production functions versus the declarative rules table.
mise run bench:conflicts
# Live, using the same configured local model as the other decision experiments.
mise run bench:conflicts -- \
--arms current,jev,rules,rules_jev,llm,rules_llm \
--llm-url http://127.0.0.1:42069/v1 \
--llm-model YOUR_CURRENT_QUERY_MODEL \
--repeats 2 --reverse
The current arm supports decision_conflict and outcome_conflict tasks
only. The LLM arm for these tasks is an experimental semantic comparator;
production uses deterministic detectors, not an LLM conflict prompt. Both
models receive the same semantic policy. The policy distinguishes unresolved
same-scope contradictions from explicit recovery, replacement and unrelated
work. That is a proposed refinement of the current product behavior, not a
restatement of the existing unit-test contract. compatible means that these
records establish no unresolved conflict; it does not establish current truth.
Conflict cases supply state.hits and optional expected (conflict or
compatible). Each hit retains summary, cwd, git_branch, captured_at,
outcome, and design_decisions with chosen, considered, and reason.
The normalizer assigns opaque hit IDs and strips all other fields, including
annotations. Staleness, source freshness and evidence lookup are outside this
comparison. No production conflict capture hook is installed.
Conflict rules use comparable_records and identical_records. They return
compatible for absent comparisons or identical substantive records, then
abstain for semantic judgments. Matching chosen strings or outcome labels alone
do not skip inference. Reverse repeats reverse hit order while preserving IDs
and timestamps. Summaries pair these tasks against current when present and
report false_conflicts and missed_conflicts. Those two counts cover successful
verdicts; failures, missing records and abstentions remain separate and count
as zero accuracy. Repeats are not independent labeled examples.
New runs also save the relevant Python source files under source/, alongside
their manifest hashes. The conflict research record
contains the fixed corpus, policy and replay instructions. Recorded requests,
responses, measured protocols, source snapshots and analyses reside under
$XDG_DATA_HOME/hippo-bench/decisions/archive/
(default ~/.local/share/hippo-bench/decisions/archive/), in
typesafe-2026-09-20-conflicts/. The initial paired
sidecar run and earlier failed run are copied under
archive/typesafe-2026-09-20-sidecars/, with their original run IDs.
archive/relocation-2026-09-22.json records the verified source and archive
SHA256 hashes. Offline replay reads archived results without rewriting them.
Real-corpus experiments use bench:knowledge to snapshot the source database
through a read-only connection, freeze source-linked questions and candidate
pools, and obtain query embeddings from the configured local server:
mise run bench:knowledge -- prepare --qa /absolute/path/qa.jsonl --count 50
probe=/absolute/external/knowledge-TIMESTAMP # Use the printed directory.
mise run bench:decisions -- --cases "$probe/cases.json" \
--arms llm,jev,rules,rules_jev,rules_llm --repeats 1 \
--llm-url http://127.0.0.1:42069/v1 --llm-model YOUR_CURRENT_QUERY_MODEL \
--jev-model jev-1.13.0
rerank_run=/absolute/external/RUN_ID # Use the decision runner's directory.
mise run bench:knowledge -- report "$probe" "$rerank_run" > "$probe/rerank-report.json"
prepare defaults to ~/.local/share/hippo/hippo.db and
~/.config/hippo/config.toml; --source and --config override them. Use the
model and endpoint frozen in protocol.json for the decision run. report
validates frozen inputs and recorded responses before comparing hit@1/5/10,
mean reciprocal rank and latency. The metric is known-source recovery: other
candidates are unjudged, not relevance negatives. It does not measure exhaustive
relevance, answer correctness or live-query prevalence.
Prepare topic classification from those frozen pools:
mise run bench:knowledge:tags -- prepare "$probe"
Before inference, independently annotate all 50 entries in
classification-label-inputs.json using the frozen knowledge_topics.json.
Write classification-reference.json with methodology, input/taxonomy SHA256
hashes and labels rows containing UUID id, numeric node_id, unique topics,
reasons and uncertainty notes. Allow empty topic lists and exclude incidental
mentions. Do not inspect model outputs while labeling. Agent-reviewed references
must be identified as such; they are not human ground truth.
After the reranking report and reference labels exist, execute classification and apply its results only to new database clones:
mise run bench:knowledge:tags -- classify "$probe" --concurrency 4
tags_run="$probe/tags-TIMESTAMP" # Use the printed classification directory.
mise run bench:knowledge:tags -- apply "$probe" "$tags_run"
mise run bench:knowledge:tags -- evaluate "$probe" "$tags_run"
Classification compares existing tags mapped through fixed keyword rules,
declarative word/phrase rules over summary/detail, and 14 Jev Noul judgments
per node using jev-1.13.0. The primary threshold is 0.8; 0.5 is diagnostic.
classify sends redacted summary/detail text to TypeSafe with one attempt per
node. evaluate reports reference-label precision/recall, errors, timing and
retrieval recovery for each clone. Connection metrics distinguish pairs sharing
any reference topic from links supported by the same specific reference topic.
The rules_tags and jev_tags clones add tags to node tags/content and FTS;
jev_tags_vectors also re-embeds topic-enriched semantic text using the local
embedding server. All three add namespaced topic entity links. These links mean
only shared topics, not duplicate identity or evidential support. The source
database, baseline snapshot, source links and command vectors remain unchanged.
Only the candidate-pool cohort is modified. All snapshots, labels, requests,
responses, clones and reports remain under the external decision directory.
Production capture, enrichment, retrieval and model selection remain unchanged.
Contracts and question design follow TypeSafe’s HTTP API ↗, reranking cookbook ↗, and citation verification cookbook ↗.