Jev utility evaluation, 2026-11-13

This record gives the measurements behind one change and two decisions: the note-v1 rerank recipe, the decision to leave topic retrieval off, and the decision not to adopt a candidate classification recipe. All labels came from model judges, not from people. The human-review criteria of the approved specification ↗ remain unmet.

Data

ItemValue
SnapshotSQLite online backup of the production database, 40,617 nodes
Queries90 distinct real queries from agent transcripts (2026-10-03 to 2026-11-12), 27 projects
Toolssearch_hybrid 49, ask 34, search_knowledge 5, agent_query 2
Split51 tune, 39 holdout, assigned by a hash of the query text
PoolsUp to 30 candidates per query from retrieval.search; 87 queries had at least 2 candidates
Classification sample160 nodes, stratified by stored probability for each of the 14 topics

Each query kept the limit, project, and mode arguments of the original call. Queries with other filters were excluded.

Method

Nine judges graded every candidate of every pool (2,579 query-candidate pairs) on a 0 to 3 scale. Each judge saw the query, the project scope, and the full note text. Candidate order was shuffled, so no judge saw the order of any arm. A grade of 2 or 3 counts as relevant.

Two judges graded the 14 topics for each of the 160 nodes on a 0 to 2 scale. A grade of 2 means a substantive subject, and 1 means a minor or borderline subject.

Agreement checkPairsResult
Relevance, double-judged270Exact 0.92, relevant or not 0.93, kappa 0.78
Topics, double-judged224Substantive or not 0.996, kappa 0.96

Ranking quality is nDCG at each query’s own limit, with gain 2^grade - 1. Intervals are 95% paired bootstrap intervals over queries. Quality arms ran with a 30-second timeout, so the 2-second production deadline did not affect them.

Rerank results

Primary result: notes written before the query

The snapshot was taken after every query was asked, so a pool can hold notes that did not exist at query time. 1,064 of 2,568 pool candidates (41%) were dated after their query, and they include 353 of the 554 candidates judged relevant. Many of them record the session that issued the query. The table below scores each arm on the candidates dated before the query. It is the result to rely on.

ArmTune nDCG (n=47)Holdout nDCG (n=35)All nDCG (n=82)
Retrieval order0.6560.8260.729
multi-v1 (shared list, 90 Scores)0.7830.8130.796
Note, one Noul per candidate0.8300.9100.864
Note with identifiers (note-v1)0.8470.9060.873
Note, one 4-level Score per candidate0.8620.8900.874
Note, Noul, jev-1.14.00.8710.9010.884
DifferenceTuneHoldoutAll
multi-v1 minus retrieval order+0.127 [+0.047, +0.216]-0.013 [-0.103, +0.069]+0.067 [+0.009, +0.126]
note-v1 minus retrieval order+0.191 [+0.101, +0.287]+0.080 [+0.027, +0.136]+0.144 [+0.089, +0.204]
note-v1 minus multi-v1+0.064 [-0.005, +0.137]+0.093 [+0.008, +0.179]+0.077 [+0.022, +0.132]

All pool candidates, including later notes

ArmTune nDCG (n=49)Holdout nDCG (n=38)All nDCG (n=87)All top-1 relevant
Retrieval order0.5810.7080.6360.644
multi-v1 (shared list, 90 Scores)0.8130.8320.8220.782
Note, one Noul per candidate0.8710.8630.8680.851
Note with identifiers (note-v1)0.8830.8970.8890.885
Note, one 4-level Score per candidate0.8910.8670.8800.816
Note, Noul, jev-1.14.00.8970.8570.8800.874

The production code path (rerank_results with the configured default recipe, note-v1) was then run on the holdout queries with all pool candidates. It scored nDCG 0.895 (n=37), against 0.897 for the experiment arm on the same queries. This run shows that the shipped code reproduces the experiment arm. It is not an independent test of the recipe.

After code review, the shipped recipe gained two things: a statement in each question that the note is untrusted text, and the note’s branch. The shipped recipe was run again on the holdout queries. On notes dated before the query it scored nDCG 0.911 (n=35), against 0.906 for the experiment arm: a difference of +0.005 [-0.004, +0.016]. Its median request was 14,405 input tokens.

Request costmulti-v1Notenote-v1
Input tokens, median23,32410,93713,458
Payload bytes, median88,29640,45449,301
HTTP time, median / 95th percentile (ms)280 / 376216 / 336193 / 311

No request in any arm took more than 2,000 ms. The production deadline misses therefore came from the client side, not from Jev latency.

Retrieval order leaves room for a reranker. Of 554 relevant candidates, 316 sat below the query’s limit in retrieval order. Four of 87 pools held no relevant candidate.

Calibration and a relevance floor

The note-v1 Noul tracks judged relevance:

Noul0.0 to 0.10.2 to 0.30.4 to 0.50.5 to 0.60.7 to 0.80.9 to 1.0
Share judged relevant0.030.280.490.670.790.87

A floor would drop delivered results below a Noul value. No floor is implemented; the relevance field lets a consumer apply one.

FloorResults per queryPrecisionRelevant keptQueries left empty
None6.210.6281.0000
0.24.970.7360.9383
0.34.090.8090.8505
0.53.010.8700.67310

Topic retrieval results

All tests used the snapshot with the classification rows of 2026-10-12 restored, so 39,840 nodes had labels.

TestResult
Alias channel (topic_retrieval = true)A topic matched 5 of 90 queries. The pool changed for 2. nDCG change -0.005 [-0.014, 0.000].
Channel fed by Jev query topics53 of 87 queries had a topic at 0.7 or above. The pool changed for 29. Of 103 admitted candidates, 8 had a Noul of 0.5 or above. The channel displaced 10 candidates judged relevant.
Topic agreement as a signalAUC 0.61 for separating relevant from other candidates in a pool. The relevance Noul scored 0.94.
Topic agreement added to the NoulHoldout nDCG change between -0.006 and -0.018 for weights 0.05 to 0.4.

No tested use of the 14 topics improved retrieval.

Classification results

Accepted labels on the 160-node sample:

ArmThresholdAcceptedSubstantive precisionLenient precisionRecall of substantive
Stored production labels0.81760.5570.8580.748
Note state, earlier wording0.81820.5880.9070.817
Earlier state, per-topic questions0.81230.7150.9590.672
Note state, per-topic questions0.81280.7110.9610.695
Note state, per-topic questions (candidate recipe)0.71640.6340.9390.794

The candidate recipe was not adopted, because no reader gains from the labels. It asks one plain question per topic, with a concrete description of what counts and what does not, and it sends only the note’s summary, intent, decisions, problems, and details. The built-in recipe also sends the note’s source and node type, which is why ordinary shell notes scored high for capture-ingestion. The candidate wording is in class_new_wording.json, and the code change is in classification-v2-and-durability.patch, both in the artifact directory below.

A second defect is recorded and not fixed. A writer process that never calls classification.configure is treated as a disabled writer. When it changes the input of a classified node, it marks the row skipped with classification disabled and clears the result. A re-embed run did this to 14,635 rows on 2026-10-12. The same patch holds a fix.

Limits

Cost and artifacts

The experiment sent 11.8 million input tokens to TypeSafe, about $0.50 at $0.042 per million. Data sent: redacted query text and redacted note text, the same classes that production already sends.

All inputs, raw answers, judge packets, and labels are in ~/.local/share/hippo-bench/decisions/retro-20261113/, outside the repository and the production data directory. eval.py, class_eval.py, and analysis2.py in that directory reproduce every table from the stored answers without a Jev call.