Agent-benefit diagnostic amendment A
Status: owner approved with the user message “approve”. This amendment changes the diagnostic resource allowance only. It does not qualify acceptance or authorize a cohort.
The original v2 allowance of one pair is consumed. Its control completed, its treatment failed before inference because an unexpected plugin was connected, and a subsequent audit found accessible host shell history. Both attempts, configurations, resource usage and the original reservation remain preserved.
The requested additional allowance is exactly one corrected pair, two attempts, 600 seconds per arm, 1,200 summed arm seconds and two million reported tokens. No automatic retries or further allowances follow. Stop on inventory mismatch, failed boundary probe, transport failure, timeout, missing usage, model reroute or unsuccessful independent task check. Token monitoring retains the disclosed possibility of one in-flight response overshoot. No dollar cost is established.
Use fresh independent workspaces from the frozen task commit and the existing pre-task memory snapshot. Keep model gpt-6-sol, medium effort, Codex 0.159.2, the same full request, guided lookup policy and independent checker. Disable apps, multi_agent, plugins and remote_plugin. Require an actual per-thread MCP inventory of zero servers in control and exactly the normal 12 Hippo tools in treatment. Require independent denial checks for Pi sessions, Claude projects, Codex history, Hippo storage, zsh history and bash history. Probe actual existing history files without reading or emitting their contents. Check private write denials. Retain task workspace access and checker access.
Approval must be recorded before reserving or starting either model turn. Freeze an amended contract with a distinct content identity and an explicit reference to the original contract and failed diagnostic. Preserve the original reservation. The new pair remains diagnostic and its reused family is excluded from population benefit estimates and acceptance. Development and acceptance budgets, populations, thresholds and human-label requirements are unchanged.
A completed pair is insufficient on its own: verify the treatment’s actual Hippo invocation, delivered source identities, pre-task factual support, interpretation and application against its trace and independently audited sources. Missing source support prevents canary qualification. Completion in both arms establishes task feasibility, not population benefit.
The pending private contract and spec are now prepared. Both fresh starting repositories are clean at the original frozen commit. Each Codex home initially contains only copied auth, frozen config and an empty temporary directory. Both arms pass static verification. Actual no-inference preflights passed for control in 2.973 seconds and treatment in 8.255 seconds. Their per-thread MCP inventories and configured OS read/write denials passed. Neither started a model turn. Original frozen inputs remain unchanged. The runner rejects a pending amendment before creating its resource reservation. These preflights occurred before approval and started no model turns.
Prepared inputs and preflight traces are private under
~/.local/share/hippo-bench/agent-benefit/2026-09-30-v2-execution/.
The spec hash is
20ececcd2966e1363f3258e5ebf968255322bc19b8052538fd558f0fc3a6e210.
An approved contract needs a recorded owner decision and a new content hash;
the pending contract must remain preserved.
The owner decision is recorded privately in amendment-a-approval.json.
The approved spec hash is
510c150df3545703dcdd02c9f9f240aec41a5c1ae93a43d861d3de2380f2f69a.
Its guided diagnostic study identity is
2a4b963b8b56a4239173a6154057fecb6d1371cc0867c8222c767d094e7ceb16.
The runner reserved the approved two attempts and started the control first.
Pending inputs and the original failed diagnostic remain preserved.
The approved pair finished both attempts. Control completed and passed the
independent checker in 95.560 seconds with 255,829 reported total tokens.
Treatment completed and passed in 70.784 seconds with 401,635 tokens. Summed
usage was 166.344 seconds and 657,464 tokens, within the approved limits.
Treatment attempted agent_query once. Codex rejected it with
“MCP tool call requires approval, but approval policy is never”. No Hippo
result was delivered. Neither task completion nor the observed elapsed-time
difference estimates population benefit for this reused diagnostic family.
The original report incorrectly marked the model-thread path verified merely
because a tool call was present. It is retained unchanged. A separate
amendment-a-run/qualification-correction.json binds the original report hash
and records path, source support, canary and expansion qualification as false.
The runner now requires a completed agent_query with a non-error result to
verify delivery. Its regression test rejects an approval-denied call even when
both task checkers pass. Ten runner tests and Python lint pass.
The approved allowance is consumed. No further model attempts are authorized. Tool-level approval configuration must be resolved before proposing another trial. Inventory discovery and OS boundary probes do not test permission to invoke a model tool. Readiness remains unproven.
The owner then requested the configuration change and an invocation test.
A separate permission diagnostic used the explicit per-tool setting
mcp_servers.hippo.tools.agent_query.approval_mode = "approve", with global
approval_policy = "never" retained. It started one model turn, bounded at
120 seconds and 100,000 reported tokens, and requested exactly one query with
no file edits. The model’s agent_query completed with no error and returned
an evidence-mode response. The turn completed in 23.100 seconds using 56,464
reported tokens. This verifies actual invocation under the corrected policy;
it does not replace either paired diagnostic or estimate agent benefit.
Private trace and result are retained under agent-query-permission-test/.
The owner authorized proceeding with one corrected pair using the tested
permission setting. Amendment B has a separate approved contract and resource
reservation. Its spec hash is
f5ea080e3c054b4f7e113c421d2637ddded454ce45a16ab3215f466a753f0b80.
The frozen inputs passed static verification. All 29,626 knowledge nodes in its
memory snapshot predate the original task timestamp.
Both task checkers passed. Control completed in 46.153 seconds using 226,522
reported tokens. Treatment completed in 71.368 seconds using 577,547 tokens.
The summed 117.521 seconds and 804,069 tokens are within the authorized limits.
Treatment successfully invoked agent_query and received five evidence hits.
None matches the four independently pre-audited relevant nodes. The treatment
stated that the returned older CI failures did not identify the failing test,
then solved it using the checkout. Its additional get_ci_status call was
denied because that tool had no explicit approval. Factual support for the
returned hits has not been independently adjudicated.
For this reused case, treatment took 25.215 seconds longer and used 351,025 more
reported tokens. Completion was tied. This demonstrates a working model query
path and a relevance miss in the tested lookup policy. It does not estimate
population benefit, prove general harm or establish successful source
application. Canary qualification and expansion remain false. Both approved
attempts are consumed. The private hash-bound amendment-b-run/mechanism-audit.json
retains delivered identities, pre-audited identities, timing checks, model
interpretation, failed-call evidence and completion/resource measurements.
Further diagnosis identified evaluation-instruction contamination in the
recorded query. The model appended the lookup instructions and original
project-path scaffolding to the human request. A read-only replay of that exact
query reproduced all five delivered identities in order. On the same snapshot,
filters, retrieval settings and clock, the original human request alone
retrieved two of the four pre-audited relevant nodes in its top five.
This corrects the interpretation of amendment B: it demonstrates a miss for
the contaminated query, not a miss for the prescribed original-request query.
The private query-fidelity-correction.json binds both replays and the original
mechanism audit by SHA-256. The original results and resource costs remain
unchanged. No model turn was started for these replays.
The current contract now specifies exact supplied JSON arguments and excludes
evaluation instructions from the query. Its content hash creates a distinct
study identity; previously frozen contracts are unchanged. The runner requires
a separately hashed original-task.txt in both arms, constructs equal prompts
with the exact arguments, and records query_policy_verified only for one
matching agent_query. Tool delivery and query adherence remain separate
measurements. The regression check rejects contaminated queries, wrong project
paths, missing calls and duplicate calls. Previous specs fail static validation
because they do not freeze the separate original task. The correction does not
authorize another pair or establish agent benefit.
The owner subsequently authorized a trial of the revised task criteria. Amendment C permits one additional pair under the same limits of 600 seconds per arm, 1,200 summed seconds and two million reported tokens. The task is a deliberately constructed policy-recovery exercise grounded in the real pre-task .dotfiles corpus. Five proposed checks must be retained or dropped according to the earlier Pi CI decisions. It is one diagnostic family, not five independent tasks, and is excluded from population estimates.
Two original Codex sessions support the frozen answer key. The earlier repair accepted local paths, Git sources and package objects, retained exact npm pins and theme-reference checks, and removed assertions for fixed model/provider preferences. A fresh single-commit repository contains only the proposals and a neutral README. Neither arm receives the historical decisions or answer key. The checker scores five decisions identically in both arms and does not require citations. Returned sources, explanations and the recalled earlier failure are audited separately for supported application. Correct citations alone cannot produce a completion win.
This trial changes the task-checker stop rule: a completed arm with a verified
incorrect task result (checker exit 1) remains in the pair and does not prevent
the other arm from running. Checker errors (exit 2 or higher, execution failure
or timeout), failed runtime isolation, transport failure and missing usage
still stop execution. Tool delivery qualification is independent of task
success. The regression check verifies both continued treatment after a control
task failure and stopping after a checker error. Original runs and contracts
remain unchanged. The approved private spec and contract are retained as
known-history-spec.approved.json and known-history-contract.approved.json.
Amendment C completed both model turns within the authorized bounds. Exact query adherence and successful Hippo delivery passed. Control recovered zero of five decisions and marked all five unknown. It made no incorrect policy claims. Treatment recovered two decisions correctly, made one incorrect decision and marked two unknown. Neither passed the full five-decision checker.
Treatment correctly retained theme-reference validation and dropped exact
model/provider assertions, citing codex-18346. Its trace explicitly applies
the retrieved record to those proposals. The cited original source confirms
both choices. Treatment incorrectly dropped exact npm pins by conflating
removal of fixed-version preference assertions with removal of pin validation.
The original source states that package pinning was retained and unpinned
variants were rejected. The delivered compact summary mentions both pinning
and removal of exact-version assertions without explaining that distinction.
Its evidence excerpt begins with session instructions rather than the decisive
work record. The trace demonstrates an incorrect interpretation; it does not
isolate the relative contribution of summary compression and model reasoning.
Four of five delivered hits describe routine workflow successes. The fifth is
one of the four audited relevant nodes. The source establishing local/Git/object
compatibility was not delivered. Treatment marked those two proposals unknown
and attempted three search_knowledge calls. All three were denied because
the trial configuration explicitly approved only agent_query, while global
approval remained never. This restricts the result to the tested permission
policy; it does not evaluate unrestricted read-only Hippo follow-up use.
Control took 35.303 seconds and 94,217 reported tokens. Treatment took 67.983 seconds and 248,937 tokens. Summed use was 103.286 seconds and 343,154 tokens; treatment added 32.680 seconds and 154,720 tokens. Token totals include cached input and do not establish dollar cost. Both approved attempts are consumed.
This diagnostic establishes corpus delivery and two source-supported policy
applications. It also exposes one wrong policy action, incomplete relevant
retrieval and blocked follow-up inspection. It is not a successful full-task
benefit result. Canary qualification and expansion remain false. The private
known-history-run/mechanism-audit.json binds the traces, checkers, answer key
audit, delivered-source audit and outputs by SHA-256. Original records remain
unchanged. One selected recovery task does not estimate ordinary-task coverage
or population benefit.
The owner subsequently requested fixes and an MCP-specific usefulness measure. Amendment D reserved one fresh pair on the unchanged policy-recovery task and corpus. Treatment passed all five decisions with source-supported reasons; control marked all five unknown. All four treatment follow-up searches were allowed. Treatment added 34.148 seconds and 209,936 reported tokens. The MCP transport replay and credential diagnosis are recorded in Hippo MCP context evaluation. This is one repeated known-history diagnostic family. It qualifies the tested delivery/application mechanism, not population benefit or HIPO-55 acceptance.