Agent-benefit diagnostic amendment A

Status: owner approved with the user message “approve”. This amendment changes the diagnostic resource allowance only. It does not qualify acceptance or authorize a cohort.

The original v2 allowance of one pair is consumed. Its control completed, its treatment failed before inference because an unexpected plugin was connected, and a subsequent audit found accessible host shell history. Both attempts, configurations, resource usage and the original reservation remain preserved.

The requested additional allowance is exactly one corrected pair, two attempts, 600 seconds per arm, 1,200 summed arm seconds and two million reported tokens. No automatic retries or further allowances follow. Stop on inventory mismatch, failed boundary probe, transport failure, timeout, missing usage, model reroute or unsuccessful independent task check. Token monitoring retains the disclosed possibility of one in-flight response overshoot. No dollar cost is established.

Use fresh independent workspaces from the frozen task commit and the existing pre-task memory snapshot. Keep model gpt-6-sol, medium effort, Codex 0.159.2, the same full request, guided lookup policy and independent checker. Disable apps, multi_agent, plugins and remote_plugin. Require an actual per-thread MCP inventory of zero servers in control and exactly the normal 12 Hippo tools in treatment. Require independent denial checks for Pi sessions, Claude projects, Codex history, Hippo storage, zsh history and bash history. Probe actual existing history files without reading or emitting their contents. Check private write denials. Retain task workspace access and checker access.

Approval must be recorded before reserving or starting either model turn. Freeze an amended contract with a distinct content identity and an explicit reference to the original contract and failed diagnostic. Preserve the original reservation. The new pair remains diagnostic and its reused family is excluded from population benefit estimates and acceptance. Development and acceptance budgets, populations, thresholds and human-label requirements are unchanged.

A completed pair is insufficient on its own: verify the treatment’s actual Hippo invocation, delivered source identities, pre-task factual support, interpretation and application against its trace and independently audited sources. Missing source support prevents canary qualification. Completion in both arms establishes task feasibility, not population benefit.

The pending private contract and spec are now prepared. Both fresh starting repositories are clean at the original frozen commit. Each Codex home initially contains only copied auth, frozen config and an empty temporary directory. Both arms pass static verification. Actual no-inference preflights passed for control in 2.973 seconds and treatment in 8.255 seconds. Their per-thread MCP inventories and configured OS read/write denials passed. Neither started a model turn. Original frozen inputs remain unchanged. The runner rejects a pending amendment before creating its resource reservation. These preflights occurred before approval and started no model turns.

Prepared inputs and preflight traces are private under ~/.local/share/hippo-bench/agent-benefit/2026-09-30-v2-execution/. The spec hash is 20ececcd2966e1363f3258e5ebf968255322bc19b8052538fd558f0fc3a6e210. An approved contract needs a recorded owner decision and a new content hash; the pending contract must remain preserved.

The owner decision is recorded privately in amendment-a-approval.json. The approved spec hash is 510c150df3545703dcdd02c9f9f240aec41a5c1ae93a43d861d3de2380f2f69a. Its guided diagnostic study identity is 2a4b963b8b56a4239173a6154057fecb6d1371cc0867c8222c767d094e7ceb16. The runner reserved the approved two attempts and started the control first. Pending inputs and the original failed diagnostic remain preserved.

The approved pair finished both attempts. Control completed and passed the independent checker in 95.560 seconds with 255,829 reported total tokens. Treatment completed and passed in 70.784 seconds with 401,635 tokens. Summed usage was 166.344 seconds and 657,464 tokens, within the approved limits. Treatment attempted agent_query once. Codex rejected it with “MCP tool call requires approval, but approval policy is never”. No Hippo result was delivered. Neither task completion nor the observed elapsed-time difference estimates population benefit for this reused diagnostic family.

The original report incorrectly marked the model-thread path verified merely because a tool call was present. It is retained unchanged. A separate amendment-a-run/qualification-correction.json binds the original report hash and records path, source support, canary and expansion qualification as false. The runner now requires a completed agent_query with a non-error result to verify delivery. Its regression test rejects an approval-denied call even when both task checkers pass. Ten runner tests and Python lint pass.

The approved allowance is consumed. No further model attempts are authorized. Tool-level approval configuration must be resolved before proposing another trial. Inventory discovery and OS boundary probes do not test permission to invoke a model tool. Readiness remains unproven.

The owner then requested the configuration change and an invocation test. A separate permission diagnostic used the explicit per-tool setting mcp_servers.hippo.tools.agent_query.approval_mode = "approve", with global approval_policy = "never" retained. It started one model turn, bounded at 120 seconds and 100,000 reported tokens, and requested exactly one query with no file edits. The model’s agent_query completed with no error and returned an evidence-mode response. The turn completed in 23.100 seconds using 56,464 reported tokens. This verifies actual invocation under the corrected policy; it does not replace either paired diagnostic or estimate agent benefit. Private trace and result are retained under agent-query-permission-test/.

The owner authorized proceeding with one corrected pair using the tested permission setting. Amendment B has a separate approved contract and resource reservation. Its spec hash is f5ea080e3c054b4f7e113c421d2637ddded454ce45a16ab3215f466a753f0b80. The frozen inputs passed static verification. All 29,626 knowledge nodes in its memory snapshot predate the original task timestamp.

Both task checkers passed. Control completed in 46.153 seconds using 226,522 reported tokens. Treatment completed in 71.368 seconds using 577,547 tokens. The summed 117.521 seconds and 804,069 tokens are within the authorized limits. Treatment successfully invoked agent_query and received five evidence hits. None matches the four independently pre-audited relevant nodes. The treatment stated that the returned older CI failures did not identify the failing test, then solved it using the checkout. Its additional get_ci_status call was denied because that tool had no explicit approval. Factual support for the returned hits has not been independently adjudicated.

For this reused case, treatment took 25.215 seconds longer and used 351,025 more reported tokens. Completion was tied. This demonstrates a working model query path and a relevance miss in the tested lookup policy. It does not estimate population benefit, prove general harm or establish successful source application. Canary qualification and expansion remain false. Both approved attempts are consumed. The private hash-bound amendment-b-run/mechanism-audit.json retains delivered identities, pre-audited identities, timing checks, model interpretation, failed-call evidence and completion/resource measurements.

Further diagnosis identified evaluation-instruction contamination in the recorded query. The model appended the lookup instructions and original project-path scaffolding to the human request. A read-only replay of that exact query reproduced all five delivered identities in order. On the same snapshot, filters, retrieval settings and clock, the original human request alone retrieved two of the four pre-audited relevant nodes in its top five. This corrects the interpretation of amendment B: it demonstrates a miss for the contaminated query, not a miss for the prescribed original-request query. The private query-fidelity-correction.json binds both replays and the original mechanism audit by SHA-256. The original results and resource costs remain unchanged. No model turn was started for these replays.

The current contract now specifies exact supplied JSON arguments and excludes evaluation instructions from the query. Its content hash creates a distinct study identity; previously frozen contracts are unchanged. The runner requires a separately hashed original-task.txt in both arms, constructs equal prompts with the exact arguments, and records query_policy_verified only for one matching agent_query. Tool delivery and query adherence remain separate measurements. The regression check rejects contaminated queries, wrong project paths, missing calls and duplicate calls. Previous specs fail static validation because they do not freeze the separate original task. The correction does not authorize another pair or establish agent benefit.

The owner subsequently authorized a trial of the revised task criteria. Amendment C permits one additional pair under the same limits of 600 seconds per arm, 1,200 summed seconds and two million reported tokens. The task is a deliberately constructed policy-recovery exercise grounded in the real pre-task .dotfiles corpus. Five proposed checks must be retained or dropped according to the earlier Pi CI decisions. It is one diagnostic family, not five independent tasks, and is excluded from population estimates.

Two original Codex sessions support the frozen answer key. The earlier repair accepted local paths, Git sources and package objects, retained exact npm pins and theme-reference checks, and removed assertions for fixed model/provider preferences. A fresh single-commit repository contains only the proposals and a neutral README. Neither arm receives the historical decisions or answer key. The checker scores five decisions identically in both arms and does not require citations. Returned sources, explanations and the recalled earlier failure are audited separately for supported application. Correct citations alone cannot produce a completion win.

This trial changes the task-checker stop rule: a completed arm with a verified incorrect task result (checker exit 1) remains in the pair and does not prevent the other arm from running. Checker errors (exit 2 or higher, execution failure or timeout), failed runtime isolation, transport failure and missing usage still stop execution. Tool delivery qualification is independent of task success. The regression check verifies both continued treatment after a control task failure and stopping after a checker error. Original runs and contracts remain unchanged. The approved private spec and contract are retained as known-history-spec.approved.json and known-history-contract.approved.json.

Amendment C completed both model turns within the authorized bounds. Exact query adherence and successful Hippo delivery passed. Control recovered zero of five decisions and marked all five unknown. It made no incorrect policy claims. Treatment recovered two decisions correctly, made one incorrect decision and marked two unknown. Neither passed the full five-decision checker.

Treatment correctly retained theme-reference validation and dropped exact model/provider assertions, citing codex-18346. Its trace explicitly applies the retrieved record to those proposals. The cited original source confirms both choices. Treatment incorrectly dropped exact npm pins by conflating removal of fixed-version preference assertions with removal of pin validation. The original source states that package pinning was retained and unpinned variants were rejected. The delivered compact summary mentions both pinning and removal of exact-version assertions without explaining that distinction. Its evidence excerpt begins with session instructions rather than the decisive work record. The trace demonstrates an incorrect interpretation; it does not isolate the relative contribution of summary compression and model reasoning.

Four of five delivered hits describe routine workflow successes. The fifth is one of the four audited relevant nodes. The source establishing local/Git/object compatibility was not delivered. Treatment marked those two proposals unknown and attempted three search_knowledge calls. All three were denied because the trial configuration explicitly approved only agent_query, while global approval remained never. This restricts the result to the tested permission policy; it does not evaluate unrestricted read-only Hippo follow-up use.

Control took 35.303 seconds and 94,217 reported tokens. Treatment took 67.983 seconds and 248,937 tokens. Summed use was 103.286 seconds and 343,154 tokens; treatment added 32.680 seconds and 154,720 tokens. Token totals include cached input and do not establish dollar cost. Both approved attempts are consumed.

This diagnostic establishes corpus delivery and two source-supported policy applications. It also exposes one wrong policy action, incomplete relevant retrieval and blocked follow-up inspection. It is not a successful full-task benefit result. Canary qualification and expansion remain false. The private known-history-run/mechanism-audit.json binds the traces, checkers, answer key audit, delivered-source audit and outputs by SHA-256. Original records remain unchanged. One selected recovery task does not estimate ordinary-task coverage or population benefit.

The owner subsequently requested fixes and an MCP-specific usefulness measure. Amendment D reserved one fresh pair on the unchanged policy-recovery task and corpus. Treatment passed all five decisions with source-supported reasons; control marked all five unknown. All four treatment follow-up searches were allowed. Treatment added 34.148 seconds and 209,936 reported tokens. The MCP transport replay and credential diagnosis are recorded in Hippo MCP context evaluation. This is one repeated known-history diagnostic family. It qualifies the tested delivery/application mechanism, not population benefit or HIPO-55 acceptance.