Hippo agent-benefit study v2
Status: HIPO-50 execution contract. The September 28 protocol and all original
diagnostic artifacts remain historical records. The machine-readable contract
is config/agent-benefit-v2.json. Its content hash identifies each study and
phase separately. Changes create a new identity and require fresh acceptance
families. Owner: carpenter. The executor freezes inputs and records failures;
independent reviewers establish judgment outcomes from complete sources.
Questions and eligibility
Availability estimates the effect of offering normal read-only Hippo tools. Zero use is a valid observed behavior. Guided use estimates the effect under the contract’s prescribed lookup policy. Queries derive only from the full request and original project path. The control receives the same conditional instructions and ordinary tools. Do not pool these studies. The first known-history guided pair is a diagnostic, excluded from both population estimates and acceptance.
Diagnostic task validity and corpus coverage
The CI-removal case is retained as a plumbing diagnostic. Its checker requires deleting the Pi test and removing its CI step. Its four audited memories concern earlier fixes that retained structural, pinning and theme checks. These sources establish related prior work, but the case does not establish that applying a historical fact changes the scored decision or prevents a measured mistake. It is unsuitable as the sole usefulness canary, even after query repair.
Use three distinct measurements:
- Deliberate known-history diagnostics select actionable facts from the frozen corpus. Audit original sources before running either arm. Specify the fact, expected application, independent scored check and advantage over the starting context. Examples include a prior failed approach, an external-state fact or a recorded constraint unavailable in the checkout. Keep the fact and expected action out of both agents’ prompts. Score retrieval, supported interpretation, action, checked outcome, repeated failures, time and tokens separately. Treatment must retrieve and correctly apply an audited fact before the mechanism is qualified. A completion tie does not prove benefit; improvement can be measured in a preregistered avoided mistake or effort endpoint.
- Deliberate no-history controls test unsupported claims, inappropriate actions and lookup overhead. Corpus absence must be independently audited. An empty retrieval result is not proof that relevant history is absent.
- Consecutive real tasks measure practical coverage and average benefit. Audit actionable history independently of retrieval and arm outcomes. Report present, absent and unknown counts. Report retrieval and correct application within the history-present group, with checked paired outcomes and effort. Keep tasks without history in the overall population estimate. Deliberately selected diagnostics cannot establish ordinary-task coverage or average benefit.
An all-zero application result on known-history diagnostics fails the mechanism gate. Zero history-present cases in a consecutive sample provides no conditional usefulness evidence; report that sample’s coverage and do not interpret it as a test of retrieval/application when usable history exists. Unknown source labels remain unknown. Corpus size and tool-call counts do not measure actionable coverage. Do not select cases using observed treatment wins.
The diagnostic input verifier now requires a source-bound history_opportunity
with its fact, expected application, scored check and advantage over the starting
context. This validates the audit’s completeness, not the truth of its claims.
Independent source and checker review remain necessary. The old CI-removal
spec lacks this evidence and cannot qualify as a usefulness canary. Its original
records remain unchanged. The changed contract creates a new content identity
without replenishing any model-run allowance.
Enroll consecutive real human maintenance and investigation requests from Hippo-captured projects in the contract’s UTC window. Preserve every candidate, timestamp, full request and attachment hash, original project, family, eligibility decision and reason. One primary request per independent incident family is eligible when both starting environments can be reset and an independent success check exists. Harness briefs, review-subagent prompts, irreversible external actions and unreconstructable states are exclusions. Relevant history, expected success and apparent difficulty are not filters. Evaluation requests in this session are not independent product tasks.
Freeze memory before requests and case investigation. Reuse
agent_benefit_intake.py; reject dirty or mutable starting environments.
Executable outcomes take precedence over unsupported self-reports. Missing
material evidence prevents adjudication and leaves the case in the frame.
Execution failures and timeouts are failed completion outcomes. Pre-thread
harness failures are recorded separately and stop expansion.
Resources and stops
The diagnostic permits one pair, two runs, 600 seconds per arm, 1,200 summed arm seconds, two million reported tokens and two reviewer hours. Development targets 20 independent pairs, at most 40 runs, four summed arm hours, 25 million reported tokens and ten reviewer hours. Acceptance is conditional, capped at 200 pairs, 400 runs, 24 summed arm hours, 250 million reported tokens and 40 reviewer hours. These caps cover both study policies together, not a fresh allowance for each. Resource reservations include failed launches. Wall and run caps are hard launch/process bounds. Token accounting includes cached input and stops at the first usage update reaching the remaining allowance; one in-flight response may overshoot. Missing usage stops further launches. There is no claim of a strict provider-side token limit or measured dollar cost. Record overshoot and actual usage, and never replenish automatically.
The consecutive window ends October 7 at 08:00 UTC. Stop at the first failed canary, an exhausted resource cap, unsafe action or recipe change. Insufficient tasks produce a terminal inconclusive result. Missing required human labels produce a hold. No automatic extensions, outcome-based exclusions or conversion of diagnostics to acceptance are permitted.
Supply, analysis and readiness
The historical screen supplied five runnable cases from 64 candidate groups, about 7.8 percent. These are development material, not prospective supply. The completed ten-arm pilot consumed 1,313.057 summed seconds and 5,052,392 input tokens. It made zero Hippo calls. Those measurements suggest a small canary is affordable; they do not establish acquisition rate, valid scoring, current model access or future resource use. Human reviewer supply is not established. Begin prospective intake without filling it with synthetic tasks.
Verified completion is the primary endpoint. Reuse agent_benefit.py for
wins, losses, ties and its conservative exact interval. Retain the five-point
minimum observed gain, positive lower 95 percent bound, ten-point sizing
effect and 80 percent power target. A 100-family minimum alone is insufficient.
Existing scenarios need 500 to 1,000 families, beyond this execution budget.
Choose a feasible fixed acceptance size only after development estimates
discordance, acquisition rate and review effort. If power and resources cannot
both be supported, record a negative feasibility decision. Do not relax a
tolerance after outcomes to obtain a passing result.
Record source-history existence, retrieval, factual support, interpretation, application and independently checked completion separately. Unknown remains unknown. Report elapsed time, failed/repeated actions, tool calls, input, cached input, output tokens and measured cost including lookup overhead. No-history and stale/conflict controls are separate from primary tasks. Severe safety harm holds readiness irrespective of completion gains.
Retain the separate accuracy gate: 300 consecutive factual agent calls, 100 answerable independent primary families, 100 independently audited absent controls, two independent human reviewers, joint-success lower 97.5 percent bound at least 90 percent and absent-failure upper bound at most 5 percent. The Pi and Codex historical censuses have no qualifying human source labels and cannot satisfy the prospective gate. Bob and Carlos remain AI diagnostics.
Software regression tests, actual deployment and final release-SHA CI remain separate readiness requirements. No study outcome authorizes a release tag or deployment. Private history and raw traces remain outside Git and Kaneo.
September 30 execution evidence
The source-bound packet export, adjudicator, study validator and bounded
app-server runner were implemented at bee491b. The authorized Bob v2 and
Carlos v2 reviewers each received only an independently ordered ZIP and its
instructions in a fresh session. Both identities are recorded as AI reviewers.
Their 28 paired item judgments contain three disagreements. All six aggregate
submission outcomes remain unresolved, and none qualifies for acceptance.
The original 125 artifact hashes were rechecked without changes.
Both reviewers pass all Gringotts rubric items against the corrected complete tracked source. A redacted required source file still prevents complete packet qualification. Firstmate lacks historical installed state and execution evidence. A subsequent anonymous executable gcampr check confirms that A creates a PR on the default branch when its HEAD is ahead of the remote. Both candidates invoke gcamp before checking the default-branch condition. These checks use mocked commands and preserve the original reviewer forms.
The frozen diagnostic reservation was consumed. The control completed its
task and passed the independently checked removal criterion in 209.140 seconds,
with 600,536 reported total tokens. The treatment stopped during runtime
inventory verification after 11.656 seconds, before any model turn or Hippo
call. Codex 0.159.2 injected creative_production_mcp despite apps=false.
The pair is a failed diagnostic, not a completion comparison or benefit result.
The runner now disables plugins and remote_plugin explicitly. A subsequent
12.520-second treatment-only preflight verified exactly the normal 12 Hippo
tools and the configured filesystem read/write probes. It started no model
turn and does not repair or replace the failed pair. The probes establish only
their configured boundaries; they do not prove denial of every possible host
history location. HIPO-51 therefore remains incomplete. No cohort expansion or
release-readiness claim is supported by this execution.
Private evidence is retained under
~/.local/share/hippo-bench/agent-benefit/2026-09-30-v2-execution/, including
the raw reviews, provenance registry, packet and label hashes, adjudication,
anonymous gcampr checker, original paired RPC traces and plugin-isolation
preflight. The original reservation remains in place. Further model execution
requires an explicit versioned diagnostic amendment and a complete boundary
audit. Prospective population supply and the required human labels are still
unestablished.
The subsequent host-history audit found that both frozen trial configurations
omit an explicit Pi session-history denial. An additional no-inference OS
probe exited with status 1 because opening host .zsh_history succeeded.
No history contents were read or emitted. The earlier boundary qualification
therefore covers only the original listed probes. It cannot qualify complete
history isolation, and the original control outcome cannot support an isolated
benefit comparison. The runner’s input verifier now independently requires
denials for Pi sessions, Claude projects, Codex history, Hippo storage, zsh
history and bash history. Omitting a path from a submitted spec cannot bypass
this check. The original configuration and failed audit remain preserved.
An independently copied treatment configuration with the six explicit denials
passed a real 9.087-second preflight. The OS probe denied opening one existing
file from each of the six history locations, and denied the private write
probe. Runtime inventory still contained exactly the 12 Hippo tools. No model
turn started. This verifies the corrected tested boundaries without replacing
the original diagnostic or authorizing cohort expansion.
The existing September 29 clean prospective memory snapshot was rechecked:
its file hash matches its recorded manifest, its schema is 25, and its recorded
repository freeze is ready. It has no attached task and remains
snapshot_ready=false; the memory itself can serve only requests after its
recorded cutoff. A read-only inventory of the active enrollment window found
35 captured segments in 11 session groups. These counts are not independent
task-family counts. Full captured prompt records and their hashes are retained
privately under prospective-intake-audit/. Prompt origin and individual time,
incident-family independence, historical starting state and independent task
checkers have not been audited. No prospective primary family is qualified by
this inventory. Diagnostic amendment A is prepared separately and remains
pending owner approval because the original contract prohibits automatic
resource extensions.
The source-provenance audit of those 11 session groups excludes six automated
sources: two Codex review subagents and four Firstmate worker sessions. Their
captured database rows all have is_subagent=0; original Codex spawn metadata
and Firstmate launch briefs contradict using that flag as proof of human
origin. Request-level timestamps in the remaining five sources yield 16
window records. Fourteen are excluded as automatic continuations, watcher
wakes, evaluation reviewer arrangements or a status-only continuation. Two
requests remain unresolved pending full context, incident independence,
resettable starting state and checker verification. Zero primary families are
qualified. This is a bounded audit of the original source inventory, not a
complete census of the evolving window. The source hashes, line references,
timestamps and exclusion reasons remain in private intake artifacts.
The two unresolved follow-ups were subsequently linked to their full original
benchmark request at source line 966, timestamp September 30 at 05:48:29.239
UTC. This precedes the 08:00 UTC enrollment boundary. They continue the same
benchmark incident and are excluded under the one-primary-request-per-family
rule. The version 2 private disposition report retains the original request,
timestamp, source hash and line reference. All 16 audited window records are
now excluded, with zero qualified new primary families in this bounded source
inventory. The original unresolved report remains preserved. This does not
establish absence of eligible tasks elsewhere in the live capture database.
The owner subsequently approved diagnostic amendment A. Its corrected pair
completed both task checkers within 166.344 summed seconds and 657,464 reported
tokens. The treatment attempted one agent_query, denied by Codex’s MCP tool
approval policy before execution. No Hippo result reached the model. The pair
therefore fails canary qualification despite successful task completion. The
approved additional reservation is consumed; further attempts are not
authorized. The report’s erroneous call-presence qualification is preserved
with a hash-bound correction, and the runner now rejects failed calls as
delivery proof. The amendment document records the full current evidence.