I got interested in a specific failure: agents fill gaps in underspecified
requests silently, and you find out three commits later.
The theory for handling this was solved a long time ago. Value of information
(Howard, 1966), reject-option classification (Chow, 1970), and belief dependency
tracking via Doyle's TMS (1979) and de Kleer's ATMS (1986). Hold a belief, track
what depends on it, un-believe the lot when the premise dies.
As far as I can tell no shipped agent product implements any of it. So I built a
measurement kit first, because I wanted a number before I wanted a product.
The obvious remedy is a gate: detect when a request is ambiguous, ask the one
question whose answer changes what you'd build. I implemented it and ran 120
trials, 10 tasks x 4 policies x 3 repeats, with a spec-bound oracle and a grader
blind to which policy produced the artifact.
Verdict: no win. Nothing separated from never-asking. Every confidence interval
overlaps never-ask's [19.2, 51.2] at n=30 per policy. The one-question gate
overlaps it by 23.8 points.
The result that does not depend on that noisy success column is the cost one, and
it surprised me:
The gate asks 3.7x fewer questions and costs 25% MORE. Running the divergence
check, generating the top-two readings of a request and comparing them, is more
expensive than just asking. Against never-ask the gate runs at 2.47x baseline on
every request, including the ones it correctly waves through. A gate is normally
pitched as a cheap pre-filter that pays for itself by suppressing questions. On
this task set it is the most expensive policy that beats nothing.
Two more things worth reporting, both unflattering.
At 81% through the run, the gate was projected clear of never-ask by 4.4 points.
The last 22 trials reversed it to a 5.7 point overlap. If I had posted the
interim I would have published the opposite conclusion.
And an earlier version of the kit truncated tool results head-only. The evidence
an inference rests on is usually at the END of a file read or a log, so the
extractor saw well-supported inferences with no visible support and labelled them
"invented." My instrument for measuring assumption errors was manufacturing them.
Two further bugs after that: it ingested its own LLM calls as tasks, and I could
not filter those by session id because `claude -p` can inherit the parent session
id and append to the very transcript being measured.
Kill criteria were written before the run rather than after. Experiment 1, the
actual base rate, has no number yet and I say so in the README. Experiment 3 has
never been run.
Stdlib only, MIT. Happy to be told this exists already and I missed it.
I got interested in a specific failure: agents fill gaps in underspecified requests silently, and you find out three commits later.
The theory for handling this was solved a long time ago. Value of information (Howard, 1966), reject-option classification (Chow, 1970), and belief dependency tracking via Doyle's TMS (1979) and de Kleer's ATMS (1986). Hold a belief, track what depends on it, un-believe the lot when the premise dies.
As far as I can tell no shipped agent product implements any of it. So I built a measurement kit first, because I wanted a number before I wanted a product.
The obvious remedy is a gate: detect when a request is ambiguous, ask the one question whose answer changes what you'd build. I implemented it and ran 120 trials, 10 tasks x 4 policies x 3 repeats, with a spec-bound oracle and a grader blind to which policy produced the artifact.
Verdict: no win. Nothing separated from never-asking. Every confidence interval overlaps never-ask's [19.2, 51.2] at n=30 per policy. The one-question gate overlaps it by 23.8 points.
The result that does not depend on that noisy success column is the cost one, and it surprised me:
The gate asks 3.7x fewer questions and costs 25% MORE. Running the divergence check, generating the top-two readings of a request and comparing them, is more expensive than just asking. Against never-ask the gate runs at 2.47x baseline on every request, including the ones it correctly waves through. A gate is normally pitched as a cheap pre-filter that pays for itself by suppressing questions. On this task set it is the most expensive policy that beats nothing.Two more things worth reporting, both unflattering.
At 81% through the run, the gate was projected clear of never-ask by 4.4 points. The last 22 trials reversed it to a 5.7 point overlap. If I had posted the interim I would have published the opposite conclusion.
And an earlier version of the kit truncated tool results head-only. The evidence an inference rests on is usually at the END of a file read or a log, so the extractor saw well-supported inferences with no visible support and labelled them "invented." My instrument for measuring assumption errors was manufacturing them. Two further bugs after that: it ingested its own LLM calls as tasks, and I could not filter those by session id because `claude -p` can inherit the parent session id and append to the very transcript being measured.
Kill criteria were written before the run rather than after. Experiment 1, the actual base rate, has no number yet and I say so in the README. Experiment 3 has never been run.
Stdlib only, MIT. Happy to be told this exists already and I missed it.