Your Agent Says "Done." I Built a Tiny Thing That Checks If It Actually Did It.

A zero-dependency, open, deterministic verifier that catches the silent failures your AI agent never tells you about — and makes it fix them.

What worked, what broke, and the thing I didn't see coming: on a live benchmark, the agent ignored 84% of the corrections — until I changed one paragraph of wording.

Here’s a failure mode that should scare you more than it does.

You ask an AI agent to do something — save a record, send an email, write a file. The tool runs. It returns {"status": "success"}. The agent reads that, says “Done! I’ve saved your note,” and moves on.

Except the note was never saved. The tool returned success and changed nothing. The agent has no idea. You have no idea. The task is silently, invisibly broken.

The community has a name for this now: “returns 200 and is wrong.” And it’s not rare. On τ-bench — a benchmark of real-world tool-use tasks — state-of-the-art function-calling agents complete under 50% of tasks, and consistency in retail drops below 25% when you ask for the same task eight times. The agent reports success far more often than it achieves it.

I kept hitting this running multiple agent harnesses every day. So I built a small thing to catch it. It’s called TrueCall, it’s open source, and this is the story of building it.

The thing it does

TrueCall is a runtime layer that verifies whether a tool call actually achieved its real-world intent — not whether it returned a success-shaped blob.

You declare a contract for a tool: a cheap, deterministic post-condition. “After create_file, a non-empty file must exist at that path.” After the tool runs, TrueCall checks the predicate. Pass → the result flows through untouched. Fail → the agent gets a structured correction (expected this, found that, here’s how to fix it) instead of a false success.

Lesson 1: Prove the gap is real before you build for a week

Before writing product code, I made myself answer one question adversarially: does this already exist? I mapped every neighbor. Most things look adjacent but aren’t:

• Guardrails / output validators check the shape of output, not whether the world changed.

• Pre-execution gates answer “should this run?”, not “did it work?”

• Offline benchmarks compare end-state to goal-state — in a test harness, not live.

• Confidence scoring asks “does the model think it’s right?”, not “did the file get written?”

The closest commercial thing, Cleanlab’s trust layer, has proven verification can cut agent failure roughly in half — which is validation, not discouragement. My angle is the complement: open, and deterministic (query real state, assert a predicate) rather than closed and confidence-based.

The adversarial pass paid for itself: one research pass confidently told me a paper I wanted to cite didn’t exist. I checked the actual listing — it existed; the agent was wrong. That’s the whole reason you verify before you commit.

Lesson 2: A good contract format is boring on purpose

One rule: deterministic first. No “ask another LLM if it looks right.” A contract is plain data:

contract({
  tool: "create_file",
  post: { check: "file_exists", path: "{{args.path}}", minSize: 1 },
})

Four built-in checks plus a custom verify escape hatch. The railing I held: fail closed. If a verifier can't produce a verdict, that's "could not verify," never a silent pass. An unverifiable call is not a trusted call.

Lesson 3: Constraints make better software

I built this on a locked-down machine: no external npm. Annoying — but it forced a discipline that became a feature: the core has zero dependencies. A verification layer you put in front of your agent shouldn’t drag a dependency tree behind it.

It also produced my favorite war story. The plan said “use Node’s built-in type stripping.” I checked — the flag was recognized — and built on it. Then: ERR_NO_TYPESCRIPT. This Node build was compiled without TS support; the flag is accepted but absent. The “yes, supported” check was a false positive because I read the wrong signal. Verify against the thing actually running.

Lesson 4: The wow moment is 30 seconds and it’s the whole pitch

=== WITHOUT TrueCall ===
tool reported: success -> agent says: done ✅
reality: record persisted = false ❌ SILENT FAILURE

=== WITH TrueCall ===
TrueCall caught it: "save_record reported success but post-condition
failed: expected record persisted in store, got predicate returned false"
agent self-corrects -> record persisted = true ✅ actually done

Without the verifier: green check, broken task, agent gone. With it: the lie gets caught mid-run, the agent self-corrects, the record actually lands. No model-confidence guessing — a cheap deterministic check turned an invisible failure into an actionable one.

Lesson 5: “Cross-harness” has to be load-bearing, not a slogan

I wired the first adapter into Claude Code’s PostToolUse hook: tool runs, hook fires with the arguments and result, the verifier runs the contract, and on failure hands back a block decision with the correction.

Then the real test: could the same contract catch the same failure in a different agent? I was skeptical of my own research (the notes came back suspiciously similar to Claude Code’s), so I read the actual Rust source in OpenAI’s Codex repo. The hooks are genuinely there.

So I refactored the verifier into one harness-neutral core and made each harness a thin wrapper. The punchline: the entire runtime difference between the two adapters is one line —

result: input.tool_output ?? input.tool_response

Claude Code calls it tool_output. Codex calls it tool_response. That's the whole difference. Same verifier, same contract, two agents.

The cross-harness claim isn't marketing; it's a single ??. One contract I wrote once now catches the identical silent failure in both harnesses — only the install config differs.

Lesson 6: Be honest about what you can’t build yet

I wanted a third harness. I researched it properly and found no public post-tool hook API exists for it today. So I did not ship an adapter that pretends to work. The stub says exactly that — blocked on a public API, here’s how it’ll plug in when one ships. An honest “not yet” beats a fake “yes.”

What broke (because something always does)

• A test that hung forever because I passed an input option to execFile, which silently ignores it — the child sat waiting on stdin that never came.

• A helper “fixing” that hang by quietly rewriting the stdin reader into a timer hack that could truncate large payloads. A review caught it; I restored the simple, correct version.

• An automated backup process that committed itself into my branch mid-refactor. Squashed it out before anything went public.

The hard part wasn’t any of these. It was the discipline to verify each claim — about the gap, the toolchain, my own research — instead of trusting the confident-sounding version. Which, fittingly, is the entire thesis of the project.

Lesson 7: I put it on a real benchmark — and the bottleneck wasn’t where I thought

Catching silent failures in a toy demo is one thing. So I wired TrueCall into τ²-bench — Sierra’s live agent benchmark — with a real agent (Gemini 2.5 Flash) doing real retail tasks, and injected silent tool failures to see if TrueCall would recover the tasks they broke.

The detection half worked perfectly: across every run, 100% of injected silent failures caught, 0 false positives. And then the task scores barely moved. That stung — detection was flawless, but completion wasn’t improving.

So I did the thing the project is about: I stopped trusting the summary and read the trajectories. What did the agent do right after a correction? It gave up. It retried the failed call only 8% of the time; 84% of the time it apologized to the user and moved on — reading my polite correction (”the effect wasn’t confirmed; verify before continuing”) as “tell the user it failed,” not “try again.”

The bottleneck was never detection. It was the wording of the correction.

I rewrote it as a blunt imperative — “re-run this exact call now; do NOT report success until it’s confirmed” — and the retry rate went from 8% to 64%. Same model, same tasks, one paragraph of phrasing. The deterministic catch is the floor; turning that into a fix is an ergonomics problem — you have to talk to the agent like you mean it. (Full data and the small-sample caveats are in the repo at bench/tau2/RESULTS.md — I’d rather show the messy truth than a clean number that didn’t replicate.)

Try it yourself. Open source, MIT, zero dependencies, on npm, same contract format in Claude Code and Codex today:

▶ try it in your browser: truecall-eosin.vercel.app

Install: npm install @avee1234/truecall

→ github.com/abhid1234/truecall

Read the spec, write a contract for one of your own tools, point a hook at it, and watch it catch a “success” that wasn’t. If you find a silent failure it can’t express, tell me — that’s the feedback that makes the format better.

I build small things to understand big problems. TrueCall started as an instinct from auditing agent trajectories offline — this is that instinct moved to live, runtime verification. If you're wrestling with agent reliability, I'd genuinely like to hear what's breaking for you.