Skip to main content

Concepts

pass^k, not pass@k​

If you've evaluated an LLM agent before, you've probably seen pass@k: did the agent succeed at least once in k tries? It's the standard metric from code-generation benchmarks (Chen et al., 2021), and it rewards luck. An agent that succeeds once in five tries scores the same as one that succeeds five times in five tries, pass@k can't tell them apart.

pass^k asks a different question: did the agent succeed every time in k tries? This is the metric tau-bench (Yao et al., 2024) calls "agent consistency", and it's what thaghr reports by default.

pass^k = E_task[ C(c, k) / C(n, k) ]

where n is the number of trials run, c is how many succeeded, and C is "n choose k". It's the unbiased estimator for "probability that every trial in a random k-subset of your n trials succeeded."

For agents heading toward production, this is the number that matters. Nobody cares that your agent can complete a task. They care whether it does, reliably, every time a user depends on it.

GDS: partial credit when pass^k goes to zero​

pass^k is unforgiving at low sample sizes. An agent making real progress, getting 80% of the way through a multi-step task, still reads as a flat 0% if it never completes the whole task cleanly. That's accurate but not always useful: "does nothing" and "almost works" look identical.

When pass^k rounds to 0%, thaghr's report card falls back to GDS (Graded Degradation Score), a weighted average across subtasks rather than a single pass/fail per episode. If you don't decompose your task into subtasks, GDS collapses to a single weight-1.0 check, same shape as pass^k, no extra work required to opt in.

Fault injection, at the transport layer​

thaghr intercepts your agent's calls to its LLM provider at the HTTP transport layer, underneath whatever SDK you use (OpenAI's client, or anything else built on httpx). This means:

  • No framework-specific adapter. Works underneath LangChain, a bare SDK call, or anything else, since it never touches your agent's code, only the HTTP layer beneath it.
  • Deterministic under a seed. The same --seed reproduces the exact same fault sequence, so a campaign is a fair comparison across runs, not noise.
  • Faults never touch the real API. When a fault fires, thaghr returns an injected response directly, the real provider never sees that call. Fault-injection campaigns don't cost more in API spend than the calls that actually get through.

robustness and fault_tolerance​

Two derived metrics, both specific to fault injection, neither defined by tau-bench:

robustness = pass^k(faulted) / pass^k(baseline). How much of the agent's fault-free reliability survives under chaos, as a fraction. thaghr compare runs both conditions and reports this directly.

fault_tolerance = survival rate among the specific trials a fault actually hit. Of the calls where thaghr injected a fault, what fraction still succeeded? This isolates the agent's actual error-handling, separate from its baseline reliability.

Next: CLI reference for every command and flag.