FIELD NOTES / 5 MIN READ

Would Your Bot's Routine Pass the Same Task Eight Times in a Row?

One successful run of your bot's routine tells you almost nothing about whether it will keep working. Sierra Research's pass^k metric, built for tau-bench, measures success across repeated trials instead of a single pass — and it exposes how fast reliability collapses even for capable models.

AI-assisted editorial: researched and drafted with AI using the primary sources linked below. Check the cited evidence and any stated test scope. Illustrative examples are not measured product results. Editorial policy.

Key takeaways

  • pass^k (the chance every one of k repeated trials succeeds) falls off fast: an 80%-per-run agent only hits about 41% at four trials, because the math is p^k, not an average.
  • A single green run of your bot's routine is a pass^1 data point. It tells you almost nothing about whether the same routine will succeed next week.
  • Before trusting a recurring bot routine, replay its representative task 5-8 times with varied inputs and track how many times it succeeded on every single attempt.

The decision: stop trusting a single successful run

If your bot's routine ran correctly once, that is a pass^1 result. It says nothing about whether the same routine will succeed the next ten times it runs on slightly different inputs. Sierra Research built a benchmark called tau-bench specifically because existing agent evaluations only checked whether a model could solve a task once, not whether it could solve the same kind of task reliably across repeated attempts.

The practical decision for a small-team bot builder: before you let a routine run unattended on a schedule, replay its representative task several times — not once — and count how many of those attempts fully succeeded. If the answer is anything less than all of them, you have a reliability problem that a single successful test run would have hidden from you.

What pass^k actually measures, and why it collapses fast

Sierra's paper introducing tau-bench defines two different metrics for repeated trials. pass@k, the metric the coding-benchmark world already used, asks whether at least one of k attempts succeeded — useful when you can generate several candidate solutions and keep the best one. For agents that act on real systems one time each, that framing is the wrong one. Sierra's team proposed a different metric instead: pass^k (pass hat k), defined as the chance that all k i.i.d. task trials are successful, averaged across tasks.

The distinction matters because the two metrics move in opposite directions as k grows. pass@k rises toward 1 with more attempts, since more tries give more chances to get lucky once. pass^k falls toward 0, because every single attempt has to succeed. Sierra's own experiments on tau-bench showed how severe this is in practice: the chance of reliably and consistently solving the same task multiple times significantly drops as the number of trials k increases — even the best-performing agent they tested, which has over 60% average task success on a single run, drops to under 25% at pass^8. That is not a weak model failing; that is a model that looks solid on one run and falls apart under repetition.

This isn't a one-off academic footnote. Anthropic's published model documentation discusses the pass^k reliability metric introduced by tau-bench to measure consistency across multiple trials, showing how tau-bench's methodology has influenced internal evaluation practices at other labs. If a frontier lab tracks consistency across repeated trials rather than trusting a single success, a small-team bot builder deciding whether a recurring routine is safe to automate should do the same, at a much smaller scale.

Sources: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 𝜏-bench.

Illustrative example: testing a weekly report routine

Illustrative example, not a measured result. Say your bot runs a weekly routine: pull last week's support tickets, summarize them, and post the summary to a team channel. You test it once on Monday, it works, and you schedule it to run automatically every week from then on.

Now instead run the same routine five times, each time with a different week's worth of tickets — some weeks with ten tickets, some with eighty, one week where a ticket has no resolution field filled in. Suppose four of the five runs post a correct summary and one run silently drops the malformed ticket instead of flagging it. That is pass^1 = 4/5 = 80% for that single trial count, but it is not the number that matters. What matters is what happens if you need the routine to get every one of several consecutive weeks right without a human catching the dropped ticket. Using the same p^k logic Sierra's team applied, an 80%-per-run routine only has roughly a 41% chance of getting four consecutive weeks fully correct, and under 33% at five weeks. A routine that looks "basically fine" on a single test can still be wrong roughly one week in five over a quarter of running unattended.

The fix in this illustrative case isn't a better model. It's adding a validation step that checks for the missing resolution field before the summary is posted, then re-testing the same five-run sequence to see whether the fix actually raised the all-trials success rate, instead of just re-running the one scenario that happened to fail.

How to apply this without building a tau-bench-style harness

You don't need Sierra's full benchmark environment to borrow the idea. The practical recipe for a personal or small-team bot: pick the one or two tasks your routine does most often, define what a fully correct outcome looks like (not just "it ran without erroring", but the specific end state you expect — the right file updated, the right message sent, the right record changed), then run that task 5 to 8 times with realistic variation in the input. Count how many of those runs hit every success condition, not just most of them.

For tasks like code generation with good verification techniques, the community has defined pass@k as the chance that at least one out of k trials is successful; for real-world agent tasks requiring reliability and consistency like customer service, pass^k is defined as the chance that all k i.i.d. task trials are successful, averaged across tasks. That second framing — all trials, not just one — is the one to apply to any routine that runs unattended and where a single bad output reaches a real inbox, calendar, or customer. If your bot's job is closer to a one-shot research query that a person reviews before acting, pass@k-style thinking (did it find a usable answer in a few tries) is closer to what you actually care about. The type of task decides which metric is the honest one.

Sources: τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.

What you miss if you only test once

Skipping repeated testing doesn't just risk an occasional miss — it hides exactly the kind of failure that compounds over time. A routine that silently fails one run in five will, left unattended for a few months of weekly runs, produce several wrong outputs that nobody caught because the first test run looked clean. The cost isn't abstract: it's the week the ticket summary quietly dropped a customer complaint, or the week the calendar routine double-booked a slot because an edge case in the input wasn't present in your one test.

The source code behind tau-bench is public, and its repository shows how to compute pass^k directly by running a task multiple times and comparing database outcomes against an annotated goal state rather than grading the conversation. You don't need the exact tooling to copy the discipline: track outcomes across repeated runs, not just the first one, before you trust a routine enough to let it run without you watching.

Sources: GitHub - sierra-research/tau-bench: Code and Data for Tau-Bench.

Primary sources

Sources checked 2026-11-07. Standards and product documentation can change; follow the linked version when implementing.

  1. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — arXiv (Sierra Research)
  2. 𝜏-bench — Sierra
  3. GitHub - sierra-research/tau-bench: Code and Data for Tau-Bench — Sierra Research (GitHub)

BotBento is in development. Suggest a correction.