Skip to main content

Quick start

pip install thaghr

That's it, no config file, no account, no external service to sign up for. thaghr is a plain Python CLI.

Your first campaign​

thaghr needs a target: a directory containing an agent.py with a run(http_client) function. thaghr ships one for you to try:

git clone https://github.com/thaghr/thaghr
cd thaghr
thaghr run examples/01-hello-agent --fault-rate 0.2 --trials 20

This runs 20 trials against the example agent, with a 20% chance per call that thaghr injects an HTTP 429 instead of letting the call through. You'll get a report card:

┌────────────────────────────────────────────┐
│ thaghr report card                         │
│ 01-hello-agent                             │
├────────────────────────────────────────────┤
│ trials      20                             │
│ passed      16/20                          │
│ pass^1: 80%                                │
│ errors      4/20 (RateLimitError)          │
└────────────────────────────────────────────┘

Gate CI on it​

Add --fail-under and the same command exits non-zero if reliability drops below your threshold:

thaghr run examples/01-hello-agent --fault-rate 0.2 --trials 20 --fail-under 0.7

See CLI reference for every flag, or jump straight to GitHub Action to wire this into CI without writing the shell yourself.

Point it at your own agent​

Any directory with an agent.py exposing run(http_client) works. http_client is an httpx.Client thaghr has already wired with its fault-injection transport, pass it straight through to whatever SDK your agent uses:

# agent.py
import openai

def run(http_client=None):
    client = openai.OpenAI(http_client=http_client)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Say hi in five words."}],
    )
    return {
        "content": response.choices[0].message.content,
        "prompt_tokens": response.usage.prompt_tokens,
        "completion_tokens": response.usage.completion_tokens,
    }

Next: Concepts explains what pass^k actually measures and why it's different from the metric you've probably seen before.