Skip to content
Easy AI How-To

AdvancedGeneralBuilding

Write simple evals to test your LLM app before you ship changes

Changing a prompt or model can quietly break answers that used to work. Evals catch that. This guide builds a small, practical eval harness in Python with code checks and an LLM grader, and explains each term along the way.

Time
About 40 minutes
Checked
Tools
Claude API docs, OpenAI API docs

If a button name on the screen does not match this guide, follow the action. These products rename buttons often. This is an advanced guide: it uses code, and each technical term is explained the first time it appears. Model names and prices change, so check the linked docs before you ship.

Steps

  1. Know what an eval is

    An eval is an automated test for AI output. Because models give different wording each time, you test properties ("mentions the refund window," "valid JSON," "under 100 words") rather than exact text. Running the same evals after every change tells you whether the change helped or broke something (a regression).

  2. Build a small test set

    Save cases as JSON Lines (evals.jsonl), one per line. Include normal cases, edge cases, and cases the app should refuse.

    {"input": "How long do I have to return shoes?", "must_include": ["30 days"], "max_words": 80}
{"input": "Can you give me a discount code?", "must_include": ["can't"], "max_words": 60}

  3. Write code-based checks first

    Code checks are fast, free, and never change their mind. Use them for anything you can define precisely.

    import json

def code_checks(case, output):
    failures = []
    for phrase in case.get("must_include", []):
        if phrase.lower() not in output.lower():
            failures.append(f"missing '{phrase}'")
    if len(output.split()) > case.get("max_words", 10_000):
        failures.append("too long")
    return failures

  4. Add an LLM grader for fuzzy qualities

    For tone or helpfulness, ask a model to grade against a short rubric (a list of criteria) and answer PASS or FAIL. Here the grader uses Claude; any provider works.

    import anthropic
grader = anthropic.Anthropic()

def llm_grade(question, output):
    rubric = ("Grade the reply. PASS only if it is polite, answers the question, "
              "and makes no promises the company can't keep. Reply with PASS or FAIL "
              "and one sentence why.")
    r = grader.messages.create(
        model="claude-haiku-5-5", max_tokens=100,
        messages=[{"role": "user", "content": f"{rubric}\n\nQuestion: {question}\n\nReply: {output}"}],
    )
    text = r.content[0].text
    return text.strip().upper().startswith("PASS"), text

  5. Run everything and print a score

    my_app(input) is your existing function that calls the model.

    cases = [json.loads(line) for line in open("evals.jsonl")]
passed = 0
for case in cases:
    output = my_app(case["input"])
    failures = code_checks(case, output)
    ok, why = llm_grade(case["input"], output)
    if not ok:
        failures.append(f"grader: {why}")
    if failures:
        print("FAIL", case["input"], failures)
    else:
        passed += 1
print(f"{passed}/{len(cases)} passed")

  6. Use it on every change

    Record the score before you change a prompt, model, or retrieval setting, then run it again after. Only ship if the score holds or improves. Add a new case every time a real user finds a bug.

Tips

  • Spot-check the grader: read 10 of its verdicts yourself. If you disagree often, rewrite the rubric.

  • Run each case 2 or 3 times when outputs vary a lot, and track the pass rate.

  • Keep evals in version control next to your prompts so changes are reviewed together.

Common questions

How many test cases do I need?

Start with 20 to 50 that reflect real use. A small set you run every time beats a big one you never run.

Can the same model grade itself?

It can, but a different or stronger model, plus code checks, gives a more independent signal.

Are there eval tools I can use instead?

Yes, model providers and open-source projects offer eval frameworks and dashboards. The idea is the same: fixed cases, clear checks, and a score you track.

More guides

Occasional notes

Hear when a new guide is up

A short note when a guide is added. No account, and no emails until this form is connected to a real list.

Placeholder. This form is not connected to an email service. Nothing you type is saved or sent.