# Write simple evals to test your LLM app before you ship changes

> Build a small eval set, score outputs with code checks and an LLM grader, and rerun it after every prompt or model change to catch regressions.

Level: Advanced
Tool: General
Category: Building
Time: About 40 minutes
Checked: 2026-10-15
Tools: [Claude API docs](https://docs.anthropic.com/en/docs/get-started), [OpenAI API docs](https://developers.openai.com/api/docs/quickstart)
HTML page: https://easyaihowtos.com/guides/write-evals-for-an-llm-app

Changing a prompt or model can quietly break answers that used to work. Evals catch that. This guide builds a small, practical eval harness in Python with code checks and an LLM grader, and explains each term along the way.

## What you need

- An LLM feature you already call from code (any provider)
- Python 3.9 or newer
- 20 to 50 real example inputs, ideally from actual users or logs

## Steps

### 1. Know what an eval is

An **eval** is an automated test for AI output. Because models give different wording each time, you test properties ("mentions the refund window," "valid JSON," "under 100 words") rather than exact text. Running the same evals after every change tells you whether the change helped or broke something (a **regression**).

### 2. Build a small test set

Save cases as JSON Lines (`evals.jsonl`), one per line. Include normal cases, edge cases, and cases the app should refuse.
```json
{"input": "How long do I have to return shoes?", "must_include": ["30 days"], "max_words": 80}
{"input": "Can you give me a discount code?", "must_include": ["can't"], "max_words": 60}
```

### 3. Write code-based checks first

Code checks are fast, free, and never change their mind. Use them for anything you can define precisely.
```python
import json

def code_checks(case, output):
    failures = []
    for phrase in case.get("must_include", []):
        if phrase.lower() not in output.lower():
            failures.append(f"missing '{phrase}'")
    if len(output.split()) > case.get("max_words", 10_000):
        failures.append("too long")
    return failures
```

### 4. Add an LLM grader for fuzzy qualities

For tone or helpfulness, ask a model to grade against a short rubric (a list of criteria) and answer PASS or FAIL. Here the grader uses Claude; any provider works.
```python
import anthropic
grader = anthropic.Anthropic()

def llm_grade(question, output):
    rubric = ("Grade the reply. PASS only if it is polite, answers the question, "
              "and makes no promises the company can't keep. Reply with PASS or FAIL "
              "and one sentence why.")
    r = grader.messages.create(
        model="claude-haiku-5-5", max_tokens=100,
        messages=[{"role": "user", "content": f"{rubric}\n\nQuestion: {question}\n\nReply: {output}"}],
    )
    text = r.content[0].text
    return text.strip().upper().startswith("PASS"), text
```

### 5. Run everything and print a score

`my_app(input)` is your existing function that calls the model.
```python
cases = [json.loads(line) for line in open("evals.jsonl")]
passed = 0
for case in cases:
    output = my_app(case["input"])
    failures = code_checks(case, output)
    ok, why = llm_grade(case["input"], output)
    if not ok:
        failures.append(f"grader: {why}")
    if failures:
        print("FAIL", case["input"], failures)
    else:
        passed += 1
print(f"{passed}/{len(cases)} passed")
```

### 6. Use it on every change

Record the score before you change a prompt, model, or retrieval setting, then run it again after. Only ship if the score holds or improves. Add a new case every time a real user finds a bug.

## Tips

- Spot-check the grader: read 10 of its verdicts yourself. If you disagree often, rewrite the rubric.
- Run each case 2 or 3 times when outputs vary a lot, and track the pass rate.
- Keep evals in version control next to your prompts so changes are reviewed together.

## FAQ

### How many test cases do I need?

Start with 20 to 50 that reflect real use. A small set you run every time beats a big one you never run.

### Can the same model grade itself?

It can, but a different or stronger model, plus code checks, gives a more independent signal.

### Are there eval tools I can use instead?

Yes, model providers and open-source projects offer eval frameworks and dashboards. The idea is the same: fixed cases, clear checks, and a score you track.

---

This guide is from Easy AI How-To (https://easyaihowtos.com). It is independent of the tools named above. Screens and free-plan limits change, so check the date before you rely on a button name.