Evaluating AI Apps

An AI app that "seemed fine when I tried it" is not tested. Evaluation means checking it on many questions, the same way every time, and measuring the results.

Why AI Apps Need Special Testing

Normal code gives the same output for the same input, so one test per case is enough. AI apps are different:

• The same question can get a different answer each time
• A small change to the prompt can fix one case and break another
• A new model version can behave differently
• Answers can be fluent but wrong

So we test with a set of questions and look at the score, not at one lucky answer.

Step 1: Build a Test Set

A test set (often called evals) is a list of real questions with what a good answer must contain. Start small, with 20 to 50 cases, and add a new case every time a user finds a problem.

Include these kinds of cases:

Step 2: Score with Simple Checks

The cheapest check is: does the answer include the words it must include? Here a fake app stands in for your real one, so the example runs offline:

test_cases = [
    {"question": "When does the evening batch start?", "must_include": ["6 pm"]},
    {"question": "Where is the institute?", "must_include": ["Orangi"]},
    {"question": "Is there a course on Zoom?", "must_include": ["online"]},
    {"question": "What is the fee for flying lessons?", "must_include": ["do not know"]},
]

def fake_app(question):
    # Stand-in for your real AI app, so this example runs offline
    answers = {
        "When does the evening batch start?": "The evening batch starts at 6 pm.",
        "Where is the institute?": "It is in Orangi Town, Karachi.",
        "Is there a course on Zoom?": "Yes, all courses are on Zoom.",
        "What is the fee for flying lessons?": "Flying lessons cost Rs 5,000.",
    }
    return answers[question]

def evaluate(app, cases):
    passed = 0
    for case in cases:
        answer = app(case["question"])
        ok = all(p.lower() in answer.lower() for p in case["must_include"])
        passed += ok
        print("PASS" if ok else "FAIL", "|", case["question"], "->", answer)
    print(f"Score: {passed}/{len(cases)}")

evaluate(fake_app, test_cases)

PASS | When does the evening batch start? -> The evening batch starts at 6 pm.
PASS | Where is the institute? -> It is in Orangi Town, Karachi.
FAIL | Is there a course on Zoom? -> Yes, all courses are on Zoom.
FAIL | What is the fee for flying lessons? -> Flying lessons cost Rs 5,000.
Score: 2/4

Look closely at the two failures:

• The flying lessons answer is a real problem: the app invented a price. This is exactly what evals are for.
• The Zoom answer is actually correct, but it does not use the word "online". The check was too strict.

Simple word checks are fast and free, but they cannot judge meaning. For that, we use a second model.

Step 3: Use an AI Judge

An LLM-as-judge is a model that grades answers against the facts. You give it the question, the true facts and the answer, and ask for a JSON verdict:

import json

JUDGE_PROMPT = """You are grading an answer from a customer support bot.
Question: {question}
Facts the answer must agree with: {facts}
Answer: {answer}
Reply with JSON only: {{"correct": true or false, "reason": "one short sentence"}}"""

prompt = JUDGE_PROMPT.format(
    question="Is there a course on Zoom?",
    facts="Every course is also offered online on Zoom with recorded classes.",
    answer="Yes, all courses are on Zoom.",
)
print(prompt)

# What a judge model might send back, and how you read it:
judge_reply = '{"correct": true, "reason": "The answer matches the facts."}'
print("Judge says:", json.loads(judge_reply)["correct"])

You are grading an answer from a customer support bot.
Question: Is there a course on Zoom?
Facts the answer must agree with: Every course is also offered online on Zoom with recorded classes.
Answer: Yes, all courses are on Zoom.
Reply with JSON only: {"correct": true or false, "reason": "one short sentence"}
Judge says: True

In a real eval, you send prompt to the chat completions API with temperature=0 and JSON output, as in Structured Output and JSON. The {{ and }} in the template become single braces after format.

What to Measure

Make It a Habit

• Run the evals before every change to the prompt, model or documents, and compare the score
• Save results with the date, so you can see if things get better or worse
• Read real conversations (with permission and without private data) to find new test cases
• Never ship a change that lowers the score without knowing why

Exercise