Home › AI Agents

System Prompt Regression Test Set (Evals for Prompts & Skills)

Edit your system prompt, CLAUDE.md, or agent skill without fear: build a small eval set, rerun it after every change, and catch regressions before users do.

Copy template

You are a prompt QA engineer. Help me build and run a regression test set for my system prompt.

## My system prompt (or CLAUDE.md / SKILL.md)
[PASTE IT HERE]

## What it's for
[e.g. a support agent for a SaaS app / a coding agent for my repo / a writing assistant]

## Step 1: Write the test set
Create 15 test cases as a table with columns: id, category, user input, expected behavior, pass criteria.
Cover these categories:
- Happy path (5): normal requests it must handle well.
- Edge cases (3): vague, very long, or multi-part requests.
- Guardrails (3): requests it must refuse, escalate, or confirm first (e.g. refunds, deleting data, sharing secrets).
- Format (2): cases where output format matters (JSON, length limits, tone).
- Regressions (2): [PASTE PAST FAILURES HERE, or invent likely ones].
Pass criteria must be checkable yes/no, e.g. "asks for confirmation before issuing a refund", not "responds well".

## Step 2: Grade a run
When I paste the model's answers, grade each case PASS or FAIL against its pass criteria, quoting the exact line that decided it. Then give: pass rate, every failure with a one-line reason, and whether any guardrail case failed (a guardrail failure blocks the change no matter the pass rate).

## Step 3: Compare versions
When I paste results for the OLD and NEW prompt, show a diff table: cases that went PASS to FAIL (regressions), FAIL to PASS (fixes), and unchanged. Recommend ship / don't ship in one sentence.

## Step 4: Suggest the smallest fix
For each regression, propose the smallest edit to the NEW prompt that fixes it without breaking passing cases. Show it as a before/after snippet.

Rules: don't rewrite my whole prompt, don't add test cases I didn't ask for after Step 1, and flag any instruction in my prompt that contradicts another.

How to use it

1. Click Copy template.
2. Paste into ChatGPT, Claude, Grok, Cursor, or your agent framework.
3. Replace the [BRACKETED] parts with your details.

FAQ

Why do prompts need regression tests?

A small edit to a system prompt can fix one case and quietly break three others. Rerunning a fixed test set after every change shows exactly what moved.

How many test cases do I need?

Start with 10 to 20 that cover normal use, edge cases, and guardrails. Add every real failure you see in production as a new case.

Can I automate this?

Yes. Save the test table as JSON or CSV and run it with promptfoo, OpenAI Evals, or a short script, using this prompt as the grader.

Related templates

Browse all AI Agents templates

New on PromptLoot