HomeCasa Evaluate LLM answers with a rubric and an LLM judge
Design an evaluation for AI answers with an anchored rubric, an LLM-as-judge prompt with JSON output, bias mitigation and calibration against human labels. Designed for ChatGPT, Claude and Gemini. It includes 4 customizable variables and you can copy it for free in one click.
Advanced · Text · Pending editorial review
Prompt
Act as an LLM evaluation engineer experienced with "LLM as a judge" and human annotation.
Context:
- What the system I want to evaluate does: {{system}}
- Quality criteria I care about: {{criteria}}
- Scoring scale: {{scale}}
- Real question and answer examples from the system:
```
{{examples}}
```
Deliverables:
1. Rubric: for each criterion, an operational definition and a description of every level on the scale with a short example (anchored levels). Include knockout criteria, such as fabricated information or an unsafe answer.
2. The full judge prompt: it receives the question, the answer and, if available, a reference answer or context; it reasons briefly per criterion before scoring and returns JSON with a score per criterion, a one-sentence justification and pass or fail.
3. The JSON schema of the judge's output.
4. Known biases and how to mitigate them: preference for longer answers, position bias in A/B comparisons (swap the order), self-preference for its own style, and drift between judge versions.
5. Calibration: how to hand-label 30-50 cases, measure judge-human agreement and refine the rubric until agreement is acceptable.
6. Apply the rubric to the examples I gave you as a demonstration and flag the borderline cases.
7. How to fit evaluation into the workflow: a fixed regression set, a threshold that blocks a change and human review of failures.
Constraints:
- Do not present judge scores as absolute truth: they are an estimate that must be validated with people.
- Do not invent reference answers: if they are needed, mark them [FILL IN].
- If the criteria are vague ("it should be good"), ask me first what that means in my case.
Format: numbered sections, the rubric as a table, the prompt and schema in code blocks.How to use it
Paste 5-10 real examples, including some bad ones. Label a sample yourself with the rubric and compare it with the judge before trusting it to evaluate prompt or model changes.
Usage example
System: assistant that answers insurance customers’ questions · Criteria: accuracy, use of context, tone, escalation to a human · Scale: 1-5.
Sample output
1. Rubric
| Criterion | Pass | Fail |
|---|---|---|
| Accuracy | Every claim appears in the lease | Some detail (amount, deadline) does not match or is not in the lease |
| Penalty clauses | Mentions every penalty clause in the lease | Omits at least one |
| Clarity | A tenant with no legal background understands it on first read | Unexplained jargon or clauses mixed together |
| No legal advice | Informs and refers to a professional when relevant | Tells the tenant what to do legally ("you can stop paying rent") |
Knockout: inventing a clause or an amount → overall fail.
2. Judge prompt (excerpt)
You receive <lease>, <question> and <answer>.
For each criterion, reason in 1-2 sentences quoting the lease, then
decide "pass" or "fail". Do not reward length.
Return only this JSON:
{"accuracy": {"result": "...", "reason": "..."},
"penalties": {"result": "...", "reason": "..."},
"clarity": {"result": "...", "reason": "..."},
"no_legal_advice": {"result": "...", "reason": "..."},
"overall": "pass | fail"}5. Calibration
Hand-label 40 cases, compare with the judge criterion by criterion, and refine the rubric where they disagree most. Judge scores are an estimate, not ground truth: every fail is reviewed by a person…
Key facts
- Category
- Programming & development · AI agents
- Output type
- Text
- Level
- Advanced
- Designed for
- ChatGPT, Claude and Gemini
- Verified in
- Pending editorial verification
- Variables
- System to evaluate, Quality criteria, Scale and Real examples
- Language
- English
- Price
- Free, no sign-up
- Updated
- Nov 7, 2026
Frequently asked questions
Can one LLM reliably evaluate another?
It can be useful and scalable if the rubric is concrete and you calibrate it against human labels. Without calibration, its scores can reflect biases such as favoring longer answers.
Is a 1-5 scale or pass/fail better?
Pass/fail per criterion usually gets higher agreement and is easier to audit. A 1-5 scale adds nuance but needs levels that are very well anchored with examples.
What is this prompt for?
Design an evaluation for AI answers with an anchored rubric, an LLM-as-judge prompt with JSON output, bias mitigation and calibration against human labels.
Which AI tools does it work with?
It is designed for ChatGPT, Claude and Gemini. We have not verified it in these tools yet, so results may vary between them.
How do I customize the prompt?
Click “Customize” and fill in the variables (System to evaluate, Quality criteria, Scale and Real examples). The text updates instantly and you can then copy it in one click. Values are stored only in your browser.
Do I need experience to use it?
It is advanced: it is meant for people with experience in the topic who can evaluate and adjust the result.
Is this prompt free?
Yes. Every HomeCasa Prompts prompt is free, with no sign-up or subscription. You can also use them in professional projects.
Can I trust the AI output?
Treat it as a high-quality draft: always check data, figures and important claims before publishing or making decisions.