Human evaluation remains the gold standard for assessing open-ended LLM outputs — but it's slow, expensive, and hard to scale. LLM-as-Judge leverages a strong language model (the "judge") to score or rank outputs from one or more candidate models, giving you human-like signal at API-call speed.
Core idea: Instead of hiring annotators, you craft a judge prompt that instructs a powerful model (e.g., GPT-4o) to evaluate responses on specific criteria — returning structured scores and explanations.
Pioneered by the MT-Bench paper (Zheng et al., 2023), this approach showed GPT-4 judgments agree with human preferences >80 % of the time — comparable to inter-annotator agreement. Since then, LLM-as-Judge has become a standard tool in evaluation pipelines at OpenAI, Anthropic, and across the industry.
Judge Prompt Design
The judge prompt is the most critical component. A well-crafted prompt includes a role definition, evaluation criteria (rubric), output format, and optionally a reference answer.
Anatomy of a Judge Prompt
# A structured judge prompt templateJUDGE_SYSTEM_PROMPT = """
You are an expert evaluator. Your task is to assess
the quality of an AI assistant's response.
## Evaluation Criteria
- Helpfulness: Does the response address the user's need?
- Accuracy: Are facts and claims correct?
- Clarity: Is the response well-structured and easy to follow?
- Completeness: Does it cover all aspects of the question?
## Scoring
Rate the response on a scale of 1-5 for each criterion.
Provide a brief justification BEFORE the score.
## Output Format (strict JSON)
{
"helpfulness": {"reasoning": "...", "score": N},
"accuracy": {"reasoning": "...", "score": N},
"clarity": {"reasoning": "...", "score": N},
"completeness": {"reasoning": "...", "score": N},
"overall_score": N
}
"""
Key rule: Always ask the judge to provide reasoning before the numeric score. This "chain-of-thought" approach dramatically improves score reliability and reduces random variance.
Basic OpenAI Judge Call
import openai, json
client = openai.OpenAI()
defjudge_response(question: str, response: str) -> dict:
"""Score a single response using GPT-4o as judge."""
completion = client.chat.completions.create(
model="gpt-4o",
temperature=0.0, # deterministic scoring
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": f"""## User Question
{question}
## Assistant Response
{response}
Evaluate the response according to the rubric."""}
]
)
return json.loads(completion.choices[0].message.content)
Setting temperature=0.0 and using response_format={"type": "json_object"} ensures deterministic, parseable output. For even stricter schema enforcement, use OpenAI's Structured Outputs with a Pydantic model.
Single-Point vs Pairwise Grading
There are two fundamental paradigms for LLM-as-Judge evaluation, each suited to different scenarios.
🎯 Single-Point Grading
The judge scores one response at a time against a rubric. Returns an absolute score (e.g., 1–5).
Best for: Regression testing, threshold-based quality gates, tracking a single model over time.
Cons: Score calibration can drift; harder to detect subtle quality differences.
⚖️ Pairwise Comparison
The judge sees two responses and picks the better one (A, B, or Tie). Used in MT-Bench and Chatbot Arena.
Best for: Model selection, A/B testing, comparing prompt variants.
Pros: More reliable for relative ranking; easier for the judge model.
Cons: O(n²) comparisons for n models; susceptible to position bias.
Pairwise Judge Implementation
PAIRWISE_PROMPT = """
Compare two responses to the same question.
Pick the better one: output "A", "B", or "Tie".
Give reasoning before your verdict.
Output JSON: {"reasoning": "...", "verdict": "A"|"B"|"Tie"}
"""defpairwise_judge(question, response_a, response_b):
"""Pairwise comparison with position-bias mitigation."""import random
# Swap order randomly to reduce position biasif random.random() > 0.5:
first, second = response_b, response_a
swapped = Trueelse:
first, second = response_a, response_b
swapped = False
user_msg = f"""Question: {question}
Response A: {first}
Response B: {second}"""
result = client.chat.completions.create(
model="gpt-4o", temperature=0.0,
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": PAIRWISE_PROMPT},
{"role": "user", "content": user_msg}
]
)
verdict = json.loads(result.choices[0].message.content)
# Un-swap the verdict if we reversed the orderif swapped and verdict["verdict"] in ("A", "B"):
verdict["verdict"] = "B"if verdict["verdict"] == "A"else"A"return verdict
Multi-Dimension Evaluation
Real-world evaluation rarely collapses to a single score. Strong judge setups evaluate across multiple orthogonal dimensions, then aggregate.
from pydantic import BaseModel
from openai import OpenAI
classDimensionScore(BaseModel):
reasoning: str
score: int # 1-5classEvalResult(BaseModel):
helpfulness: DimensionScore
accuracy: DimensionScore
safety: DimensionScore
coherence: DimensionScore
overall: int # weighted aggregate
client = OpenAI()
defmulti_dim_judge(question: str, response: str) -> EvalResult:
"""Evaluate across 4 dimensions with structured output."""
completion = client.beta.chat.completions.parse(
model="gpt-4o-2024-08-06",
temperature=0.0,
messages=[
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": f"""
Question: {question}
Response: {response}
Evaluate each dimension. For overall, use:
0.3*helpfulness + 0.3*accuracy + 0.2*safety + 0.2*coherence"""}
],
response_format=EvalResult
)
return completion.choices[0].message.parsed
MT-Bench approach: MT-Bench uses multi-turn evaluation with two-turn conversations. The judge sees both turns and scores them independently, capturing the model's ability to follow up and refine answers — a dimension single-turn evals miss entirely.
Known Biases & Mitigation
LLM judges inherit systematic biases. Understanding and mitigating these is critical for trustworthy automated evaluation.
Position Bias: The #1 Problem
In pairwise evaluation, LLMs tend to prefer whichever response appears first. The standard mitigation is to run each comparison twice with swapped order, then check for agreement:
defdebiased_pairwise(question, resp_a, resp_b):
"""Run pairwise eval in both orders, flag disagreements."""
v1 = pairwise_judge(question, resp_a, resp_b) # A first
v2 = pairwise_judge(question, resp_b, resp_a) # B first# Normalize v2 verdict (un-swap already handled in function)if v1["verdict"] == v2["verdict"]:
return {"verdict": v1["verdict"], "confidence": "high"}
else:
return {"verdict": "Tie", "confidence": "low",
"note": "Position-dependent — flag for human review"}
Self-preference is real: Research shows GPT-4 prefers GPT-4 outputs by ~10 percentage points over equally-good Claude outputs. When the candidate model and judge model share the same family, consider using a different judge model or an ensemble of multiple judge models.
Verbosity Bias Mitigation
Add explicit rubric instructions to counter verbosity bias:
ANTI_VERBOSITY_CLAUSE = """
IMPORTANT: A concise, correct answer is ALWAYS preferred over
a verbose answer that adds unnecessary detail. Penalize
responses that pad their length without adding value.
Do NOT reward length — reward information density.
"""
Correlation with Human Evaluation
The ultimate question: does the LLM judge agree with humans? Multiple studies establish strong but imperfect correlation.
📊 Agreement Rates
GPT-4 vs. human experts: ~80% agreement on pairwise preferences (MT-Bench)
Human vs. human: ~81% inter-annotator agreement (same benchmark)
GPT-3.5 as judge: ~66% agreement — significantly worse
Claude as judge: ~75-78% agreement (varies by task)
⚠️ Where Judges Fail
Math/logic: LLM judges often can't verify complex reasoning chains
Factual accuracy: Judges may not catch subtle hallucinations
Domain expertise: Medical, legal, and scientific claims need specialist review
Cultural nuance: Humor, tone, and style are poorly evaluated
Measuring Judge Quality
import numpy as np
from scipy.stats import spearmanr, kendalltau
defevaluate_judge_quality(human_scores, judge_scores):
"""Compute correlation metrics between human and judge scores."""
spearman_r, sp_p = spearmanr(human_scores, judge_scores)
kendall_t, kt_p = kendalltau(human_scores, judge_scores)
# Pairwise agreement rate
agree = 0
total = 0
for i inrange(len(human_scores)):
for j inrange(i + 1, len(human_scores)):
h_pref = np.sign(human_scores[i] - human_scores[j])
j_pref = np.sign(judge_scores[i] - judge_scores[j])
if h_pref == j_pref:
agree += 1
total += 1
return {
"spearman_r": round(spearman_r, 3),
"kendall_tau": round(kendall_t, 3),
"pairwise_agreement": round(agree / total, 3)
}
Rule of thumb: If your LLM judge achieves a Spearman ρ > 0.7 with human annotations on a calibration set of 100–200 examples, it's reliable enough for automated use. Below 0.6, invest in prompt tuning or switch judge models.
Production Implementation
Moving LLM-as-Judge from notebooks to production requires batching, cost management, logging, and fail-safes.
Full Evaluation Pipeline
import asyncio, json, time
from openai import AsyncOpenAI
from dataclasses import dataclass
from typing import List
client = AsyncOpenAI()
@dataclassclassEvalItem:
id: str
question: str
response: str
reference: str = ""async defjudge_single(item: EvalItem, semaphore) -> dict:
"""Score one item with concurrency control."""async with semaphore:
try:
t0 = time.time()
completion = await client.chat.completions.create(
model="gpt-4o",
temperature=0.0,
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": JUDGE_SYSTEM_PROMPT},
{"role": "user", "content": f"""
Question: {item.question}
Response: {item.response}
Reference: {item.reference}
Evaluate the response."""}
]
)
scores = json.loads(completion.choices[0].message.content)
scores["id"] = item.id
scores["latency_s"] = round(time.time() - t0, 2)
scores["tokens"] = completion.usage.total_tokens
return scores
except Exception as e:
return {"id": item.id, "error": str(e)}
async defrun_eval_pipeline(items: List[EvalItem],
max_concurrent: int = 10):
"""Run judge over a dataset with controlled concurrency."""
semaphore = asyncio.Semaphore(max_concurrent)
tasks = [judge_single(item, semaphore) for item in items]
results = await asyncio.gather(*tasks)
# Summary statistics
scores = [r["overall_score"] for r in results if"error"not in r]
errors = [r for r in results if"error"in r]
print(f"Evaluated {len(scores)} items | "f"Mean: {sum(scores)/len(scores):.2f} | Errors: {len(errors)}")
return results
Cost awareness: Judging 1,000 responses with GPT-4o at ~800 tokens per judge call costs roughly $4–8. For large-scale eval, consider using GPT-4o-mini for initial screening (10× cheaper) and reserving GPT-4o for borderline cases. Always log token usage per evaluation.
Logging & Observability
Every judge call should log: the item ID, judge model, prompt version hash, raw scores, token usage, latency, and timestamp. This enables:
Score drift detection — track mean scores over time after model or prompt changes
Cost attribution — billing per evaluation campaign
Prompt version A/B testing — comparing rubric variants
Failure rate monitoring — alerting on parse errors or judge refusals
Calibrating & Improving Judges
An uncalibrated judge may consistently rate too high or too low, or compress the score range. Calibration aligns judge scores with human expectations.
Calibration Workflow
Step-by-step calibration
Step 1 — Build a calibration set: Collect 100–200 examples with human scores covering the full quality spectrum (including intentionally bad responses). This is your ground truth.Step 2 — Run the judge: Score all calibration examples with your judge prompt. Compute Spearman ρ, Kendall τ, and pairwise agreement against human labels.Step 3 — Iterate on the prompt: Adjust rubric wording, add few-shot examples of scored responses, and modify the scale (e.g., 1–5 vs 1–10). Re-run until correlation exceeds your threshold.Step 4 — Monitor in production: Periodically sample judge outputs for human review. Track score distributions — sudden shifts indicate model updates or prompt drift.
Few-Shot Calibration
Including scored examples in the judge prompt significantly improves calibration. These "anchor" examples define what each score level means concretely:
FEW_SHOT_EXAMPLES = """### Example 1 (Score: 5 - Excellent)
Question: What causes seasons on Earth?
Response: Seasons are caused by Earth's ~23.5° axial tilt.
As Earth orbits the Sun, different hemispheres receive
more direct sunlight. This tilt — not distance from the
Sun — drives temperature variation.
Reasoning: Accurate, concise, addresses common misconception.
### Example 2 (Score: 2 - Poor)
Question: What causes seasons on Earth?
Response: Seasons happen because the Earth moves closer to
and farther from the Sun during its orbit.
Reasoning: Contains a common factual error. Distance from
the Sun is not the primary cause of seasons.
### Example 3 (Score: 3 - Acceptable)
Question: What causes seasons on Earth?
Response: The tilt of the Earth causes seasons.
Reasoning: Correct but too brief. Lacks explanation of the
mechanism (axial tilt relative to orbital plane).
"""
Key insight: The best judge prompts combine a clear rubric, chain-of-thought reasoning, structured output, and 3–5 few-shot examples with diverse score levels. This combination achieves the highest correlation with human evaluators — often within 2–3 percentage points of human inter-annotator agreement.