# LLM-as-a-judge

> LLM-as-a-judge scores model output with a fixed rubric: DAG for pass/fail, G-Eval for subjective quality, QAG for fact checks. Cap pipelines at ~5 metrics.

URL: https://bartlomiejkrupa.dev/notes/llm-as-a-judge/
Category: concept | Updated: 2026-07-20
Summary: LLM-as-a-judge scores model output with a fixed rubric: DAG for pass/fail, G-Eval for subjective quality, QAG for fact checks. Cap pipelines at ~5 metrics.

**LLM-as-a-judge** - a model grades another model's output (or full trajectory) against a written rubric, instead of exact-match metrics like BLEU/ROUGE that ignore semantics.

## Which judge shape

| Need | Prefer |
| --- | --- |
| Clear success condition | DAG (deterministic yes/no tree) |
| Subjective quality | G-Eval (steps then score) |
| Fact-check against sources | QAG (closed yes/no questions) |

Self-evaluation (same model scoring its own answer) is calibration, not introspection - Ren et al. showed it beats sequence-likelihood baselines on TruthfulQA and TL;DR.

## On this site

The Claude Code [`/goal`](/articles/claude-code-goal-command) path is production LLM-as-a-judge with a hard gate: the evaluator only reads the transcript and a [verifiable completion condition](/notes/verifiable-completion-condition) ([transcript-only rule](/notes/goal-evaluator-transcript-only)). Full pipeline sizing: [Five metrics for an agent eval pipeline](/articles/five-metrics-for-an-agent-eval-pipeline).

Sources:
- Introduction to LLM metrics: https://deepeval.com/docs/metrics-introduction
- Self-Evaluation Improves Selective Generation in Large Language Models: https://arxiv.org/abs/2312.09300
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment: https://aclanthology.org/2023.emnlp-main.153/