You cannot optimize what you cannot score. Cost levers in Stop paying full price for repeat LLM calls cut the bill; an eval pipeline tells you whether the cheaper path still ships correct work. Cap the pipeline at roughly five metrics - enough to catch regressions, few enough that humans still read the failures.
Definitions
LLM-as-a-judge - using a model with a fixed rubric to score another model’s output (or trajectory), instead of exact-match string metrics.
G-Eval - the judge generates its own evaluation steps via chain-of-thought, then fills in a score (often 1-5). Introduced by Liu et al. in G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (EMNLP 2023), which reported a Spearman correlation of 0.514 with human judgment on summarization - well ahead of the statistical scorers it was measured against.
DAG - a deterministic LLM decision tree: objective criteria with clear success conditions, walked as yes/no branches rather than a free-form score.
QAG - question-answer generation for fact-checking: the judge asks closed yes/no questions against the output (and sources, when present).
SelfCheckGPT - sample multiple model outputs on the same prompt and treat inconsistency as a hallucination signal.
Self-evaluation - asking the same model to verify its own answer as a chained step; Ren et al. reformulate open-ended generation as token-level prediction and show better calibration than sequence likelihood (arXiv 2312.09300).
Scorer families
| Family | Examples | Use when | Failure mode |
|---|---|---|---|
| Statistical | BLEU, ROUGE, METEOR | Closed reference text exists | Semantically blind |
| Model-based (non-LLM) | BERTScore, NLI, BLEURT | Soft lexical/semantic overlap | Capped by training data |
| LLM-as-a-judge | G-Eval, DAG, QAG, SelfCheckGPT | Subjective, multi-turn, or rubric-heavy work | Probabilistic; needs rubric discipline |
Selection rule of thumb (Confident AI’s framing, not a law of physics):
- Clear success condition → DAG
- Subjective quality → G-Eval
- Fact-checking against sources → QAG
- Reference-less hallucination sniff test → SelfCheckGPT (sample multiple outputs, check consistency)
The five-metric rule
1–2 custom metrics for the use case, plus 2–3 generic architecture metrics:
| Slot | Agent default | What it catches |
|---|---|---|
| Custom #1 | Task completion (pass/fail against a verifiable condition) | “Looks done” that isn’t |
| Custom #2 | Plan adherence or rubric score (G-Eval/DAG) | Drift from the agreed steps |
| Generic | Tool correctness | Wrong tool, wrong args, invented APIs |
| Generic | Faithfulness / citation (if RAG) | Answer ignores retrieved context |
| Generic | Cost proxy (tokens or $ per successful task) | Quality held while spend exploded |
Production cousin of custom #1 on this site: the Claude Code /goal evaluator only reads the transcript and a completion condition - see verifiable completion condition and goal evaluator is transcript-only. That is LLM-as-a-judge with a hard gate, not a vibes check.
Copy-paste: minimal judge prompt
You are an eval judge. Score ONLY against the rubric. No coaching.
Rubric:
1. Task completion (0 or 1): did the transcript meet the completion condition?
2. Tool correctness (0 or 1): every tool call used a real tool name and plausible args?
3. Plan adherence (1-5): how closely did the trajectory match the plan steps?
Completion condition:
{{COMPLETION_CONDITION}}
Plan steps:
{{PLAN_STEPS}}
Transcript:
{{TRANSCRIPT}}
Reply as JSON:
{"task_completion":0,"tool_correctness":0,"plan_adherence":1,"notes":"..."}Run the judge on a held-out set, not the same examples you tuned prompts against. Prefer a cheaper judge tier for volume (see Claude tier guide); reserve the frontier tier for contested failures.
The judge is a biased instrument
One finding from the G-Eval paper deserves more attention than it gets, because it lands directly on this use case: LLM-based evaluators show a bias toward LLM-generated text. You are pointing a model at another model’s output and asking “is this good?” - and the instrument tilts toward answering yes.
Three consequences for the pipeline:
- Gate on deltas, not absolutes. A plan-adherence score of 4.1 means nothing on its own. A drop from 4.1 to 3.2 against the same rubric and the same judge means something. Store a baseline and fail on regression.
- Pin the judge. Changing judge model or version silently re-baselines every score you have. Treat a judge upgrade as a migration: re-score the baseline set before trusting the new numbers.
- Keep a human sample. Read ~20 judged failures per release. Judge drift shows up there first, and it is the only check that does not share the bias.
Suggested starting thresholds: task completion is the only hard gate (fail the build on any regression against baseline), tool correctness fails below 0.95, and plan adherence stays advisory until you have enough runs to know its variance.
When evals meet cost
Evals are how you keep trim-output and cache changes honest. Log $ per successful task alongside the quality metrics - a 40% token cut that drops task completion is not an optimization win. For agent loops where history itself is the cost, pair this pipeline with Prune the log, not the window.
Next move: pick one agent path, write one verifiable completion condition, wire the three-score JSON judge above, then add a cost proxy and a faithfulness/citation slot to reach five total - and fail the build when task completion regresses. Hub: Optimization. Garden note: LLM-as-a-judge. In the field, the same eval discipline is the validation phase of last-mile FDE work - demo success is not customer-task success.