You cannot optimize what you cannot score. Cost levers in Stop paying full price for repeat LLM calls cut the bill; an eval pipeline tells you whether the cheaper path still ships correct work. Cap the pipeline at roughly five metrics - enough to catch regressions, few enough that humans still read the failures.
Definitions
LLM-as-a-judge - using a model with a fixed rubric to score another model’s output (or trajectory), instead of exact-match string metrics.
G-Eval - the judge writes evaluation steps, then scores (often 1-5). In Confident AI’s survey, it ranked highest on rank correlation with human ratings among common NLG metric families.
DAG - a deterministic LLM decision tree: objective criteria with clear success conditions, walked as yes/no branches rather than a free-form score.
QAG - question-answer generation for fact-checking: the judge asks closed yes/no questions against the output (and sources, when present).
SelfCheckGPT - sample multiple model outputs on the same prompt and treat inconsistency as a hallucination signal.
Self-evaluation - asking the same model to verify its own answer as a chained step; Ren et al. reformulate open-ended generation as token-level prediction and show better calibration than sequence likelihood (arXiv 2312.09300).
Scorer families
| Family | Examples | Use when | Failure mode |
|---|---|---|---|
| Statistical | BLEU, ROUGE, METEOR | Closed reference text exists | Semantically blind |
| Model-based (non-LLM) | BERTScore, NLI, BLEURT | Soft lexical/semantic overlap | Capped by training data |
| LLM-as-a-judge | G-Eval, DAG, QAG, SelfCheckGPT | Subjective, multi-turn, or rubric-heavy work | Probabilistic; needs rubric discipline |
Selection rule of thumb (Confident AI’s framing, not a law of physics):
- Clear success condition → DAG
- Subjective quality → G-Eval
- Fact-checking against sources → QAG
- Reference-less hallucination sniff test → SelfCheckGPT (sample multiple outputs, check consistency)
The five-metric rule
1–2 custom metrics for the use case, plus 2–3 generic architecture metrics:
| Slot | Agent default | What it catches |
|---|---|---|
| Custom #1 | Task completion (pass/fail against a verifiable condition) | “Looks done” that isn’t |
| Custom #2 | Plan adherence or rubric score (G-Eval/DAG) | Drift from the agreed steps |
| Generic | Tool correctness | Wrong tool, wrong args, invented APIs |
| Generic | Faithfulness / citation (if RAG) | Answer ignores retrieved context |
| Generic | Cost proxy (tokens or $ per successful task) | Quality held while spend exploded |
Production cousin of custom #1 on this site: the Claude Code /goal evaluator only reads the transcript and a completion condition - see verifiable completion condition and goal evaluator is transcript-only. That is LLM-as-a-judge with a hard gate, not a vibes check.
Copy-paste: minimal judge prompt
You are an eval judge. Score ONLY against the rubric. No coaching.
Rubric:
1. Task completion (0 or 1): did the transcript meet the completion condition?
2. Tool correctness (0 or 1): every tool call used a real tool name and plausible args?
3. Plan adherence (1-5): how closely did the trajectory match the plan steps?
Completion condition:
{{COMPLETION_CONDITION}}
Plan steps:
{{PLAN_STEPS}}
Transcript:
{{TRANSCRIPT}}
Reply as JSON:
{"task_completion":0,"tool_correctness":0,"plan_adherence":1,"notes":"..."}Run the judge on a held-out set, not the same examples you tuned prompts against. Prefer a cheaper judge tier for volume (see Claude tier guide); reserve the frontier tier for contested failures.
When evals meet cost
Evals are how you keep trim-output and cache changes honest. Log $ per successful task alongside the quality metrics - a 40% token cut that drops task completion is not an optimization win. For agent loops where history itself is the cost, pair this pipeline with Prune the log, not the window.
Next move: pick one agent path, write one verifiable completion condition, wire the three-score JSON judge above, then add a cost proxy and a faithfulness/citation slot to reach five total - and fail the build when task completion regresses. Hub: Optimization. Garden note: LLM-as-a-judge.