LLM-as-a-Judge vs Human Labels: 6 Rules for Evaluating Ticket Categorization
For evaluating ticket categorization, use human labels as the ground truth and an LLM judge as a calibrated, cheaper proxy for scale. LLM-as-a-judge works well for checking individual labels once it agrees with your human reviewers about as often as they agree with each other. It works poorly for judging whether the taxonomy itself is useful, and it should not grade a classifier built on the same model. The six rules below cover when to use each, how to calibrate the judge, and which checks must stay human.
The question is rarely "judge or humans." It is which decisions each one is trusted to make. Teams get into trouble when a judge that was calibrated for one decision (is this label right?) is quietly used for another (is this taxonomy good?).
Where LLM judges help and where they mislead
An LLM judge is good at local questions with a clear rubric: given this ticket and this label definition, does the label fit? It is fast, cheap and consistent enough to score thousands of records per run, which humans cannot do on every pull request.
It is weak at whole-structure questions. Enterpret's taxonomy evaluation paper scored six feedback taxonomies with common generic LLM-judge checks. All six passed every one: name-description match was 0.97 to 1.00 and description completeness was 1.00. Yet the AI-generated taxonomies had 97.7 to 100% of leaf names restating a parent, and 34.7% to 76.4% of records leaking across top-level branches, against about 20% for production references. A product-coverage check even preferred the generated trees (0.74 to 0.91 vs 0.72 to 0.80).
6 of 6 Feedback taxonomies that passed every generic LLM-judge quality check, including AI-generated ones where up to 76% of records leaked across team boundaries. Source: Enterpret Research, "From Plausible Hierarchies to Useful Taxonomies" (arXiv, 2026)
The paper's conclusion was that surface plausibility is not enough; evaluation has to measure the whole tree as well as each node.
The 6 rules for LLM judges vs human labels
1. Humans define the truth; the judge approximates it
Build the reference set from blind human labels by people who know the product, with disagreements adjudicated and recorded. The judge is scored against that set, never the other way round. Our guide to creating a golden dataset for feedback classification evals covers the labeling process.
2. Calibrate against human agreement, not a fixed cutoff
Measure how often two of your human labelers agree, then measure how often the judge agrees with the adjudicated label. If the judge is close to human-human agreement, it is doing about as well as an extra labeler. OneUptime's guide to calibrating an LLM judge with Cohen's kappa makes the same point: there is no universal kappa threshold that makes a judge safe, so set acceptance from your use case and your human baseline, and report a confidence interval rather than one number.
3. Freeze the judge and keep calibration data separate
Pin the judge's model version, prompt, examples and output format. Record parse failures and refusals as errors instead of dropping them. Never tune the judge's prompt on the same records you use to measure its agreement, or the agreement number will flatter it. Recalibrate on fresh human labels whenever the judge or the traffic changes.
4. Do not let a model grade itself
If your classifier runs on one model, judge it with a different model family where possible, and always spot-check with humans. A judge that shares the classifier's blind spots will agree with its mistakes. The same applies to golden sets: if an LLM wrote the reference labels, the eval measures agreement with that LLM.
5. Keep structure and ownership checks human or metric-based
Questions like "is this label informative?", "can one team own this branch?" and "should this label be split?" need product knowledge and whole-tree metrics, not per-node judgments. Compute leaf restatement and branch leakage directly, and have a product owner review the taxonomy after changes.
6. Block on deterministic checks, warn on judge scores
In CI, gate merges on checks that cannot flake: exact-match per-label recall against the human golden set. Report judge scores as warnings until they have held steady across several runs. Langfuse's regression testing guide recommends this same pattern of promoting a judge metric to blocking only after its baseline is stable.
Who should decide what
| Decision | Human labels | Calibrated LLM judge | Computed metric |
|---|---|---|---|
| Golden set ground truth | Yes | No | No |
| Per-record label check at scale | Spot-check | Yes | Exact match vs golden |
| PR merge gate | No | Warn only | Yes |
| Label informativeness | Yes | No | Leaf restatement rate |
| Team ownership | Yes | No | Branch leakage |
| Production audit | Weekly sample | Daily triage | "Other" rate |
Where Enterpret fits
Enterpret's Adaptive Taxonomy is built with structural checks of the kind the taxonomy paper measured, so you start from labels designed to be informative and ownable rather than just plausible. The Customer Context Graph ties each theme to owners and teams, and Wisdom or the Enterpret MCP server lets you pull labeled samples for human review into the tools you already use. You should still keep a human-labeled reference set for anything you build on top. For the full eval setup, see the seven steps to build evals for an LLM that classifies customer feedback.
Example: a judge that agreed too much
Picture a team grading a ticket classifier with a judge on the same model family. Agreement looked excellent. A human audit of 200 records found the classifier and judge shared a blind spot: both labeled "cannot log in after password reset" as "account settings" rather than "authentication." Switching the judge to a different model family and adding ten adjudicated boundary examples brought the judge's agreement down to a realistic level, and closer to the humans. What worked: the human audit. What did not: three weeks of green CI runs built on a judge that was grading its own reflection.
What changes next
LLM judges will keep getting cheaper and better at local rubrics, so more evaluation will shift to them. The risk shifts too: a plausible-looking, judge-approved taxonomy can still be useless. Expect whole-structure metrics and periodic human review to become the standard complement to judge-based scoring.
FAQ
Can I use LLM-as-a-judge to evaluate ticket categorization?
Yes, for per-record label checks at scale, once it is calibrated against blind human labels and agrees with them about as often as your humans agree with each other. Keep the ground truth and taxonomy-quality decisions human.
What kappa should an LLM judge reach?
There is no universal cutoff. Compare the judge's agreement with your human labelers' agreement with each other, report a confidence interval, and set the bar by the cost of a wrong label in your use case.
Why not use an LLM judge to grade a taxonomy?
Because generic judge checks score each node, not the whole tree. In Enterpret's taxonomy research, all six taxonomies passed every generic check while AI-generated ones had 97.7 to 100% of leaf names restating a parent and up to 76.4% branch leakage.
Should the judge use the same model as the classifier?
Avoid it where possible. A judge from the same model family tends to share the classifier's blind spots and agree with its mistakes.
To start from a taxonomy built for structure, not just plausibility, see how Enterpret works. Either way, measure your human-human agreement before trusting any judge number.
Research cited
- Chellingi, P., G, R., & Bagal, V. (2026). From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback. arXiv:2610.09377.
Heading
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.



