The 7 Steps to Build Evals for an LLM That Classifies Customer Feedback

November 11, 2026

To build evals for an LLM that classifies customer feedback, test three things separately: whether each record gets the right label, whether the same input gets the same label on every run, and whether the taxonomy itself is useful to the teams who own it. The seven steps: freeze a versioned taxonomy, build a labeled golden set, score per label instead of one overall number, add a repeatability test, add whole-tree checks, gate every prompt and model change in CI, and audit production samples on a schedule. Most internal builds only do the first half of step two.

That gap matters because a feedback classifier can look healthy on accuracy and still be unusable. In Enterpret's study of run-to-run instability across eight frontier models, a raw model asked for the top themes in the same feedback twice changed more than half its theme set at 100 records and nearly three quarters at 5,000 (theme churn 0.552 rising to 0.726). No single-run accuracy test would catch that.

Why feedback classifiers need their own evals

Most eval guides assume a fixed label set and a fixed input distribution. Customer feedback breaks both. Your product ships weekly, so the label set changes. New channels, launches and incidents shift what customers write about, so the input distribution changes. And the output is not consumed by one person grading answers: it feeds dashboards, routing rules and roadmap decisions that compare this week to last week.

That gives a feedback classifier three failure modes a generic eval misses:

  • Correct but unstable. The label is defensible on each run, but it differs between runs, so trend lines move without any change in what customers said.
  • Correct but uninformative. Labels are technically right but too broad or too repetitive for anyone to act on.
  • Correct but unowned. Records land in branches that span several teams, so nobody owns the problem.
Accuracy tells you if a label is right. Stability tells you if the trend is real. Structure tells you if a team can act on it. A feedback classifier eval needs all three.

The 7 steps to build evals for a feedback classifier

1. Freeze and version the taxonomy you are testing against

An eval needs a target. Write every label as an ID, a name, a one-sentence definition, and two or three inclusion and exclusion examples. Version the file. When the taxonomy changes, the golden set changes with it, and old runs stay comparable to the version they were scored against.

If you are still designing the label set, decide on depth before you write evals. Two levels (area and issue) is usually enough to start; three levels triples the labeling cost of every later step.

2. Build a labeled golden set from real feedback

Sample real records across every channel you ingest: tickets, app reviews, survey verbatims, call notes, community threads. Stratify so rare but important labels are present, label with at least two people, and keep the disagreements. Our guide to creating a golden dataset for feedback classification evals covers sampling, sizing and adjudication in detail.

3. Score per label, not one overall number

Overall accuracy hides the labels that matter. A classifier that nails "billing question" and "login issue" can post a strong aggregate while missing every "data export broken" record, which is the label your enterprise accounts care about. Report precision and recall per label, plus a confusion matrix for the labels that are most often mixed up.

Eval layerWhat it testsMetricRuns when
Label correctnessRight label per recordPrecision and recall per labelEvery PR
RepeatabilitySame label, same themes, same counts across runsTheme churn, label flip rateEvery PR and model change
StructureLabels informative, branches ownableLeaf restatement rate, branch leakageEvery taxonomy change
Production auditReal traffic still matches the golden setSampled agreement, "other" rateWeekly

For guidance on picking thresholds per label, see what precision and recall to target for an AI feedback tagging system.

4. Add a repeatability test

Run the classifier on the same slice several times and measure how often labels flip, and if you produce summaries, how much the top themes and their counts change. This is the test most internal builds skip, and it is the one most closely tied to whether people trust the dashboard.

86 to 88% Reduction in theme churn when an agent is grounded in a persistent taxonomy instead of re-analyzing raw feedback on every run (0.104 vs 0.765 for a raw model, same model and data). Source: Enterpret Research, "Same Feedback, Different Answer" (arXiv, 2026)

The same study found that once a theme appeared in two grounded runs, its reported count was identical every time, while raw models' counts moved 7.5 to 9.8% on matched themes. Caveat: the study measured stability, not accuracy. A stable classifier can repeat the same wrong answer, which is why step 3 still matters.

5. Add whole-tree checks, not just node checks

If an LLM generated or extended your taxonomy, node-level checks will flatter it. Enterpret's taxonomy evaluation paper found that AI-generated taxonomies passed every generic quality check, yet 97.7 to 100% of their leaf names just restated a parent's name, and 34.7% to 76.4% of records leaked across top-level branches, compared with about 20% in production references. Add two checks: what share of leaf labels add information beyond their parent, and what share of records land in more than one team's branch.

6. Gate every prompt and model change in CI

Treat the prompt, the model version and the taxonomy file as code. Run the golden set and the repeatability test on every pull request that touches them, compare against an approved baseline, and block the merge if any priority label drops below its threshold. Langfuse's regression testing guide recommends keeping the PR gate in the tens to low hundreds of items and running the full set nightly. The full process is in our guide to catching regressions in a feedback classifier.

7. Audit production samples on a schedule

The golden set ages. Every week, sample recent production records, have a human label them blind, and compare. Track the share of records landing in "other" or a catch-all bucket; when it climbs, the taxonomy is falling behind the product. Feed disagreements back into the golden set so it reflects what customers write about now.

What to use for each layer

Label correctness and CI gating fit well in an eval platform such as Langfuse, Braintrust or LangSmith: datasets, experiment runs and score history are what they do. Repeatability and whole-tree checks usually need custom scripts, since most eval tools score one output at a time. Production auditing needs a labeling queue and a way to pull fresh samples by channel and segment.

If the classifier output feeds an AI product's own evals, the same discipline applies one layer up; see the 7 steps to turn user feedback into eval sets.

Where Enterpret fits

Enterpret is built for teams who would rather not own steps 1, 4, 5 and 7 themselves. Adaptive Taxonomy builds a product-specific taxonomy at ingestion and keeps it current as you ship, so themes and record-level predictions are reused rather than re-clustered per question, the taxonomy-grounded approach studied in the stability paper. The Customer Context Graph joins each record to accounts, revenue and owners, and Wisdom and the Enterpret MCP server let you pull labeled records into your own eval tooling from Claude, Claude Code or Cursor. Setup takes real work: integrations need connecting and the taxonomy needs processing time before it can anchor a golden set.

Example: a prompt tweak that passed accuracy and failed stability

Picture a team running a Claude-based classifier over support tickets. An engineer rewrites the system prompt to handle multilingual tickets. Per-label accuracy on the golden set holds within a point. The repeatability test tells a different story: on three runs over the same 500 records, about one record in eight changes label, mostly between "sync error" and "data import failed." The weekly dashboard would have shown a spike in sync errors that never happened. The fix was a sharper definition and two exclusion examples for each label. What worked: the repeatability gate caught it before merge. What did not: the original golden set had too few examples near that boundary, which went into the next labeling round.

What changes next

Feedback classifiers are moving from batch jobs to inputs for AI agents, which raises the bar. An agent that routes tickets or files bugs acts on every label, so instability turns into wrong actions, not just noisy charts. Expect repeatability and structure to become standard eval layers, next to accuracy.

FAQ

How many labeled examples do I need to evaluate a feedback classifier?

Enough per label to see a real change, not just a total. A practical start is a PR gate in the tens to low hundreds of items with every priority label represented, plus a larger nightly set. Labels with only a handful of examples will swing on a single error, so report them separately.

Is accuracy enough to evaluate an LLM feedback classifier?

No. Enterpret's research found raw frontier models changed more than half their top themes between two runs on the same 100 records, while a taxonomy-grounded agent cut theme churn by 86 to 88% with the model held constant. Add a repeatability test and whole-tree structure checks next to per-label accuracy.

Can I use an LLM as a judge to grade feedback labels?

For triage, yes, if you calibrate it against human labels first. Do not rely on it for taxonomy quality: in Enterpret's taxonomy paper, generated trees passed every generic LLM-judge check while 97.7 to 100% of leaf names restated a parent. See LLM-as-a-judge vs human labels.

How often should I rerun feedback classifier evals?

On every change to the prompt, model version or taxonomy file, and on a weekly production sample. Model provider updates and product launches both shift results without any change in your own code.

If you would rather not maintain the taxonomy and stability layers yourself, see how Enterpret works. Either way, add a repeatability test this week and tell us what it finds.

Research cited

  1. Bagal, V., Ganta, R., & Chellingi, P. (2026). Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis. arXiv:2610.08036.
  2. Chellingi, P., G, R., & Bagal, V. (2026). From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback. arXiv:2610.09377.

Heading

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

This is some text inside of a div block.
Related Guides
See all guides

AI That Learns Your Business

Generic AI gives generic insights. Enterpret is trained on your data to speak your language.

Book a demo

Start transforming feedback into customer love.

Leading companies like Perplexity, Notion and Strava power customer intelligence with Enterpret.

Book a demo