The 7 Steps to Create a Golden Dataset for Feedback Classification Evals
To create a golden dataset for feedback classification evals, sample real records from every channel, stratify so small but important labels are covered, label each record with at least two people working independently, adjudicate disagreements and keep them, and split the set into a fast gated slice and a fuller nightly slice. The seven steps: write label definitions first, sample across channels and time, stratify by label and risk, double-label blind, adjudicate and record ambiguity, split and version, and refresh from production every month. A golden set that is never refreshed measures last year's feedback.
The golden set is the most reused artifact in a feedback classifier's life. It decides whether a prompt change merges, whether a model upgrade ships and whether stakeholders trust the numbers. It deserves more care than most teams give it.
What makes a feedback golden set different
Customer feedback is messier than most classification data. One support ticket can mention three problems. A call transcript can cover five products. App reviews are short and emotional, survey verbatims are often half a sentence. The labels themselves are product-specific and change as you ship. A golden set for this data has to represent that mess, or it will reward a classifier that only handles clean, single-topic records.
It also has to reflect structure, not just records. Enterpret's taxonomy evaluation paper found AI-generated taxonomies passed every generic node-level check (name-description match 0.97 to 1.00), yet 97.7 to 100% of their leaf names just restated a parent. A golden set labeled against an uninformative taxonomy will score well and teach you nothing.
97.7 to 100% Share of leaf names in AI-generated feedback taxonomies that just restated an ancestor's name, vs 13.9% and 2.9% in production references. Source: Enterpret Research, "From Plausible Hierarchies to Useful Taxonomies" (arXiv, 2026)
A golden set is only as good as the taxonomy it is labeled against, so fix the label definitions before you label anything.
The 7 steps to create a golden dataset for feedback classification
1. Write label definitions before labeling
For every label: an ID, a name, a one-sentence definition, two or three examples that belong, and two or three near-misses that do not. Check that leaf labels say something their parent does not ("Export to CSV times out" rather than "Export issue" under "Export"). Decide upfront whether records can carry more than one label; most feedback needs multi-label.
2. Sample across channels and time
Pull from every channel you classify in production, roughly in proportion to volume, and from several months rather than one week. A set built from one month inherits that month's incidents and launches. Strip personal data before anyone labels; our guide to what to check before putting customer feedback into an LLM covers the privacy side.
3. Stratify by label and by risk
Random sampling under-represents the labels that matter most: rare, high-cost ones like data loss, security concerns or enterprise-only features. Oversample them deliberately so each priority label has enough examples to be scored on its own. Add a slice of known hard cases: multi-topic records, sarcasm, other languages, and records near the boundary between two similar labels.
| Slice | What goes in it | Why |
|---|---|---|
| Representative | Random sample, proportional to channel volume | Estimates real-world performance |
| Priority labels | Oversampled rare, high-cost labels | Lets each be scored on its own |
| Boundary cases | Records between similar labels | Where prompt and model changes cause regressions |
| Hard inputs | Multi-topic, multilingual, very short, very long | Where new channels break the classifier |
| Fresh production | Last month's records, added monthly | Catches drift the original set cannot |
4. Double-label blind
Have two people label every record without seeing each other's labels or the classifier's output. People who see a model's label tend to agree with it. Use labelers who know the product; a contractor who has never used the feature cannot tell "sync conflict" from "import failure."
5. Adjudicate and record ambiguity
Where labelers disagree, a third person decides, and the original votes are kept. Records that stay contested after discussion are informative: they point to labels that need sharper definitions. Keep an "ambiguous" flag and report classifier performance with and without those records, so you do not penalize the model for disagreements humans could not settle either. Measure agreement between labelers; if it is low for a label, fix the definition before evaluating any model on it.
6. Split and version the set
Split into a development slice (for writing prompts and examples), a gated slice that runs on every change, and a held-out nightly slice. Never tune prompts on the held-out slice. Langfuse's regression testing guide suggests keeping a pull request gate in the tens to low hundreds of items and versioning the dataset so runs stay comparable. Tie each dataset version to the taxonomy version it was labeled against.
7. Refresh from production every month
Each month, label a fresh sample of recent production records and add it as a new slice. Retire examples for features that no longer exist. When the classifier scores well on the original set but poorly on the fresh slice, that gap is drift; see the six signals of taxonomy drift.
How to use the golden set
Score per label, never just in aggregate. Run the gated slice on every prompt, model or taxonomy change. Add a repeatability check by running the same slice several times. In Enterpret's study of run-to-run instability, raw frontier models changed more than half of their top themes between two runs at 100 records, so a single run can pass by luck. The full process is in the seven steps to build evals for an LLM that classifies customer feedback.
This is a different artifact from an eval set for your AI product. If you are turning user complaints into test cases for your own model, see the seven steps to turn user feedback into eval sets.
Where Enterpret fits
Enterpret helps with the parts that are slowest to do by hand. Feedback from 50+ channels is already unified, so sampling across channels and time is a query rather than an export project. Adaptive Taxonomy provides a product-specific, informative label structure to start from, and the Customer Context Graph lets you stratify by segment, plan or revenue. Through Wisdom or the Enterpret MCP server you can pull a stratified sample into Claude Code or Cursor and write it straight to an eval file. The labeling itself still needs people who know your product.
Example: the golden set that missed the launch
Picture a team whose 400-record golden set was built in January. By May, the classifier still scored well on it. A fresh 100-record production slice told a different story: a quarter of the records mentioned a workflow builder launched in March, which had no label and no examples. The team added the label, labeled 60 boundary cases between "workflow builder" and "automations," and made the monthly refresh a calendar item. What worked: the fresh slice exposed the gap in a day. What did not: three months of evals had been green while the classifier missed a launch.
What changes next
As teams run more classifiers and agents on the same feedback, a shared, versioned golden set becomes infrastructure: the reference every model change is judged against. Expect labeling to shift from a one-time project to a small, continuous process owned by product ops.
FAQ
How many examples should a feedback classification golden set have?
Size by label, not in total. Each priority label needs enough examples to be scored on its own, rare labels are oversampled, and the PR gate stays small enough to run on every change, typically tens to low hundreds of items, with the rest nightly.
Should golden set labels come from humans or an LLM?
Humans who know the product, labeling blind. An LLM can pre-sort records for review, but if it writes the labels the set measures agreement with that model, not correctness. See LLM-as-a-judge vs human labels.
How often should a golden set be updated?
Add a fresh production slice monthly and after major launches, retire examples for removed features, and version every change so older runs stay comparable.
What should I do with records labelers disagree on?
Adjudicate them, keep the original votes, and flag the ones that stay contested. Persistent disagreement on a label usually means its definition needs work.
If unifying and sampling feedback is the slow part, see how Enterpret works. Then label one fresh slice this month and compare it with your original set.
Research cited
- Chellingi, P., G, R., & Bagal, V. (2026). From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback. arXiv:2610.09377.
- Bagal, V., Ganta, R., & Chellingi, P. (2026). Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis. arXiv:2610.08036.
Heading
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.



