The 7 Steps to Turn User Feedback Into Eval Sets

To turn user feedback into eval sets, build from failure themes, not from individual thumbs-downs. The seven steps: find failure themes across every channel, size them by volume, segment and revenue, pull representative examples with their original context, write expected behavior and grading criteria, keep the categories stable, refresh the set as new themes appear, and confirm the theme declines in user feedback after the fix ships. Your eval platform runs the tests. The feedback layer decides which tests deserve to exist.

Most teams get the categories wrong. In Enterpret's study of run-to-run instability across eight frontier models, a raw model asked twice for the top themes in the same feedback changed more than half its theme set at 100 records and nearly three quarters at 5,000 (theme churn 0.552 rising to 0.726). Eval categories built that way cannot map to anything that holds still.

Why thumbs-down traces are not enough

The tracing side is well covered. Langfuse's guide to turning user feedback into evaluation datasets shows how to capture ratings as scores on traces, filter for negative ones, add them to a dataset, use expert corrections as expected outputs, and gate CI on experiment runs. If your main signal is a thumbs rating on a trace, start there.

The gap is everything that never touches a trace. At an AI-native startup, the most specific failure reports live in community channels, Reddit, GitHub issues, app reviews, tickets and, once enterprise deals start, sales calls. A thumbs-down says a response was bad, rarely why, and Langfuse's guide itself notes that some negative ratings are about latency or tone rather than correctness. That gives you bad responses, not a ranked list of the failure modes costing you users.

Asking a general model to cluster a CSV works once (see our guide to leveraging ChatGPT AI for customer feedback analysis). It breaks when you need the same categories next month to compare pass rates across a model swap.

Traces tell you what the model did. User feedback tells you what users needed and what the failure cost you. Build the eval set from the second.

The 7 steps to turn user feedback into eval sets

1. Find failure themes across every channel

Pull ratings, chat logs, tickets, community threads, reviews and call notes into one place, then group by failure mode: "agent edits the wrong file in a monorepo," "invents an API that does not exist," "refuses a valid request." Only model behavior goes in the eval set; UX gaps and missing features go to the roadmap.

If you are stitching sources by hand, start with a system for managing multi-source customer feedback.

2. Size and prioritize by volume, segment and revenue

Rank themes by volume, trend since the last release, segment (free users vs enterprise trials), and attached accounts and revenue. Top themes get blocking CI tests; the long tail gets scheduled runs, in line with Langfuse's advice to keep the gated set small.

Counts only help if they repeat. In the same study, taxonomy-grounded runs showed zero volume disagreement across all 117 valid run pairs, while raw models' counts moved 7.5 to 9.8% on matched themes, enough to reorder a priority list.

3. Pull representative examples with the original context

For each prioritized theme, pick examples that cover its variations, not the five loudest. Keep the original input, prior turns, model and prompt version, account tier, and the user's own complaint in their words. Link the trace when one exists, and strip personal data.

4. Write expected behavior and grading criteria per theme

Write the rubric once per theme, then apply it to every example in it. "Only references functions that exist in the provided repo" is gradable. "Better code" is not. Decide what each criterion needs: a deterministic check, an LLM judge, or human review.

5. Keep categories stable so the set maps to real themes over time

Tag every eval item with a theme ID from a persistent taxonomy, not a label a model invented this week. After a model swap, compare pass rates per theme to see which user problems got better or worse.

86 to 88% Reduction in theme churn when an AI agent is grounded in a persistent taxonomy instead of re-clustering raw feedback on every run (0.104 vs 0.765 for a raw model, same model and data). Source: Enterpret Research, "Same Feedback, Different Answer" (arXiv, 2026)

Caveat: the study measured stability, not accuracy. A stable taxonomy can still be wrong, so review theme definitions in step 4.

6. Refresh the set as new themes appear

After every release and every model change, check for new or spiking themes and add a slice for each one that is model behavior. Move resolved themes to a regression suite, and version the dataset (Langfuse versions every change) so old runs stay comparable.

Spotting new themes fast is its own capability; see customer voice tools for identifying trending product issues.

7. Close the loop: check the theme declines in user feedback

A higher pass rate is half the proof. The other half is the theme's volume dropping in user feedback after the fix ships, normalized for usage, inside the segment that reported it. If the eval passes and complaints continue, the rubric measures the wrong thing. Rewrite it.

This is the same discipline as closing the customer feedback loop at scale, applied to model quality.

Where Langfuse and eval platforms fit

Datasets, experiments and runs belong in your eval platform: Langfuse, Braintrust, LangSmith or Arize Phoenix. The output of steps 3, 4 and 6 lives there. Unifying feedback across channels, keeping categories stable and attaching revenue (steps 1, 2 and 7) happen upstream. For a side-by-side of both layers, see the 6 best tools to turn customer feedback into an AI eval set.

Where Enterpret fits

Enterpret is the user-feedback side, not an eval runner. It brings feedback into one place, including Discord and Reddit (see MCP servers for Discord community feedback), with four capabilities mapped to the steps:

• Adaptive Taxonomy (steps 1, 5, 6): consistent themes that update as you ship, so theme IDs hold across model swaps.
• Customer Context Graph (step 2): joins each theme to accounts, segments and revenue.
• Wisdom and the MCP server (step 3): pull examples in natural language, or from Claude Code and Cursor into an eval file.
• AI agents and alerts (steps 6 and 7): flag new or spiking themes after a release in Slack, Linear or Jira.

Setup is not instant: integrations need connecting and the taxonomy needs processing time before themes can anchor an eval set.

Example: a model swap on an AI coding agent

Picture a team shipping an AI coding agent that upgrades its default model on a Tuesday. By Friday, "edits the wrong file in monorepos" is rising in Discord and tickets, including three enterprise trial accounts. The PM pulls examples with prior turns, writes one rubric (edits stay inside the referenced package), and adds the slice to the gated dataset. The new model fails cases the old one passed. A context fix turns the gate green, and over two weeks the theme falls inside the enterprise segment. What worked: the gate now catches the regression. What did not: nobody wrote that test until users reported it, the case for steps 1 and 6.

What changes next

Evals are becoming a product artifact, not just a model team artifact. When every slice maps to a theme users report, pass rates become a shared language across model, product and sales teams. The eval set should mirror user feedback, refreshed every release, not sit as a benchmark written once.

FAQ

Can I build an eval set from thumbs-down ratings alone?

You can start there; Langfuse makes it straightforward to turn negative scores on traces into dataset items. The limit is context: thumbs say a response was bad, not why, and the most specific failure reports arrive in tickets, community threads and calls. Use thumbs as one input, not the whole set.

Why not just ask an LLM to cluster feedback into eval categories each sprint?

Because the categories will not hold still. Enterpret's research found raw frontier models changed more than half of their top themes between two runs on the same 100 records, while a taxonomy-grounded agent cut theme churn by 86 to 88% with the model held constant.

How often should I refresh an eval set built from user feedback?

After every release and every model change, since either can shift quality overnight. Add slices for new model-behavior themes, move resolved ones to a regression suite, and version the dataset.

Do I still need Langfuse or Braintrust if I use Enterpret?

Yes. Enterpret finds, sizes and tracks the failure themes in user feedback; an eval platform stores datasets, runs experiments and gates CI. Use both, with theme IDs as the join.

If you want the theme layer handled, see how Enterpret works. Then run the seven steps on one theme and tell us where the process breaks.

Research cited

1. Bagal, V., Ganta, R., & Chellingi, P. (2026). Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis. arXiv:2610.08036.