The 5 Ways to Check Whether Feedback Is Landing in the Right Theme

In Enterpret research, six feedback taxonomies were run through every standard LLM-judged quality check (name matches description, children fit parents, siblings are distinct), and all six passed. Several of them still spread a third or more of their records across more than one top-level branch. That is the problem with checking categorization by looking at the theme list: the list can look right while the feedback underneath it is landing in the wrong places. The only reliable check starts from the records.

Five checks tell you whether feedback is landing in the right theme: read a random sample from each high-stakes theme, search for a problem you know exists and see where it landed, have two people classify the same sample, rerun the classification on unchanged data, and measure how much feedback lands in more than one top-level branch. The first two test precision and recall, the third sets the ceiling for agreement, and the last two catch the failures a spot-check misses.

What a categorization check needs to measure

A good check scores four properties, not one.

1. Precision. Of the records filed under a theme, how many belong there?
2. Recall. Of the records that describe the problem, how many made it into the theme? Low recall is invisible from inside the theme, which is why it is the more expensive failure.
3. Stability. Does the same input produce the same theme assignments and counts? A structure that is re-derived on every run cannot pass this, which is the case for a persistent, learned structure such as an adaptive taxonomy.
4. Consequence. Are the themes being checked the ones that drive decisions? A customer context graph ties each theme to accounts and revenue, so you can check the themes behind this quarter's renewals first instead of sampling evenly.

The 5 ways to check whether feedback is landing in the right theme

1. Read a random sample from each high-stakes theme

Pull 50 random records from a theme and mark each as belongs, belongs elsewhere, or unclear. The share that belongs is your precision. Do this for the ten themes that feed your next roadmap or renewal decision, not for every theme, and repeat it after any change to the structure. Random matters: the first 50 records sorted by date or relevance will flatter the result.

What a failure tells you: records that belong elsewhere point to a vague theme definition. Records marked unclear point to a missing theme.

2. Search for a problem you know exists and see where it landed

Pick three problems you know customers raised last quarter, because a CSM escalated them or they came up in a call. Search the raw feedback for them with several phrasings, then check which themes those records were assigned to. If most of them sit in the right theme, recall is fine. If they are scattered across adjacent themes or the catch-all, the theme is undercounting. Why feedback counts understate the problem explains how this shows up in reporting.

What a failure tells you: the theme is named for one symptom while customers describe the problem several other ways.

3. Have two people classify the same sample

Give two people who know the product the same 50 records and the theme definitions, and have them classify independently. Their agreement rate is the ceiling: no method, human or automated, will be judged more consistent than two careful people reading the same definitions. If the two disagree on a quarter of the records, fix the definitions before blaming the tool.

What a failure tells you: the boundary between two themes is undefined. Write the rule down and apply it everywhere.

4. Rerun the classification on unchanged data

Run the categorization twice on the same records and compare the theme assignments and counts. Any movement is noise that will show up as a false trend next month. In Enterpret research, counts from raw model runs moved 8 to 10% between identical runs, while in a grounded system a theme's count never changed once it had appeared in two runs. That is a measure of consistency rather than correctness, but a count that moves on its own cannot be trusted to show a real change. Why Claude gives different feedback categories each run covers the cause.

What a failure tells you: the structure is being re-derived rather than persisted, so every run is a new taxonomy.

5. Measure how much feedback lands in more than one top-level branch

Count the share of records assigned to themes under two or more different top-level categories. Some overlap is legitimate, since one message can raise two unrelated problems. A high share means the branches overlap and teams will each count the same feedback. In Enterpret research, production taxonomies sat at about one record in five, while taxonomies built by an off-the-shelf coding agent on two proprietary corpora ranged from roughly a third to three quarters. Testing whether your categories are MECE covers how to fix the boundaries.

What a failure tells you: the top level is organized around customer vocabulary rather than product structure.

Why a theme can look right and still be wrong

Most teams check categorization by reading the theme list: do the names make sense, do the children fit the parents, are the siblings distinct. Those checks are necessary and nowhere near sufficient. They grade the labels, not the assignments, and a structure can be well named while splitting a third of its feedback across branches that different teams own.

The second blind spot is the check itself. A one-time audit tells you the structure was right on the day of the audit. Categorization quality is a property of how the structure is maintained, which is why checks 4 and 5 belong on a schedule rather than in a project. Can you audit or edit an AI-generated feedback taxonomy covers the structural audit; the five checks here cover whether the feedback is actually landing where the structure says it should.

How to run the checks

Monthly, run check 1 on the ten themes driving current decisions and check 4 on the full dataset. Quarterly, run checks 2, 3, and 5. After any change to the taxonomy, rerun check 1 on every theme that was touched. Start with the themes tied to the most revenue, because a misfiled record under a theme nobody uses costs nothing, and a misfiled record under the theme driving a renewal decision costs a great deal.

FAQ

How do you know if customer feedback is categorized correctly?

Read a random sample of 50 records from each important theme to measure precision, then search the raw feedback for problems you know exist to see whether they landed in the right theme. Rerunning the classification on the same data shows whether the assignments are stable.

What is a good accuracy rate for feedback categorization?

Benchmark against human agreement. Have two people classify the same 50 records; their agreement rate is the realistic ceiling. A method that agrees with careful humans about as often as they agree with each other is performing well.

Why do feedback counts change when nothing changed in the data?

Because the categorization is being re-derived on each run instead of persisted. Language models in particular assign different themes to the same records across runs, which shows up as counts that move without any change in what customers said.

How often should you check feedback categorization?

Monthly for the themes driving current decisions, quarterly for a fuller review, and after any change to the taxonomy. Checks that run on a schedule catch drift; a one-time audit only confirms the structure was right on the day it was done.

How does Enterpret keep feedback in the right theme?

Enterpret's adaptive taxonomy learns the theme structure from your feedback and persists it, so assignments and counts stay stable between runs while new themes are proposed as new problems appear. Its customer context graph ties each theme to accounts and revenue, so teams can verify the themes behind their biggest decisions first.

To see how a persistent, learned structure holds up under these checks, explore Enterpret's adaptive taxonomy.

‍