How to Analyze Millions of Pieces of Customer Feedback

Years ago, a customer told us over lunch that she loved demoing our product to her colleagues. She just always opened the demo with a caveat: "Treat the numbers as indicative. They're normally 30 to 40 percent of what they actually should be."

We build a customer feedback analytics platform. The numbers were ours.

She wasn't angry. That was the worst part. She had measured the wrongness, built her workflow around it, and moved on. Our product was reading feedback beautifully, counting it badly, and every insight it produced arrived pre-discounted by a person who had learned exactly how much to trust us.

This article is about how to never earn that caveat. It's about turning millions of pieces of raw customer feedback into numbers a team can bet a roadmap on. The answer has a name, a feedback taxonomy, and a set of design rules we learned mostly by getting them wrong first. We'll show you the rules, the reasons behind them, and a live 653,456-record example you can poke at.

Building is cheap now. Knowing what to build is not.

Code generation collapsed the cost of shipping software. Most teams now build faster than their roadmap process can decide what's worth building, which means the bottleneck has moved. When everyone can build fast, the advantage goes to whoever figures out what to build and learns it faster than the competition.

Speed raises the stakes here. Every wrong thing you ship quickly is still waste, and a fast team pointed the wrong way isn't merely wasting a sprint. It's moving away from the answer with great conviction, and it will spend a future sprint walking back.

The most reliable signal for direction is customer feedback. Nobody disputes this. Companies don't fail at "listen to your customers" because they disagree with the idea; they fail because feedback arrives as millions of unstructured fragments (support tickets, app reviews, sales calls, surveys, community threads, churn interviews) and at that scale nobody can read it all.

AI seems like the obvious fix, since models can now read everything. But after processing hundreds of millions of pieces of feedback, we learned something uncomfortable about where the difficulty actually lives.

LLMs can read. They can't prioritize.

Why does raw AI extraction fail at scale?

A large language model is excellent at understanding one piece of feedback. It is unreliable at staying consistent across tens of millions of them.

Ask a model to freely extract topics from your feedback and you get labels that are each individually reasonable and collectively useless: "login issues," "problems logging in," "sign-in failures," "can't access account." Four labels. One problem. And no way to discover, after the fact, that the biggest issue in your product is four times bigger than any dashboard shows.

There's a second failure, and it's sneakier. Ask a model the same question about the same feedback twice and you can get different answers. Not wildly different. Different enough that the ranking changes, which is the part your VP was going to act on. If a system disagrees with itself even a tenth of the time, that disagreement is a ceiling on everything built downstream of it, and the wobble grows as the pile of feedback does.

Both failures come from the same root: broken math. And broken math is fatal, because the entire point of feedback analysis is prioritization. That's what the caveat at lunch was really about. The product read every record correctly; the counts underneath disagreed with each other; so a human learned to discount everything it said. If you can't count it, you can't prioritize it.

The gap between trustworthy reading and untrustworthy counting is the problem a feedback taxonomy exists to solve. It turns tagging feedback from a guessing game into a counting game.

What is a feedback taxonomy?

A feedback taxonomy is a controlled, hierarchical vocabulary for classifying customer feedback. It defines, in advance, the complete set of labels feedback can receive: the aspects of your product and business that customers talk about, and the themes of what they say about each aspect, with every theme carrying an intent. Each piece of feedback is then classified against this fixed structure instead of being labeled ad hoc.

That one constraint changes the nature of the task. The model no longer invents a label for each record; it picks from a closed set. Counts become comparable across time, sources, and teams. Duplicates can't fragment. Quality becomes measurable, because precision and recall only mean something against a fixed label space.

Database engineers have known this principle for fifty years: schema comes before writes. You don't let every insert invent its own columns. A feedback taxonomy is the schema for the voice of your customer. (Yes, we just compared the voice of your customer to database administration. It's less romantic. It's also why the numbers add up.)

What does a taxonomy look like in practice?

About the example: everything below comes from a live demo workspace we built on Duolingo's public feedback: app-store reviews, community posts, and other openly available sources. Duolingo is not an Enterpret customer and wasn't involved. We built this workspace for demonstrations, precisely so we could show a real taxonomy at real scale without touching anyone's private data. It currently holds 653,456 classified feedback records.

The upper three levels are the aspect hierarchy. An L1 is a product area, an L2 is a feature inside it, an L3 is a specific capability. Below is the actual structure with live counts, presented the way our product presents it: as columns you walk through, left to right. Click any row.

Notice the shape while you're in there. Performance / Reliability and Usability sit at the top level as their own areas rather than being repeated inside every feature, because "the app is slow" is a fact about a capability and a quality dimension, and the structure lets it be both without inventing duplicate nodes. Depth is expensive. Every level a classifier has to descend is a place it can go wrong, so the tree stays flat and wide, and no level exists just to hold a list that could live one level up.

Below the aspects sit the last two levels, themes and sub-themes: what customers are actually saying, with an intent attached. Here is one complete tagged path, all five levels, with real record counts at every step:

Because all 1,671 records under that complaint carry the same label, the number 1,671 means something. You can trend it across releases, weigh it against the 577 records of praise next door, slice it by user segment, and decide whether it outranks the next candidate on the roadmap. The entire value of a taxonomy is visible in that one path.

Why do teams need a taxonomy?

For humans, a taxonomy is a shared language. Product, support, CX, and leadership all mean the same thing by "Speaking Exercise issues," so the conversation stops being about what to call the problem and starts being about what to do with it.

That shared language pays each team differently. Product gets feedback tied to aspects they can actually build against, so the roadmap reflects what customers keep asking for instead of whoever shouted loudest last week. Support gets routing that works: a ticket goes to the queue that owns the aspect, not to whoever opened it first. CX gets the trend line, and can watch an aspect climb week over week before it spikes into a fire. Leadership gets a rollup that survives being drilled into, where the top-line number and the underlying detail tell the same story. And a new PM onboards into the customer's world in an afternoon instead of a quarter, because the map already exists.

Why do AI agents need one even more?

This is the part that changed recently. Agents now read feedback, answer questions about it, monitor quality, draft PRDs, and file tickets. Everything an agent does downstream inherits whatever structure sits underneath it, including the absence of one.

A taxonomy gives agents four things raw text can't. A bounded action space: in our production pipeline the model's output is literally an index into a list of existing taxonomy paths, so it cannot emit a category that doesn't exist, no matter how creative it's feeling. Disambiguation: the word "refund" alone doesn't tell you whether the problem is billing, a broken feature the user wants their money back for, or a policy complaint, and a taxonomy with well-described nodes forces the classification to the root cause. Repeatability: same record, same taxonomy, same answer, which is what makes an agent's output something a human can audit and correct. And local fixes: when precision slips in one area, you sharpen that node's description and leave the rest of the system alone.

Without a taxonomy, every agent re-derives the shape of your product from scratch, on every task, slightly differently each time. That's the "four labels, one problem" failure again, except now it's automated, and it files tickets.

The standard: what we learned building taxonomies at scale

The principles below come from building and operating taxonomies over hundreds of millions of feedback records, and from the scars of getting each one wrong at least once first.

Two layers, two vocabularies, two clocks

The upper layer, the aspects, describes your product's topology: what customers talk about. The lower layer, the themes, is the voice of the customer: what they're saying about it, always carrying one of four intents (complaint, help, improvement, or praise).

The two layers are built from different sources, and we learned this one backwards. Our first pipeline clustered the feedback itself and derived the structure from the clusters. It felt rigorous. What it actually produced was a taxonomy shaped by whoever complained loudest that quarter: a funhouse mirror of the product, with a huge lobe for last month's outage and nothing at all for the feature nobody had discovered yet. So we reversed it. Aspects now come from what the product is, meaning your documentation, your knowledge base, and your product's public surface. Themes come from what customers say, generated from real feedback with citations.

The vocabularies differ for the same reason. Aspect names use the product team's own terminology. We call them keywords precisely because they work as searchable markers, the terms your team already types into search boxes and recognizes on sight. Theme names use the customer's language, because a theme is only useful if the team reading it recognizes their users' actual words in it.

The clocks differ too. The product map changes slowly and your team owns it; the themes change as fast as the feedback does. Keeping the layers separate keeps the structure stable while the insights stay fresh. Mixing them, which is the single most common taxonomy mistake we see, gets you a structure that is somehow both stale and jittery.

MECE: every insight gets exactly one home

Every level of the taxonomy must be MECE: mutually exclusive, collectively exhaustive. Mutually exclusive means an insight fits exactly one label per level, so nothing is double-counted. Collectively exhaustive means every piece of feedback has a home, so nothing silently disappears. It sounds like consulting jargon, and it is consulting jargon. It also happens to be the property that makes the numbers true.

Here's the failure in miniature. Suppose your performance area has two sibling capabilities: Lesson Loading Speed and Lesson Loading Delay. Read them again. A delay is just slowness by another name, so every complaint that fits one fits the other. The classifier faces a coin flip on every record, the volume splits roughly in half, and both labels now look half as urgent as the problem actually is. Neither makes the roadmap cut. You built two labels and lost one problem.

The fix is never "pick better synonyms." It's to split siblings along a dimension one piece of feedback cannot straddle. The demo's Performance area splits by stage: is the app slow to load, slow to respond to your tap, or slow to finish what you submitted? A complaint happens at one stage. One home, every time.

We now treat sibling overlap as a pipeline invariant rather than a cleanup task, and we earned that lesson the annoying way: an early version of our duplicate detection compared themes across different branches of the tree but never among siblings under the same parent, which is exactly where overlap does its damage. Customers found the duplicates before we did. Twice.

The mirror-image mistake is real too. Sounding similar isn't overlap: "slow performance" and "unpredictable performance" stay separate nodes, because a user suffering one is not suffering the other. The merge test is the coin-flip test run in reverse. Merge only when future feedback would have one obvious home either way.

And overlap isn't hypothetical. Our own demo has a live one. Profile Visibility currently sits in two different branches of the tree, and a real question ("is my profile hidden?") splits almost exactly in half between them:

Whose voice is it?

There's a failure mode upstream of all this structure, and it embarrassed us enough to deserve its own rule. Feedback channels carry more than customer voices. Sales calls contain the sales rep. Support tickets contain the agent. Community threads contain your own PMs, cheerfully explaining the roadmap.

In one workspace, call recordings meant the pipeline spent months counting a sales team's own demo pitches as customer praise. The dashboards looked wonderful. The sales team was, to be fair, extremely positive. A related audit found a fifth of "customer complaints" in one channel were actually a seller narrating hypothetical problems the product would solve. The taxonomy was fine; the voices feeding it weren't eligible to speak.

So: decide whose speech counts as feedback before anything gets tagged, and filter by speaker and intent at ingestion. No taxonomy can fix a count that was fed the wrong voices. Garbage in, confidently quantified garbage out.

Structural honesty: General and Miscellaneous nodes

Not all feedback is specific. "App is terrible!" mentions no feature. Forcing vague feedback into precise buckets fabricates precision; the numbers get cleaner and mean less. So the standard includes two special nodes at every level, and the distinction between them matters.

General is vague on purpose. Miscellaneous is specific but unmapped. A General node means the customer addressed the parent area and said nothing more detailed. "The lessons are great" stops at Lessons, honestly. A Miscellaneous node means the customer said something precise that matches nothing in the taxonomy yet: they named a capability you haven't mapped. The first is a statement about the feedback. The second is a statement about your taxonomy. Conflate them and you've hidden your blind spots inside your customers' vagueness, which is a very tidy way to never find them again.

Miscellaneous doubles as a nursery. When similar records pile up in a Miscellaneous bucket, they cluster and graduate into a new named sibling, so the taxonomy grows out of evidence instead of somebody's opinion in a naming meeting. You can watch a nursery filling up in the demo right now:

We watch the Miscellaneous share the way an infrastructure team watches uptime. In one internal analysis, a workspace where roughly a third of feedback landed in catch-alls got the verdict it deserved: when everything is misc, nothing is useful. Catch-all volume is a direct measure of taxonomy debt.

Hierarchical continuity: the counts must reconcile

If 113,461 insights are attributed to Lessons & Practice, then its children, plus its General and Miscellaneous buckets, must account for exactly 113,461. Nothing lost, nothing double-counted, at every level of the tree. Here is that property holding, live, in the demo:

This sounds obvious, and almost no feedback system has it. The original sin of feedback analytics is tagging topics with one system and reasons with another, then hoping the counts agree. They never do, and that disagreement is precisely what produced the "30 to 40 percent" caveat at lunch. Continuity is only achievable when all five levels are predicted together, as one complete path, by one system. The theme anchors the count, the aspects above it inherit it, and the arithmetic holds because it was never given a chance to drift.

Evidence, or the node doesn't exist

Every theme must cite the real feedback records that justify it. In our creation pipeline, a theme without citations is an invalid output, and "no themes found" is an acceptable answer where an invented one is not. A node without evidence is a hallucination with a label, and it's worse than no node at all because it lends fake structure to dashboards. The same rule runs up the tree: if a capability can't be verified against your documentation or the product's public surface, leave it out. An aspirational taxonomy is just fiction with indentation.

Names are an API

Node names get read by humans and machines thousands of times a day, so the standard treats naming as a contract. Each name carries one idea. Conjunctions are banned outright, because "Pricing and Billing Issues" is two labels hiding in one string. Phrasing carries the intent ("Difficulty Saving Drafts" tells you more than "Saving Drafts" ever will), and a complaint names the problem rather than a proposed solution.

Descriptions matter even more than names, because the classifier reasons against them. In our pipeline a node without a description is a hard error; classification refuses to run. Descriptions are written to disambiguate against siblings ("assign here when…, not when…"), which is what makes fixes local. In one retrieval experiment, representing nodes by their descriptions instead of bare path strings lifted candidate-retrieval recall from 48.7% to 92.3%. The words you put on a node are a performance lever. We were surprised too.

Why build the taxonomy first and tag second?

The most transferable architecture lesson we have is to separate the expensive thinking from the repeated work.

Construction is the expensive thinking. It runs once per product, then incrementally: reasoning models synthesize your documentation and samples of real feedback into the aspect hierarchy, themes get generated from retrieved feedback with citations, and humans review the result. Tagging is then a cheap classification task performed millions of times against that fixed structure. One design choice from our pipeline says more than any diagram: the filter that narrows candidate labels is built on the rule that dropping a correct candidate is ten times worse than keeping a wrong one, because a dropped candidate is gone forever while a kept-but-wrong one still gets caught downstream. Retrieval errors are permanent. Classification errors are cheap. The whole pipeline is arranged around that asymmetry.

Splitting the work this way also makes change safe. Taxonomy edits ship like database migrations: versioned changesets with explicit mappings from old paths to new, re-tagging exactly the affected records. Merging two duplicate nodes physically re-links every historical record onto the surviving path, so the merged node's trend line is continuous from day one. You never trade history for tidiness.

Our own evaluations keep pointing at the same place:

• 68% — of precision failures in a 242-node, ~1,200-prediction audit traced to taxonomy structure, not the model.
• 48.7 → 92.3 — retrieval recall (%) when nodes were represented by descriptions instead of bare path names.
• 58 → 94 — production recall (%) in one period after expanding taxonomy coverage, with no model change.

Honest footnote: these were specific subsystems on specific workspaces, with the sample sizes internal audits have rather than the ones academic papers have. We'd hedge harder if the three of them didn't keep pointing the same direction.

In the agent era, the bottleneck is taxonomy design, not model capability.

Building your own? Start here.

If you take one practical instruction from this article, take this one: organize the upper levels around a structure your team can verify. For most companies that's the product's structure. For some it's the org chart, where teams own product areas anyway. The specific choice matters less than the test it must pass: an owner should be able to read their slice of the taxonomy and say "yes, that's my area, and nothing is missing." A taxonomy nobody can verify is a taxonomy nobody will trust, and trust is the entire point of the exercise.

Three more habits that follow from everything above:

• Build the aspect layer from your knowledge base and documentation, not from the feedback. Docs give you the product's real shape, in the vocabulary your team already searches with. Feedback-derived structure gives you a map of last quarter's complaints.
• Let feedback build the theme layer, with receipts. Themes come from real records, cite real records, and graduate out of the Miscellaneous nursery only when volume earns them a name.
• Run the coin-flip test on every set of siblings. Take a plausible piece of feedback and ask five colleagues where it goes. Two answers means an overlap, and your counts are already splitting.

The taxonomy is one layer of the customer context graph

A taxonomy tells you what customers are saying. It doesn't tell you who they are, what they pay, where they are in their journey, or what happened after they spoke.

That's the job of the customer context graph: a connected model of customer and business reality in which taxonomy nodes are first-class objects alongside accounts, users, revenue, support tickets, sales opportunities, competitors, and work items. The taxonomy is the join key between unstructured voice and everything else you know. It's what turns "speech recognition complaints" into "speech recognition complaints from at-risk premium accounts, up 40% since the last release, linked to these three open engineering issues."

Agents query this graph the way engineers query a database, traversing typed relationships instead of re-reading raw text. Trustworthy quantification begins with the taxonomy. The graph is what makes it decision-grade.

From feedback to outcome

Classification was never the goal. It's the intermediate step that makes action possible.

Once the taxonomy holds and is joined to full customer context, the same systems that read the feedback can act on it: flag a regression the week it starts, draft a PRD grounded in cited verbatims and affected revenue, open the tracker issue against the team that owns the aspect, and close the loop with the exact customers who reported the problem once the fix ships.

Building is cheap now. Building the right thing fast is the moat, and the right thing is written down every day in your customers' own words, hundreds of thousands of them. A feedback taxonomy is the schema that turns those words into numbers worth trusting.

Somewhere out there, a customer is demoing your product's numbers to her colleagues. The standard exists so that she opens with the insight, not the caveat.

FAQ

What is a feedback taxonomy?

A controlled, hierarchical set of labels for classifying customer feedback. It has two parts: product aspects (what customers talk about, in the product's vocabulary) and themes with intent (what they're saying, in the customer's vocabulary). Because every record is classified against a fixed structure, feedback becomes quantifiable and comparable over time.

How is a taxonomy different from AI topic extraction?

Topic extraction generates a fresh label for each record, which produces overlapping near-duplicates that cannot be counted reliably, and the answers drift from run to run. A taxonomy is built first, and every record is classified against it. The label space stays closed, counts stay consistent and repeatable, and precision and recall can be measured.

What does MECE mean in a feedback taxonomy?

Mutually Exclusive, Collectively Exhaustive: each insight fits exactly one label per level (no double-counting), and every piece of feedback has a home (no silent gaps). MECE is what makes feedback counts trustworthy.

Should the taxonomy mirror my product structure or my team structure?

Whichever your team can verify and own. Usually that's the product's structure; sometimes it's the org chart, where teams map cleanly to product areas. The test is ownership: someone should be able to read each slice and confirm it's complete and correct. Build it from your documentation so the labels use terms your team already searches with.

Why do AI agents work better with a taxonomy?

A taxonomy bounds the agent's label space so it cannot hallucinate categories, resolves ambiguity through node descriptions, makes outputs repeatable and auditable, and lets teams fix errors by editing structure instead of retraining models.

How often should a feedback taxonomy change?

Continuously, but through governed changes. New patterns accumulate in Miscellaneous buckets and graduate into dedicated nodes when volume justifies them. Merges and renames ship with explicit old-to-new mappings so historical trends survive every edit.

How many levels should a feedback taxonomy have?

Enough to be specific without becoming unmaintainable. The standard we converged on is five: three aspect levels (area, feature, capability) plus theme and sub-theme. Every record receives complete five-level paths, with General nodes filling any level the feedback doesn't specify.

The worked examples and live counts in this article come from a demonstration workspace Enterpret built on publicly available Duolingo feedback (app-store reviews, community posts, and other open sources). Duolingo is not an Enterpret customer and was not involved in this article.

Enterpret builds and maintains feedback taxonomies automatically, from your knowledge base, your product's public surface, and the feedback itself, then classifies every piece of customer feedback against them. See it live →

‍