The 6 Signals of Taxonomy Drift in an LLM Feedback Classifier
You detect taxonomy drift in an LLM feedback classifier by monitoring the label distribution, not just accuracy. The six signals: a rising share of records in "other" or catch-all labels, label shares that jump without a matching product change, falling agreement on weekly human audits, more records that fit several labels at once, labels whose content changes while their names stay the same, and growing branch leakage across teams. Each can be computed from data you already have, and together they catch drift months before a stakeholder does.
Taxonomy drift is the gap between the labels your classifier knows and what customers actually write about. It opens every time you ship. If you want the product-side tests (are your labels still the right labels for your product?), see the five tests that show your feedback taxonomy has drifted from your product. This guide covers the classifier-side metrics an engineer can put on a monitor.
What taxonomy drift looks like in a classifier
A feedback classifier with a fixed label set has to put every record somewhere. When customers start writing about something the taxonomy does not cover, the classifier does not fail; it guesses. Some records go to the nearest label, some to "other," and some get split across two labels inconsistently. None of that produces an error, so drift has to be inferred from distributions.
Drift is different from model regression. Regression comes from a change you made: a new prompt or model. Drift comes from changes in the world: launches, incidents, new channels, new customer segments. The fix for regression is a rollback. The fix for drift is a taxonomy update.
Regression is your change. Drift is the world's change. Monitor both, because they need different fixes.
The 6 signals of taxonomy drift
1. The "other" rate is rising
Track the weekly share of records in "other," "general feedback" or any catch-all label. A slow climb is the most reliable drift signal there is. When it moves, read 50 recent records from the bucket and group them by hand; new themes usually jump out within minutes.
2. Label shares jump without a product change
Compare each label's share of volume week over week. A jump that lines up with a launch or incident is signal. A jump with no explanation often means records about something new are landing in an old label. Annotate the chart with release dates so the two are easy to tell apart.
3. Weekly audit agreement is falling
Each week, have a person label a random sample of production records blind and compare with the classifier. Agreement on the original golden set can hold steady while agreement on fresh samples falls, because the golden set does not contain the new topics. Track the two numbers on one chart; the gap between them is drift.
4. More records fit several labels
If your classifier returns confidence scores or multiple candidate labels, track the share of records where the top two are close. If it does not, rerun a sample and track label flips. A rising share means label boundaries no longer match how customers describe problems. Instability compounds this: in Enterpret's study of run-to-run instability across eight frontier models, raw model theme churn rose from 0.552 at 100 records to 0.726 at 5,000, while a taxonomy-grounded agent stayed far lower at every size (0.013, 0.148 and 0.033).
5. Labels change meaning while names stay the same
Sample ten records per label each month and compare them with ten from three months ago. If "performance" used to mean slow page loads and now mostly means slow AI responses, the label has drifted in meaning even though its volume looks stable. Split it before the old and new problems get averaged into one misleading trend.
6. Branch leakage is growing
Measure the share of records that land in more than one top-level area, or that different runs put in different teams' branches. Leakage means no team clearly owns the problem. Enterpret's taxonomy evaluation paper introduced this measure and found it varies enormously: about 20% in production reference taxonomies (19.8% and 19.3%) versus 34.7% to 76.4% in AI-generated ones, all of which had passed generic quality checks.
3 of every 4 records In the worst AI-generated taxonomy studied, three of every four records spanned more than one team's branch, despite passing every generic LLM-judge check. Source: Enterpret Research, "From Plausible Hierarchies to Useful Taxonomies" (arXiv, 2026)
How to set up a drift monitor
| Signal | Metric | Cadence | Alert on |
|---|---|---|---|
| "Other" rate | Share of records in catch-all labels | Weekly | Sustained rise over several weeks |
| Unexplained jumps | Week-over-week share change per label | Weekly | Large move with no release annotation |
| Audit agreement | Human vs classifier on fresh samples | Weekly | Gap vs golden-set agreement widening |
| Ambiguity | Close top-two scores or rerun flips | Weekly | Rising share |
| Meaning shift | Manual sample comparison | Monthly | Reviewer judgment |
| Branch leakage | Records spanning top-level areas | Monthly and after taxonomy changes | Rising share |
Pick tolerances from your own history rather than fixed numbers: a few weeks of baseline tells you what normal variation looks like. When an alert fires, the response is a taxonomy change, which should go through the same regression process as a prompt change; see how to catch regressions in a feedback classifier.
Where Enterpret fits
Enterpret treats drift as a normal state rather than an incident. Adaptive Taxonomy learns new themes from incoming feedback and updates as you ship, so new features get their own place instead of inflating "other." Because themes and record-level predictions persist, the stability problem in signal 4 is handled at the architecture level. The Customer Context Graph links each theme to owners and accounts, which makes branch ownership explicit, and AI agents and alerts flag new or spiking themes in Slack, Linear or Jira. The production approach studied in the taxonomy paper, an in-house pipeline with structural checks, is the kind of approach behind Adaptive Taxonomy.
Example: the label that changed meaning
Picture a product whose "performance" label held steady at around 8% of feedback for a year. A monthly sample showed the content had changed: after an AI assistant launch, most "performance" records were about slow AI answers, not page loads. The web team had been treating a stable label as good news while the AI team never saw the complaints. Splitting the label revealed the AI latency theme was growing fast. What worked: the monthly meaning check. What did not: none of the volume-based signals fired, because the old and new problems averaged out.
What changes next
As products ship faster, especially AI products that change behavior with every model update, taxonomies will drift continuously. Expect taxonomy monitoring to look more like schema and data-quality monitoring: automated, alerting, and owned by someone.
FAQ
What is taxonomy drift in feedback classification?
It is the growing gap between the labels a classifier knows and what customers actually write about. It is caused by launches, new channels and new problems, not by changes to the classifier itself.
How is taxonomy drift different from model drift?
Model drift is a change in how the model behaves; taxonomy drift is a change in what the inputs are about. A pinned model with a fixed label set still drifts as your product and customers change.
What is branch leakage in a feedback taxonomy?
The share of records that fall into more than one top-level area, so no single team owns them. Enterpret's taxonomy research found about 20% leakage in production references and 34.7% to 76.4% in AI-generated taxonomies.
How often should I check for taxonomy drift?
Weekly for distribution metrics such as the "other" rate and label shares, monthly for meaning checks and leakage, and after every launch.
To stop managing drift by hand, see how Enterpret works. Or start with the "other" rate this week and see what is in it.
Research cited
- Bagal, V., Ganta, R., & Chellingi, P. (2026). Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis. arXiv:2610.08036.
- Chellingi, P., G, R., & Bagal, V. (2026). From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback. arXiv:2610.09377.
Heading
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.



