The 6 Ways to Tell Model Quality Complaints Apart From Product and UX Complaints

To tell model quality complaints apart from product and UX complaints, sort every "it's bad" into one of six categories by asking who would have to change something to fix it: output correctness and consistency go to the model and evals team, latency goes to infra, workflow/UX and missing capability go to product and design, and expectation gaps go to docs, onboarding and positioning. Users describing the answer point at the model; users describing the screen point at the product.

The two-question test: would a perfect model with today's UI have prevented this complaint? Would a perfect UI with today's model? If only the first is yes, it is a model complaint. If only the second, it is a product complaint. If both, fix the cheaper one.

Why "it's bad" is the hardest feedback to route

A 2025 Scientific Reports study of 3 million reviews across 90 AI-powered mobile apps annotated 1,000 candidate reviews and found seven distinct types of user-perceived hallucination. Factual incorrectness was 38%, nonsensical or irrelevant output 25%, fabricated information 15%. That is the model side alone. Add slow responses, a hidden control, a missing feature and a mismatched expectation, and one star rating can mean six different bottlenecks.

AI-native teams have two extra problems. A model swap can change quality overnight while the UI stays identical. And as Jeff Gothelf points out, every piece of feedback on an AI feature is one sample from a range of possible outputs, not a report on "the" product.

Stop classifying complaints by sentiment or by feature. Classify them by who owns the fix. A complaint is not a theme until you know whether the next action is an eval, a design change, a roadmap item, or a docs page.

The 6 categories, and the cues that give each away

1. Output correctness: the answer is wrong

Cues: users quote or describe the output itself. "Made up," "doesn't exist," "wrong function," "cited a paper that isn't real," "ignored my instructions." Owner: model and evals team.

2. Consistency: the answer changes

Cues: time words and comparisons. "Worked yesterday," "since the update," "sometimes," "got a different answer when I asked again." Owner: model and evals team, with release management. Timestamp them against your deploy log.

3. Latency: the answer is slow

Cues: "takes forever," "spinning," "timed out," "stopped mid-response." Owner: infra or platform team. Users often call a slow but correct answer "bad."

4. Workflow and UX: the right control is hidden

Cues: users describe the screen, not the answer. "Couldn't find," "how do I," "had to copy paste," "lost my chat," "no way to undo." Owner: product and design.

5. Missing capability: the product cannot do it

Cues: "wish it could," "does it support," "Cursor does X," "need an integration with." Owner: product, as a feature request. Competitor names are the strongest cue, and a churn signal.

6. Expectation gap: the user expected something else

Cues: "I thought it would," "isn't it supposed to," "the demo showed," "why won't it read my whole codebase." The product works as designed; the mental model does not. Owner: docs, onboarding, and marketing. They are easy to misroute, because they read like correctness complaints.

Example complaints mapped to category and owner

The complaints below are illustrative examples written for this guide, not quotes from real users.

The last row matters most. Many real complaints carry no cue. Join them to the conversation, model version and account before classifying.

Better model or better product? Four tests

Once complaints are classified, four tests decide where the next sprint goes.

1. The swap test

Replay the complaining session on your strongest available model. If the complaint disappears, it is a model problem. If it persists, no model upgrade will save you.

2. The deploy-log test

Plot each category against model, prompt, and UI releases. A correctness or consistency spike that lines up with a model change is a regression, not a theme.

3. The behavior-after-output test

Gothelf recommends tracking what users do next: accept, edit heavily, regenerate, or abandon. Regenerate loops point at the model. Abandonment after a correct output points at the workflow.

4. The revenue test

Weight each category by the accounts behind it. Ten prosumer complaints about latency and two enterprise complaints about a missing audit log are not the same priority.

The deploy-log test only works if your categories are stable. If an LLM re-clusters feedback from scratch each week, a "spike" may be the categorizer moving. In Enterpret's study of run-to-run instability across eight frontier models, a raw model asked for top themes on the same 1,000 records had theme churn of 0.765 between runs. An agent grounded in a persistent taxonomy had 0.104.

86 to 88% Reduction in theme churn when an AI agent is grounded in a persistent taxonomy instead of re-clustering raw feedback on every run, with the model held constant. Source: Enterpret Research, "Same Feedback, Different Answer" (arXiv, 2026)

Stability is not accuracy, but without it you cannot tell a regression from categorizer noise.

How to route each category to the right owner

Routing breaks when one complaint lands in two teams' queues. Enterpret's taxonomy evaluation paper measured exactly this: in AI-generated taxonomies, 34.7% to 76.4% of records leaked across more than one top-level branch, versus about 20% in production references built with structural checks. Rules that keep feedback routing and team-based alerting clean:

• One primary owner per record. Secondary tags, never secondary owners.
• Correctness and consistency become eval cases, not tickets. Send them to whoever owns your eval suite.
• Workflow and missing capability go to the product backlog in Linear or Jira with counts and accounts attached.
• Expectation gaps go to docs and onboarding as a weekly digest, the cheapest category to fix.

Where Enterpret fits

Eval and tracing tools like Braintrust, Langfuse and Arize tell you how the model performs on cases you chose, and they are the right home for correctness work. What they do not see is user-side signal across tickets, app reviews, community channels and sales calls. Enterpret covers that side, which is why we compared customer intelligence platforms for AI and LLM products separately from eval tools.

The permutation that matters: Adaptive Taxonomy keeps categories stable as you ship weekly, so before and after a model swap are comparable. The Customer Context Graph joins each complaint to the account and revenue behind it. Wisdom answers "did correctness complaints rise after Tuesday's release?" in natural language, also available in Claude, Claude Code and Cursor via the MCP server. AI agents and alerts flag trending product issues after a release and deliver them to Slack, Jira or Linear. Sources include Zendesk, Intercom, Slack, Discord, Reddit and the Apple App Store and Google Play, so you are managing multi-source customer feedback in one place. The honest gap: the context graph needs integrations and processing time before the joins are useful.

Example: a model swap week

Picture a coding agent team that swaps its default model on a Tuesday and ships a new diff viewer the same week. By Thursday, negative feedback is up and the room splits: roll back, or blame the diff viewer? Classified, the picture separates. Say consistency complaints doubled and cluster on the new model version, while diff viewer workflow complaints are flat. Decision: roll the model back for long-context tasks, keep the diff viewer, and add the consistency cases to the eval suite before the next swap.

What changes next

As models converge, fewer complaints will be fixable by a better model and more only by a better product. The teams that win will prove which is which within days of a release. The working hypothesis: expectation gaps are the most under-counted category in AI products today. Run the six categories on last month's feedback, count each bucket, and note where the scheme breaks.

FAQ

What is the fastest way to tell if a complaint is about the model or the product?

Check what the user describes. If they describe the answer (wrong, made up, different from yesterday), it is a model complaint. If they describe the screen or the steps (couldn't find, had to copy paste, no way to), it is a product or UX complaint. With no cue, replay the session on a stronger model.

Can I use an LLM to classify complaints into these categories?

Yes for a first pass, but keep the categories fixed rather than asking the model to invent them each time. Enterpret's research found raw frontier models had theme churn of 0.765 between runs on the same 1,000 records, versus 0.104 for an agent grounded in a persistent taxonomy; the paper reports an 86 to 88% churn reduction against raw and multi-agent designs. Stable categories are what make before and after comparisons around a model swap meaningful.

Who should own hallucination complaints?

The model and evals team, with each complaint converted into an eval case. First check it is not an expectation gap, which can look like a hallucination in raw text.

How is this different from what eval tools like Braintrust or Langfuse do?

Eval and tracing tools measure model behavior on cases and traces you define. This classification works on what users say across tickets, reviews and community channels, and decides which complaints become eval cases. The two are complementary.

If you are evaluating how to run this classification across every feedback source, see the best AI-native customer intelligence platforms or see how Enterpret works.

Research cited

1. Bagal, V., Ganta, R., & Chellingi, P. (2026). Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis. arXiv:2610.08036.
2. Chellingi, P., G, R., & Bagal, V. (2026). From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback. arXiv:2610.09377.