The 6 Things That Break First When You Build Feedback Deduplication In-House

Deduplicating customer feedback looks like a weekend problem. Embed the text, cluster by similarity, set a threshold, ship it. A capable engineer can have something working against a sample in an afternoon, and the first results look convincing enough that the project gets a slot on the roadmap.

That instinct is right, and the first version usually works. What follows is the interesting part. Six things break first, roughly in this order: near-duplicates that share no vocabulary, the same issue described by different personas, records that arrive in incompatible formats, thresholds that drift as the product changes, merges that destroy history, and cluster definitions that nobody owns after launch. None of them are visible in the prototype. All of them arrive within a few months.

Why teams build this themselves and why that is the right instinct

Teams that build feedback deduplication in-house are not being naive. They usually have the strongest possible case for it: their feedback lives in systems they control, their taxonomy has to match how their product is organized, and the generic version of this problem is genuinely solvable with available models.

They also tend to be right about the alternative. Off-the-shelf tools that group feedback by keyword or by a fixed category list will underperform a thoughtful internal build, because an internal build knows what the product is. The instinct to build is a reasonable read of the market.

What changes the calculation is not difficulty at the start. It is the shape of the maintenance curve, which is flat for about a quarter and then is not.

The 6 things that break first when you build feedback deduplication in-house

1. Near-duplicates that share no vocabulary

The prototype catches obvious repeats. What it misses is two customers describing one problem with no words in common: the export times out and I never get my file are the same defect, and simple similarity scoring treats them as unrelated. Fixing this means moving from surface similarity to something closer to intent, which is a different engineering problem than the one you scoped.

2. The same issue described by different personas

An admin, an end user, and a buyer describe one product gap in three registers, at three levels of abstraction, with three different ideas of what is broken. Clustering that treats all feedback as one population will either split those three or collapse distinctions that matter. Handling it well requires knowing who is speaking, which means the identity data has to be present at cluster time rather than joined later.

3. Records that arrive in incompatible formats

A two-line app store review, a forty-message support thread, and an hour-long sales call are not comparable units. A call can hold ten distinct pieces of feedback across several product areas, so treating the transcript as one record produces a cluster that is technically correct and analytically useless. Most in-house builds start with tickets, work well, and hit this wall when the second channel arrives. See extracting feature requests from Gong calls for how different that unit of work is.

4. Thresholds that drift as the product changes

Similarity thresholds are tuned against the corpus you had when you tuned them. Ship a new surface, acquire a different customer segment, or change your pricing page, and the language distribution shifts underneath the threshold. Clusters that were clean start merging things they should not, and the symptom is not an error, it is slowly degrading quality that nobody has an alert for.

5. Merges that destroy history

Eventually two clusters need combining, and the question becomes what happens to everything already filed under both. If the merge does not map old clusters onto the new one, the combined count starts from the merge date and every trend spanning it becomes unreadable. This is the layer teams most often skip in the first build, because it only matters once there is history worth protecting.

6. Cluster definitions that nobody owns after launch

The hardest one. The person who tuned the model understood why the boundaries sit where they do. Six months later they are on another project, clusters are drifting, and the people who notice cannot change the logic. The system does not fail, it degrades, and degradation without an owner is indistinguishable from the tool simply not being very good.

The layer that is harder than it looks

Deduplication and taxonomy are usually treated as two projects. They are one.

A cluster is a category. The moment you group feedback, you have made a claim about which distinctions matter, and that claim needs to stay true as the product changes. Every problem on the list above is really the same problem at different depths: the structure was fixed at a point in time and the product kept moving.

Which is why the teams that get furthest with an internal build tend to conclude that the classification was never the hard part. Reading feedback and grouping it is solved. Keeping the grouping accurate through a year of shipping, across channels with different shapes, with history intact and an owner who can change it, is the part that consumes engineering attention indefinitely. An adaptive taxonomy is a bet on that specific problem: the structure is derived from the feedback continuously rather than defined once, so new phrasing for an existing problem attaches to the existing group instead of opening a parallel one.

What to decide before you build more

The useful question is not build or buy. It is which layer you want to own.

Owning ingestion and routing is usually worth it, since those are specific to your systems and change slowly. Owning the clustering model is defensible for a quarter and expensive after that, because the maintenance is continuous and the work is not differentiating. Owning the context layer, where feedback ties to accounts and revenue, is where internal builds most often stop short, and where a customer context graph does work that is hard to replicate with a join.

Before adding to the internal build, get an honest answer on three things: who owns the cluster definitions in a year, what happens to your history the first time two clusters merge, and how the thresholds get retuned after the next launch. If those answers are uncomfortable, that is the real finding, and it is more useful than another round of tuning.

FAQ

Is building feedback deduplication in-house a mistake?

No. The first version usually works and the instinct behind it is sound. The cost shows up in maintenance rather than in the build, which is why teams that regret it tend to regret the second year rather than the first.

What breaks earliest in an in-house build?

Near-duplicates with no shared vocabulary, usually within weeks of running against real data rather than a sample. The second wave arrives when a channel with a different record shape gets connected.

Why do clustering thresholds degrade over time?

Because they are tuned against a snapshot of customer language, and language shifts as the product, pricing, and customer mix change. The degradation is gradual and produces no error, so it is typically noticed by someone losing confidence in the output rather than by monitoring.

How does Enterpret approach this differently?

Enterpret's adaptive taxonomy derives the structure from the feedback continuously rather than fixing it at tuning time, and matches on meaning rather than surface similarity, so two customers describing one problem in different words land together. The customer context graph keeps account, segment, and revenue attached through that process, which is the layer internal builds usually postpone.

Can an internal build and a platform coexist?

Often, yes. Teams keep the ingestion and routing they have built and move the classification and context layers, which is where the recurring engineering cost sits.

If you are weighing what to keep building, see platforms for a unified feedback taxonomy.

‍