The 6 Maintenance Jobs That Eat Engineering Time in an In-House Feedback Analysis System
Maintaining an in-house feedback analysis system takes engineering time in six recurring jobs: keeping connectors alive, updating the taxonomy, running evals on every prompt and model change, explaining and fixing numbers stakeholders question, joining feedback to account and revenue data, and controlling LLM cost as volume grows. The build is a one-time cost; these six jobs recur every month. The honest answer to "how much engineering time" is to estimate each job from your own release cadence, channel count and stakeholder demand, which this guide shows how to do.
Published figures for this vary widely because the inputs vary widely: one channel and a stable label set is a side project, while ten channels, weekly releases and a sales team asking for account-level answers is a standing team. The jobs are the same either way. Only the frequency changes.
Why maintenance is the real cost
The first version of an LLM feedback pipeline is quick to build, which is why build estimates are usually optimistic. What the estimate misses is that the system depends on four things that change on their own schedule: your product, your data sources, the model provider and the questions people ask. Each change creates work. Our breakdown of the hidden costs of building customer feedback analytics in-house covers the budget view; this guide covers the engineering hours.
The build happens once. The maintenance happens every release. Estimate it per release, not per year.
The 6 maintenance jobs
1. Keeping connectors alive
Every source (support desk, app stores, survey tool, call recorder, community, CRM) has an API that changes, rate limits that bite at volume, and auth tokens that expire. Each new channel also brings new text shapes that need parsing and deduplication; see what breaks first when you build feedback deduplication in-house. Driver: number of channels.
2. Updating the taxonomy
Launches, renames and deprecations mean new labels, merged labels and relabeled history. Someone has to notice the change, edit definitions and examples, and backfill. Throwing more compute at taxonomy generation does not remove this job: in Enterpret's taxonomy evaluation paper, the heaviest generation run on each corpus (9.5M and 12.2M input tokens) was that corpus's worst on branch leakage and diversity. Driver: release cadence.
3. Running evals on every change
Every prompt edit, model upgrade and taxonomy update needs a regression run, a record-level diff and a review. The golden set needs a monthly refresh from production. See how to catch regressions in a feedback classifier. Driver: number of changes per month.
4. Explaining and fixing questioned numbers
When a theme spikes or a count disagrees with what support hears, an engineer investigates. Much of this work traces back to instability rather than real change. In Enterpret's study of run-to-run instability across eight frontier models, raw models' counts moved 7.5 to 9.8% on matched themes between runs of the same question on the same data. Every one of those moves can become a Slack thread. Driver: number of people reading the output.
5. Joining feedback to account and revenue data
Sooner or later someone asks which accounts are affected and what they pay. Each new slice (plan, segment, ARR band, open deals, CSM owner) is a join between feedback and CRM or warehouse data, with identity matching across sources. Driver: number of business questions.
6. Controlling LLM cost as volume grows
Designs that send raw feedback to a model for every question get more expensive as data grows. In the same stability study, the experiment's raw-model runs cost $8.79 at 100 records, $51.25 at 1,000 and $143.25 at 5,000, while a taxonomy-grounded agent stayed around $15 to $18 at every size (excluding ingestion cost). Engineers end up adding caching, sampling and batching, each of which needs its own maintenance. Driver: volume and query frequency.
$143.25 vs about $15 to $18 Cost of the experiment's runs at 5,000 records for a raw model sent the full corpus, vs a taxonomy-grounded agent at the same size. Source: Enterpret Research, "Same Feedback, Different Answer" (arXiv, 2026)
How to estimate your own maintenance load
Use a simple worksheet: for each job, multiply how often it happens per month by how long one occurrence takes your team. Pull the frequencies from your own history (release notes, connector incidents, Slack threads asking "is this number right?"), not from vendor claims, including ours.
| Job | Frequency input | Where to find it | Hours per occurrence |
|---|---|---|---|
| Connectors | Connector incidents per month | On-call log, failed job alerts | Your median fix time |
| Taxonomy | Launches and renames per month | Release notes | Edit + backfill + review |
| Evals | Prompt, model and taxonomy changes per month | Git history | Run + diff + review |
| Questioned numbers | "Is this right?" threads per month | Slack search | Median investigation time |
| Context joins | New slices requested per month | Data team tickets | Join + identity match + test |
| Cost control | Cost alerts or optimization tickets | Cloud bill, tickets | Median ticket time |
Then add two adjustments. First, ownership: if one person holds the context, double the investigation times when they are out. Second, growth: channels, releases and stakeholders all tend to increase, so project the next four quarters, not just this one. If the total is creeping toward a full engineer, compare it against the build vs buy signals.
Where Enterpret fits
Enterpret takes over jobs 1, 2, 5 and most of 4 and 6. It ingests from 50+ feedback channels, Adaptive Taxonomy updates product-specific themes as you ship, the Customer Context Graph joins feedback to accounts, revenue, tickets and owners, and persistent record-level predictions keep counts stable and query cost flat as data grows. Wisdom and the Enterpret MCP server let your own tools and agents query it from Claude, Claude Code, Cursor, Codex, Glean or Notion. You still own job 3 for anything you build on top, and initial setup (connecting integrations, taxonomy processing) takes real time.
Example: the worksheet that changed the plan
Picture a team that believed its feedback pipeline took "a few hours a month." Filling in the worksheet from Slack and git history told a different story: weekly releases meant taxonomy edits every sprint, and the sales team's account questions had turned into a recurring data ticket. Most of the time sat in jobs 2, 4 and 5, none of which had ever been on a roadmap. What worked: counting from real logs instead of memory. What did not: the original build estimate, which only counted job 1.
What changes next
As agents start consuming feedback directly, jobs 3 and 4 get heavier: an agent acting on an unstable label creates real work, not just a confusing chart. Maintenance load will become the main input to build vs buy decisions, ahead of license price.
FAQ
How much engineering time does an in-house feedback analysis system take?
It depends on channels, release cadence and stakeholder demand. Estimate it from six recurring jobs (connectors, taxonomy, evals, questioned numbers, context joins, cost control) using your own logs for frequency and fix time.
What is the biggest hidden maintenance cost?
For most teams, taxonomy updates and investigating numbers people question. Both scale with how fast you ship and how many people read the output, and neither appears in a typical build estimate.
Does LLM cost grow with feedback volume?
It does if raw feedback is sent per question. In Enterpret's stability study, raw-model runs cost $8.79 at 100 records and $143.25 at 5,000, while a taxonomy-grounded agent stayed around $15 to $18 at every size.
Can a more powerful model reduce maintenance?
Not on its own. Enterpret's taxonomy research found the heaviest generation runs produced the worst branch leakage and diversity, and stability gaps were not closed by prompt variants.
If the worksheet total surprises you, see how Enterpret works, or read the seven things you would have to build to run customer intelligence in-house.
Research cited
- Bagal, V., Ganta, R., & Chellingi, P. (2026). Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis. arXiv:2610.08036.
- Chellingi, P., G, R., & Bagal, V. (2026). From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback. arXiv:2610.09377.
Heading
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.



