The 7 Steps to Evaluate an LLM That Extracts Pain Points and Feature Requests From Sales Calls

November 12, 2026

To evaluate an LLM that extracts pain points and feature requests from sales call transcripts, build a gold set of 30 to 50 real calls in which two people mark every pain point and feature request as a quoted span, with the speaker, the type and the intent. Then score the model on seven things: recall (did it find them), precision (are they real), speaker attribution (customer or rep), intent (asked for, asked about, mentioned or rejected), grounding (does the quote exist word for word), normalization (does it map to the right theme) and repeatability (does a second run agree). Extraction fails differently from classification. The costly errors are crediting the rep's words to the customer and treating a passing question as a requirement.

Why sales calls are harder than tickets

A support ticket usually has one author and one problem. A sales call has several speakers, runs for 30 to 60 minutes and mixes real needs with everything else: the rep pitching features, hypotheticals ("if we rolled this out to the whole team..."), competitor comparisons, objections and polite small talk. "Do you support SSO?" might be a hard requirement or idle curiosity. An extractor that cannot tell those apart will inflate your feature request counts with things nobody asked for.

The 7 steps to evaluate an extraction LLM

1. Write the schema and a labeling guide

Decide exactly what the model should return before you test it. A workable schema for each extracted item: type (pain point, feature request, objection, competitor mention), verbatim quote, speaker role, intent, product area and a timestamp. Write a one-page guide with examples of each type and each intent, including the borderline cases. Disagreements between annotators will mostly trace back to gaps in this guide.

2. Build a gold set from real calls

Pick 30 to 50 calls across deal stages, segments and call types, including a few with no product talk at all. Have two people label each call independently, then resolve disagreements. Measure how often they agreed before adjudication; if people disagree on a type, the model will too. The method mirrors building a golden dataset for feedback classification, with spans instead of labels.

3. Match extractions to gold items and score recall and precision

Pair each extracted item with a gold item when they refer to the same moment in the call and the same need. Exact text match is too strict, so use overlap with the gold span or a judge model with a clear rubric, and spot-check the judge. Report recall and precision separately for each type. A single blended score hides the usual pattern: high recall on feature requests, poor precision because rep statements leak in.

4. Check speaker attribution

For every matched item, check that the speaker is the customer, not the rep. Diarization errors in the transcript and models summarizing the conversation both cause misattribution. This one check often moves precision more than any prompt change.

5. Score intent

Count an item as correct only if the intent matches too:

IntentExampleCounts as a request?
Requested"We need audit logs before security signs off."Yes
Asked about"Do you have audit logs?"Track separately
Mentioned"Our old tool had audit logs, I think."No
Rejected"We don't care about audit logs."No, but useful
Rep suggestedRep: "Audit logs are coming next quarter."No

6. Verify grounding

Every quote the model returns should appear in the transcript, word for word or nearly so. This check is cheap and deterministic: search the transcript for each quote and flag the ones that are missing. Paraphrased or invented quotes are the fastest way to lose a sales team's trust in the output.

7. Test normalization and repeatability

Extracted items are only useful if they roll up. Check that each one maps to the right theme in your taxonomy, so "audit trail," "activity log" and "who changed what" count together. Then run the same calls twice and compare. In Enterpret's study of run-to-run instability across eight frontier models, raw models summarizing the same feedback changed most of their theme set between runs, and their counts on matched themes moved 7.5 to 9.8%.

0.552 to 0.726 Average theme churn for raw frontier models between two runs on the same feedback, at 100 and 5,000 records. A taxonomy-grounded agent stayed at 0.013 and 0.033. Source: Enterpret Research, "Same Feedback, Different Answer" (arXiv, 2026)

That study measured open-ended theme analysis, not span extraction, but the lesson carries over: fix the list of themes the extractor maps to, and the counts stop moving.

Set thresholds by use

There is no universal pass mark. Pick thresholds by what the output feeds. If it feeds roadmap counts and revenue-weighted priorities, precision, attribution and intent matter most, because false requests inflate demand. If it feeds discovery, where a PM will read the quotes anyway, recall matters more. Record the thresholds next to the gold set and rerun the eval whenever you change the prompt, the model or the transcript provider. The same discipline applies if you use a judge model to score; here are the rules for LLM judges versus human labels.

Where Enterpret fits

Enterpret ingests call recordings and transcripts alongside tickets, surveys and reviews, classifies them into the same Adaptive Taxonomy and links each call to its account and opportunity in the Customer Context Graph, so a feature request from a call counts together with the same request from a ticket. Through the MCP server, the find_user_quote tool returns verbatim quotes from the records behind an answer. Enterpret's documentation notes that speaker or company attribution is not always available, which is exactly what step 4 tests. For tool comparisons, see tools to extract feature requests and competitor mentions from Gong calls.

Example: the request the rep made

Picture a team whose extractor reported "role-based permissions" as the top feature request from enterprise calls. The gold set showed why: in a third of those calls, the rep had brought up permissions as part of the pitch, and the model credited the customer. After adding a speaker check and a "rep suggested" intent, the request fell to sixth, and "bulk user import," which customers raised on their own, moved to the top. What worked: scoring attribution separately. What did not: trusting a single precision number.

What changes next

As revenue teams lean on call extraction to forecast demand, extraction errors become forecasting errors. Expect evals for call extraction to look more like finance controls: a fixed gold set, a named owner and a rerun on every change.

FAQ

How many sales call transcripts do I need to evaluate an extraction model?

Start with 30 to 50 calls labeled by two people, spread across deal stages, segments and call types. That is usually enough to see precision, recall and attribution problems by type. Add calls when you add new types or segments.

Can an LLM judge score extraction quality?

Yes, for matching extracted items to gold items, if you give it a clear rubric and spot-check its decisions against people. Keep grounding checks deterministic: string search finds missing quotes more reliably than a judge.

What is the most common error when LLMs extract feature requests from calls?

Misattribution and misread intent: counting something the rep said, or a passing question, as a customer request. Score speaker and intent separately from whether the item was found.

Should I split long transcripts into chunks?

Usually not for a single call. Current Claude models such as Sonnet 5.5 and Haiku 5.5 have 1M token context windows, enough for a long call, and chunking can separate a question from its answer. Chunk only when you hit limits, and overlap the chunks.

If you want call extraction that counts together with every other channel, see how Enterpret works.

Research cited

  1. Bagal, V., Ganta, R., & Chellingi, P. (2026). Same Feedback, Different Answer: Measuring Run-to-Run Instability in Frontier-Model Customer Feedback Analysis. arXiv:2610.08036.

Heading

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique. Duis cursus, mi quis viverra ornare, eros dolor interdum nulla, ut commodo diam libero vitae erat. Aenean faucibus nibh et justo cursus id rutrum lorem imperdiet. Nunc ut sem vitae risus tristique posuere.

This is some text inside of a div block.
Related Guides
See all guides

AI That Learns Your Business

Generic AI gives generic insights. Enterpret is trained on your data to speak your language.

Book a demo

Start transforming feedback into customer love.

Leading companies like Perplexity, Notion and Strava power customer intelligence with Enterpret.

Book a demo