{"id":11936,"date":"2026-08-26T12:48:44","date_gmt":"2026-08-26T16:48:44","guid":{"rendered":"https:\/\/canny.io\/blog\/?p=11936"},"modified":"2026-08-26T15:16:09","modified_gmt":"2026-08-26T19:16:09","slug":"build-ai-feedback-capture-in-house","status":"publish","type":"post","link":"https:\/\/canny.io\/blog\/build-ai-feedback-capture-in-house\/","title":{"rendered":"What it actually takes to build AI feedback capture in-house"},"content":{"rendered":"\n

It sounds simple enough. AI reads your sales and support conversations, pulls out the feature requests, and files them neatly in your project tracker. Claude can do this with one prompt. How hard can it be?<\/p>\n\n\n\n

Let’s go through it together. We’ll keep a running tally of engineering hours and monthly costs as we go, just like we did when we built a feedback portal<\/a>.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Reading a conversation<\/h2>\n\n\n\n

Fable can legitimately do this in five minutes. Paste in a support ticket, get back the feature requests, bug reports, and pricing complaints inside it. It even quotes the customer.<\/p>\n\n\n\n

Nobody on the team is impressed. It’s 2026. Everyone here has run this exact prompt. That’s precisely the trap: the demo is so easy that the project looks done before it starts.<\/p>\n\n\n\n

Effort: 0 hours. Running total: 0 hours. Cost: $0\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Extraction<\/h2>\n\n\n\n

We wire the prompt to our support tool’s API with a small script and run it on last week’s conversations.<\/p>\n\n\n\n

Turns out most conversations aren’t feedback. Password resets. “Great support, thanks!” A customer venting about their day. The extractor dutifully files “customer seems frustrated” as a feature request.<\/p>\n\n\n\n

So we tune the prompt. Then we build a benchmark set, because “seems better” is not a metric. Then we tune again. Precision versus recall makes its first appearance. It will not be the last.<\/p>\n\n\n\n

The bill is real too. At our volume, about 2,500 support conversations and 400 sales calls a month, running a frontier model (like Fable) over everything lands around $3,000\/mo. That’s not a typo. Sales transcripts are long, and frontier tokens aren’t cheap. Your number will vary with your volume, but it won’t be small.<\/p>\n\n\n\n

Effort: 16 hours. Running total: 16 hours. Cost: $3,000\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Model costs<\/h2>\n\n\n\n

$3,000\/mo to read tickets will not survive a budget review.<\/p>\n\n\n\n

The fix is a cheaper model. The catch is that cheaper models miss things. So we benchmark recall and precision across models, tune prompts for each, and decide how many missed feature requests we can live with.<\/p>\n\n\n\n

One problem: benchmarks need labeled data. Hundreds of conversations, hand-labeled with the right answers. We don’t have that. Someone spends a week making it.<\/p>\n\n\n\n

The bill drops to around $200\/mo at our volume. The week is gone forever.<\/p>\n\n\n\n

Effort: 20 hours. Running total: 36 hours. Cost: $200\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

The ingestion pipeline<\/h2>\n\n\n\n

A month in, a PM asks about a feature request she saw in a ticket. The pipeline never saw it.<\/p>\n\n\n\n

Our little script, it turns out, has been quietly missing things. It dies on rate limits. It skips conversations that update after it runs. It double-processes others. Nobody noticed, because nobody notices missing data.<\/p>\n\n\n\n

So we build it properly. Webhooks, plus polling to catch what webhooks miss. A queue with retries. A database, so every conversation is stored, tracked, and processed exactly once. A backfill job for everything we missed.<\/p>\n\n\n\n

A pipeline is not a prompt. A pipeline is plumbing.<\/p>\n\n\n\n

Effort: 24 hours. Running total: 60 hours. Cost: $220\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Feedback types<\/h2>\n\n\n\n

We launched with feature requests. That lasted a month.<\/p>\n\n\n\n

Support wants bug reports. Success wants churn reasons. Finance wants pricing feedback. Each type is a new prompt, a new benchmark, a new round of precision tuning, and a new debate about what counts. Is “the export is slow” a bug or a feature request? Two teams, two answers, one field in the database.<\/p>\n\n\n\n

More types also means more AI passes per conversation. The bill ticks up.<\/p>\n\n\n\n

Effort: 20 hours. Running total: 80 hours. Cost: $250\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Do we already have this?<\/h2>\n\n\n\n

The pipeline keeps filing requests for features we already have.<\/p>\n\n\n\n

Customers ask for things that shipped last year. They just never found them. The extractor doesn’t know our product, so it can’t tell “doesn’t exist” from “exists, but hidden.” Fixing that means keeping a synced list of what we’ve built and running one more AI check on every idea. One more prompt, one more benchmark, one more way to be subtly wrong.<\/p>\n\n\n\n

Silver lining: requests for existing features are their own insight. That’s a discoverability problem, and someone should hear about it.<\/p>\n\n\n\n

Effort: 20 hours. Running total: 100 hours. Cost: $250\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Linear\/Jira integration<\/h2>\n\n\n\n

Time to put ideas where engineering lives. New ideas create issues in Linear, with the customer quote attached. Honestly, this part isn’t so bad.<\/p>\n\n\n\n

Then someone asks: when engineering ships an issue, can we find out? We’ll want that later. Two-way sync. Slightly worse.<\/p>\n\n\n\n

Effort: 12 hours. Running total: 112 hours. Cost: $250\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Deduplication<\/h2>\n\n\n\n

The pipeline runs. It works! Engineering asks us to please stop.<\/p>\n\n\n\n

Most ideas already exist in Linear. The tenth request for SSO should attach to the SSO issue, not create “SSO (10).” Text search catches exact matches. Semantic search catches most of the rest. Then an LLM gets each new idea plus the closest candidates and makes the call: dupe or not, and of which issue.<\/p>\n\n\n\n

On Jira? Worse news. Jira has no semantic search, so you’re building your own: embeddings for every issue, a vector index, kept in sync as issues change.<\/p>\n\n\n\n

Then the familiar grind. Benchmark, tune, decide how many false merges you can tolerate versus how many dupes slip through. This takes weeks, not days.<\/p>\n\n\n\n

Effort: 20 hours. Running total: 132 hours. Cost: $300\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Auto grouping<\/h2>\n\n\n\n

A thousand ideas in a flat list is not organization. It’s a landfill with search.<\/p>\n\n\n\n

Related ideas should cluster. Everything about exports in one group, everything about permissions in another. So the pipeline learns to group, which means deciding when two ideas are related but not duplicates. Somehow a harder question than when they’re the same. Another prompt, another benchmark, another precision target.<\/p>\n\n\n\n

Effort: 20 hours. Running total: 152 hours. Cost: $300\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Theme detection<\/h2>\n\n\n\n

Leadership doesn’t want a thousand ideas. They want five themes.<\/p>\n\n\n\n

“What are customers complaining about this quarter?” should have an answer. So we build theme detection<\/a>: recurring topics, tracked over time, with movement up and down. The themes have to stay stable month to month, or the trend chart is fiction. Stability is its own tuning problem, and yes, its own benchmark.<\/p>\n\n\n\n

Effort: 20 hours. Running total: 172 hours. Cost: $300\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Traceability<\/h2>\n\n\n\n

A PM asks: “Who actually asked for this?”<\/p>\n\n\n\n

Fair question. An idea without receipts is just a sentence. So every idea needs links back to its source conversations, the customer quotes, and the people who asked. When the tenth request attaches to the SSO issue, all ten requesters should be listed on it.<\/p>\n\n\n\n

Now the pipeline has memory, and the memory needs a schema.<\/p>\n\n\n\n

Effort: 12 hours. Running total: 184 hours. Cost: $300\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Undoing actions<\/h2>\n\n\n\n

The AI files an idea under the wrong group. It merges two requests that aren’t duplicates. It doesn\u2019t happen often, but it happens.<\/p>\n\n\n\n

The first time it happens, a PM asks two questions: why did it do that, and how do I fix it? So every automated action needs a trail, merges need to be reversible without losing the linked conversations, and a human’s correction has to survive the next pipeline run. Otherwise the AI quietly re-makes the same mistake tomorrow night.<\/p>\n\n\n\n

In Canny, undo is a button. In your pipeline, undo is a Slack message to whoever built it.<\/p>\n\n\n\n

Effort: 20 hours. Running total: 204 hours. Cost: $300\/mo.<\/p>\n\n\n\n

<\/p>\n\n\n\n

\n\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
<\/div>\n\n\n\"Form\n<\/picture>\n<\/div><\/div>\n\n\n\n
<\/div>\n\n\n\n

Connecting all the sources<\/h2>\n\n\n\n

Support was just the beginning.<\/p>\n\n\n\n

Sales wants their calls in there, so we integrate Gong. Except half the AEs use a different recorder. CS runs on a different tool entirely. Then someone asks about NPS surveys. Then app store reviews. Then G2.<\/p>\n\n\n\n

Every source is a new integration: new API, new auth, new data shape, new failure modes. Then the sales team switches tools next quarter, and we rebuild one from scratch. The pipeline never gets finished. It just gets more inputs.<\/p>\n\n\n\n

Every new source is also more volume, and the AI bill grows with it.<\/p>\n\n\n\n

Effort: 32 hours. Running total: 236 hours. Cost: $380\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

CRM integration<\/h2>\n\n\n\n

We now know what customers want. We don’t know what it’s worth.<\/p>\n\n\n\n

An idea plus a list of users is only so useful. It gets really useful when those users roll up to companies, the companies have revenue attached, and open deals link to the features blocking them. “27 people want SSO” is interesting. “SSO is blocking $840k in pipeline” gets a meeting.<\/p>\n\n\n\n

So we sync account and opportunity data from Salesforce<\/a> or HubSpot<\/a>. Field mapping, sync jobs, rate limits, retries. Someone owns that sync now. Forever. They know who they are.<\/p>\n\n\n\n

Effort: 24 hours. Running total: 260 hours. Cost: $380\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Deal filters<\/h2>\n\n\n\n

Sales lives in segments. “Show me feedback from deals we lost last quarter.” “Just open opportunities.” “Only deals closing this quarter.” “New business only, not renewals.”<\/p>\n\n\n\n

Each filter is easy. Supporting all of them means the CRM sync has to carry deal stages, close dates, amounts, and deal types, and keep them all fresh. A filter on stale data is a lie with a dropdown.<\/p>\n\n\n\n

Effort: 8 hours. Running total: 268 hours. Cost: $380\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

User segmentation<\/h2>\n\n\n\n

Same story on the customer side. “Only companies paying over $50k.” “Only the enterprise plan.” “Only accounts using the API product.”<\/p>\n\n\n\n

So we sync company traits from billing and our app, then build the filtering UI. Then we keep the traits fresh, because stale data is worse than none.<\/p>\n\n\n\n

Effort: 8 hours. Running total: 276 hours. Cost: $380\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Reporting<\/h2>\n\n\n\n

Leadership has questions. Which features will help us close deals? Which deals are blocked right now, and on what? We shipped three of those features last quarter. Did the deals actually close?<\/p>\n\n\n\n

That last one is the killer. Answering it means joining feedback, issues, and CRM history across time. We build dashboards. Every exec wants a slightly different slice, and every slice is a ticket.<\/p>\n\n\n\n

Effort: 24 hours. Running total: 300 hours. Cost: $380\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

MCP access<\/h2>\n\n\n\n

The team is living in Claude now. They don’t want another dashboard. They want to ask: “What are enterprise customers complaining about this month?” “Do we already have this feature?”<\/p>\n\n\n\n

So we build an MCP server<\/a> over the data: the ideas, the themes, the customer quotes behind them. Tool definitions, authentication, updates every time the schema changes. It’s genuinely useful. It’s also a second interface to maintain, and it has to respect every permission rule the first one has.<\/p>\n\n\n\n

We take a moment. We built an AI pipeline to avoid buying an AI tool, and now we’re building AI tooling so our AI can read our AI pipeline.<\/p>\n\n\n\n

Effort: 8 hours. Running total: 308 hours. Cost: $380\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Team notifications<\/h2>\n\n\n\n

A feature ships. Three closed-lost deals wanted it. Nobody tells the reps.<\/p>\n\n\n\n

So we build notifications. When an issue ships, the AEs and CSMs whose accounts asked for it get a ping: “Acme wanted this. It’s live. Go win them back.” It’s one of the most valuable things in the whole system. It’s also a join across feedback, issues, accounts, and deal owners that has to stay accurate forever.<\/p>\n\n\n\n

Effort: 8 hours. Running total: 316 hours. Cost: $380\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Close the loop<\/h2>\n\n\n\n

Then the bigger version: email every customer who asked for a feature when it ships.<\/p>\n\n\n\n

Which means we’re sending email now. A provider, a send queue, unsubscribe links, a preferences page. Bounce handling, because sending to dead addresses tanks deliverability, and once our sender reputation slips, every email we send lands in spam. Sanitizing anything user-generated that ends up in an email. Email is an entire discipline, and now it’s ours, bolted onto the side of a data pipeline.<\/p>\n\n\n\n

Effort: 16 hours. Running total: 332 hours. Cost: $400\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Privacy<\/h2>\n\n\n\n

It hits us one night. We built a database that stores copies of every customer conversation. Support tickets, sales calls, all of it. Names, emails, the occasional pasted API key.<\/p>\n\n\n\n

That needs access controls, a retention policy, and deletion that actually works. When a customer deletes their data in the support tool, our copy has to go too. GDPR doesn’t care that our copy lives in a side project.<\/p>\n\n\n\n

Effort: 12 hours. Running total: 344 hours. Cost: $400\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

SOC 2<\/h2>\n\n\n\n

A prospect’s security team sends a questionnaire. One question: “Do any internal tools process customer conversation data?”<\/p>\n\n\n\n

Yes. Yes they do. The pipeline is now in scope for our SOC 2 audit<\/a>. Access reviews, evidence collection, a pen test that covers one more application, vulnerability scanning, a slightly pricier audit. Call it another $300\/mo once the annual bills are amortized.<\/p>\n\n\n\n

Our compliance lead just learned the pipeline exists. They have follow-up questions.<\/p>\n\n\n\n

Effort: 12 hours. Running total: 356 hours. Cost: $700\/mo.<\/p>\n\n\n\n

<\/div>\n\n\n\n

And this is just the start<\/h2>\n\n\n\n

One morning, an email arrives: the model we tuned everything on is being deprecated in 90 days.<\/p>\n\n\n\n

Every prompt, every benchmark, every precision number we fought for, re-validated against its replacement. This will happen again, roughly once a year. It’s nobody’s fault. It’s just weather now, and we live in it.<\/p>\n\n\n\n

That’s one storm. There’s also the customer who writes in German. The exec who wants Zoom calls in the pipeline. The quarter where support volume triples, and so does the bill.<\/p>\n\n\n\n

A pipeline isn’t a project with an end date. It’s a subscription you pay in engineers.<\/p>\n\n\n\n

<\/div>\n\n\n\n

The real cost<\/h2>\n\n\n\n

Notice what the tally doesn’t include: an owner.<\/p>\n\n\n\n

Every pipeline needs one. Someone who re-runs the benchmarks, fixes the Gong sync, answers “why did it miss this ticket.” That person is one of your best engineers, because accuracy work is hard. Now a chunk of their year belongs to an internal tool.<\/p>\n\n\n\n

The worse version is having no owner. The pipeline drifts, the numbers quietly rot, and the roadmap meeting runs on data nobody can vouch for. A wrong number in that meeting costs more than every line item in this post. You’ll just never see the invoice.<\/p>\n\n\n\n

You wanted this automated because the decisions riding on feedback are big ones. That logic cuts both ways. The bigger the decisions, the less you want them running on a side project.<\/p>\n\n\n\n

Ours isn’t a side project. It’s the whole company.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Was it worth it?<\/h2>\n\n\n\n

Read the tally back. 356 hours. Twenty-seven grand of engineering time, $700 a month in bills, a quarter and a half on the calendar. For a pipeline that does, roughly, what the five-minute demo appeared to do.<\/p>\n\n\n\n

Most internal tools degrade gracefully. Half-finished, they’re still half-useful. This one isn’t. If the pipeline misses a third of the feedback, or invents requests nobody made, your team notices. They stop trusting it, and they quietly go back to gut feel. Below a certain accuracy bar, the whole thing is a decoration.<\/p>\n\n\n\n

Getting above that bar is the real work. It’s why a third of the tally is pure benchmarking. It’s also where a decade matters. To know whether a model is good enough, you need datasets to benchmark against. We spent a long time building ours, and we re-run those benchmarks every time a model changes, a prompt changes, or a new feedback type ships. That’s how Autopilot gets high accuracy without flagship costs. You’d be starting that work from zero, at flagship prices.<\/p>\n\n\n\n

So sure, spend the quarter. Or spend an afternoon connecting your tools to Autopilot<\/a>, and spend the quarter on the product your customers are giving you all this feedback about.<\/p>\n\n\n\n

They’re already telling you what to build. Don’t make hearing them a side project.<\/p>\n\n\n\n

<\/p>\n\n\n\n

\n\n
\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n
<\/div>\n\n\n\"Get\n<\/picture>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"

AI can pull feedback from a support ticket in five minutes. Here’s the 356 hours of pipeline, benchmarking, and compliance work that makes it real.<\/p>\n","protected":false},"author":3,"featured_media":11940,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[6,7],"tags":[15],"class_list":["post-11936","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-founder-stories","category-product-management","tag-product-management"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t