Everyone running a support team right now is looking at AI and asking the same question: where does this fit? We’re no different. At Paid Memberships Pro we handle about 370 tickets a month, and we’ve been experimenting with an AI agent that adds internal notes to incoming tickets before a human picks them up.
I’m not here to tell you this is the way. I want to walk through how we’re doing it, and more importantly, how we improved it — because that’s the part I think is actually interesting.
There are a lot of projects floating around that promise AI agents that improve themselves automatically. I think something like that is probably coming. But here’s what’s working for us right now: a human-in-the-loop feedback cycle where the team tells the AI what it got wrong, and we fix it together.
The Setup
We built a script that runs on cron, checks for new tickets, and posts a private moderator-only note. Not a reply to the customer — an internal note the support agent sees when they open the ticket. It includes a summary, the most likely cause, relevant doc links, suggested tags from our SOP, and if the ticket mentions code or a specific plugin, it searches our codebase and documentation database.
We chose notes over direct replies deliberately. Support is a relationship. The agent still owns the conversation, reads the ticket, understands the context, writes the reply in their own voice. They just get a head start because someone already did the research. Human to AI to Human.
The Real Work: Feedback
We ran the script on a batch of tickets and asked the team what they thought. This is where it got good.
Mary noticed the AI misidentified which payment gateway a customer was using. The customer said “new PayPal Gateway” and the AI assumed PayPal Express. Close, but wrong in a way that matters. She also caught it debugging code for a plugin the customer wasn’t even running — the site health data showed which plugins were active, but the AI wasn’t checking that.
Sam flagged a refund ticket where the note was way too long. Code findings about how our refund system works internally. Nobody needs that for a straightforward refund request. He also caught the AI claiming the customer had other open tickets when they didn’t. Hallucination. The kind of thing that erodes trust fast.
David noticed it couldn’t read a GitHub Gist a customer had shared in their ticket. It also suggested asking for information that was already available in the site health data.
Every piece of feedback was specific. Every piece pointed to a fix.
Turning Feedback Into Fixes
This is where working with an AI coding agent (we use Claude Code) made the iteration fast.
I sat down with Flint — that’s our AI agent — and walked through each piece of feedback. Some fixes were prompt adjustments: tell the AI to quote the customer’s exact words instead of inferring what product they’re using. Check which plugins are actually active before diving into code investigation. Keep notes short for account tickets like refunds. Never claim a customer has other tickets unless you can actually see them in the data.
Some were code changes. We added URL fetching so the AI can read linked gists and code snippets customers share. We pulled in the customer’s ticket history so it knows about prior conversations. We loaded our approved tag list from the support handbook directly into the prompt, because the AI was inventing tags that sounded reasonable but didn’t match our SOP.
Testing Without Cheating
Here’s a problem we ran into: after the team had already replied to the feedback tickets, re-running the AI against those tickets didn’t tell us much. It would read Mary’s reply and say “Mary’s reply was correct.” No kidding.
So we added a test mode that strips out all moderator replies, all prior AI notes, everything except the original post and any follow-up comments from the ticket author. Now we can re-run the AI against any historical ticket and see what it would have said if it got there first. Clean comparison, no hindsight contamination.
We re-ran all eight feedback tickets with the changes. The refund ticket went from a D to an A. The gist ticket went from a C to an A-. The PayPal gateway issue improved. The donation ticket correctly said “I don’t know what recipe they used, can you share it?” instead of guessing.
Not everything was perfect. But the trajectory is clear.
The Timing Problem
We also did a timing analysis. Pulled four weeks of ticket data — when tickets come in, when the first moderator reply lands, where the gaps are.
Our team is mostly in African timezones, working roughly 4am to 2pm Eastern. Two dead zones stood out: overnight after the team signs off, and late US afternoon when tickets are still coming in but nobody’s picking them up until the next morning. Weekend tickets were waiting 40 hours on average.
We set up three cron runs targeting those windows. The goal is for the note to already be there when the agent opens the ticket in the morning. Not adding noise while they’re actively working.
What’s Actually Working
The self-improving AI agent is a compelling idea, and I think we’ll get there. But right now, the thing that’s actually making our AI support notes better is a pretty old-fashioned loop: ship it, get feedback from the people who use it, fix the specific things they flag, test the fixes, ship again.
The AI part speeds up the research and the coding. But the judgment about what’s good and what isn’t still comes from Mary and Sam and David actually using the notes on real tickets. That feedback is the most valuable input in the whole system.
If you’re building AI into your support workflow, I’d focus less on making the AI autonomous and more on making the feedback loop tight. The team knows what’s helpful. Let them tell you.