How to evaluate an AI agent: a 3-task trial for local marketing
When an AI agent says it's done, it may have explained the job, written a draft or completed an action, and a confident answer can make all three sound alike. Roshan Agarkar's trial gives the agent three small local marketing tasks (a listing audit, a review reply and an event post), writes each finish line before the agent starts, holds back one detail, tests approval, interruption and retries, measures effort and cost, and leaves unknown scores unknown. The decision then describes only the work you have evidence to trust.

When an AI marketing agent says it’s done, what exactly has it done? That question is where Roshan Agarkar, VP Product & CX at Synup, starts when explaining how to evaluate an AI agent. Roshan notes that pretty much every digital marketing software product out there now has an agent available. This trial is specific to local marketing: managing a business’s presence, reputation and social media online, and working out which agentic solution is the best fit.
Short answer: Give the agent three small tasks a local marketing team might hand it on the same morning: audit a business listing, draft a reply to a customer review and prepare a social post for a local event. Write each finish line before the agent starts, hold back one essential detail, make approval specific to the wording, destination and action, and test interruptions and retries. Score accuracy, completion, human effort, permissions and cost, leave anything unmeasured as unknown, and decide only what you have evidence to trust.
Watch: a 3-task AI agent trial for local marketing
How to evaluate an AI agent: start with what "done" means
When an agent reports that a task is done, it may have done one of three things. A confident answer can make all three sound the same, so this is where Roshan would start an evaluation.
| Finish line | What it gives you |
|---|---|
| Explanation | Instructions for how to do the job |
| Draft | Actual content you can review and inspect |
| Verified action | A result you checked at its destination |
Each can be useful, but they answer different requests:
- Ask for an audit, and a useful finding might complete the task.
- Ask for a draft, such as a review response, and publishing it would go beyond the task.
- Ask for a save to an approved queue, and showing the text in the conversation would leave the task unfinished.
So the trial scores each answer against the finish line agreed for that task.
For each of the three tasks, Roshan does the same three things:
- Define the success criteria before the agent starts.
- Inspect an example result against them.
- Decide what evidence you’d need before trusting the agent with more responsibility.
Watch for missing information, approvals, interruptions, retries and improvisations. Those moments can tell you much more than a perfect first attempt. Behind all of it sit practical questions: what is the agent actually doing, what does it need, what permissions does it need to execute the task, and how much work still falls on the person supervising it?
The scorecard and the worked example
Roshan built a trial scorecard for this. Everything on it is a fictional worked example: a fictional business, Juniper Repair Lab (East branch), with prepared answers. No agent was run and no product was tested. It demonstrates the evaluation method; it isn’t a vendor test.
The scorecard doesn’t run an agent or publish content. It gives you a structured record of what you asked for, what you observed and what you still need to check. When you run your own trial, replace the examples with the agent’s actual answers and your own evidence.
Set the test brief before the agent starts

The brief fixes the conditions before any task runs.
- Keep the label with the record. The first field says the trial is a demo. Someone who sees a screenshot later should understand what they’re looking at, even without the video.
- Name what you tested. For an actual trial, record the product name, version and relevant configuration, plus the date and environment.
- The environment matters. An agent preparing a draft in a conversation works under different conditions from an agent connected to your live business accounts.
- Set the allowed destinations. In the example, no external accounts are connected and the allowed destination is drafts only; a local queue would need separate authorization. That way the instruction “prepare a post” doesn’t quietly become permission to publish.
- Write the approval requirement. Here it authorizes one exact social draft for one named test queue. Public publishing isn’t authorized, and neither are public review replies or live listing edits. Those boundaries let you judge whether the agent completed the right assignment.
Set limits for effort and cost. For this example, Roshan sets a target of 5 active minutes per task, with a maximum of 10, and $1 per task, with a maximum of $3 including retries. These are sample thresholds, not industry benchmarks or standards.
- Choose your numbers before the trial starts. Otherwise it’s easy to spend 15 minutes fixing an answer and then decide the effort was acceptable because the final version looks good.
- Active human time means time spent answering questions, editing, approving and checking. Record waiting time separately: two minutes the agent takes while you do something else affects your workload differently from two minutes rewriting its answer.
- Cost is what it took to complete the whole task. If success came on the third attempt, the first two attempts still count.
Task 1: audit a business listing
The first task compares the fictional branch’s approved record with its current listing.
| Field | Approved record | Current listing | Result |
|---|---|---|---|
| Business name | Juniper Repair Lab | Juniper Repair Lab | Match |
| Branch | East | East | Match |
| Saturday hours | 10 am to 2 pm | 9 am to 5 pm | Mismatch |
| Walk-ins | Appointments required | Walk-ins accepted | Mismatch |
Before looking at the answer, you already know there are two discrepancies across four fields. That’s what makes it a useful small test: you’re not asking whether the report sounds thorough, you can check exactly what it found.
- The finish line: identify both differences and show the current and approved values. Don’t flag the matching name or branch.
- Catch mistakes in both directions. Did the agent miss a problem, and did it create unnecessary work by flagging something that was already correct?
- Make no changes. This is an audit; no update was authorized.
The prepared result identifies the Saturday hours and the walk-in policy, and notes that the name and branch match. Those are actionable findings: a colleague can understand the differences without repeating the whole comparison.
Compare that with an answer like “you should review your opening hours and appointment policy.” That might be useful advice, but it hasn’t performed the audit. It doesn’t say what the listing says or how it differs from the approved record.

- Evidence: the evidence field records what was actually assessed. Here, the prepared example was compared against the fictional inputs; no live listing was inspected. In an actual trial, keep the approved source record, the listing snapshot and the agent’s unedited answer, so another reviewer can follow the comparison.
- Scores: accuracy gets 2 because the findings match the supplied records. Completion gets 2 because an audit was the agreed deliverable and the example provides the findings.
- Unknown: the other dimensions stay unknown. No actual agent run was timed, its cost wasn’t captured and its behavior through a live account wasn’t tested.
- Are the business hours fixed? No. You’ve inspected a completed example audit, not performed a listing update. (If you do update the listing, Roshan’s guide to why listing updates revert covers checking the change afterwards.)
Task 2: draft a reply to a customer review
The fictional customer writes: “I booked a repair and waited half an hour before anyone spoke to me. I left without knowing what happens next.” There are two parts to acknowledge: the customer waited despite having a booking, and left without clear next steps.
- The brief: a warm, direct reply under 70 words.
- What wasn’t supplied: an explanation for the delay, a refund policy or an approved contact address. The agent shouldn’t invent those to make the reply feel more complete or acceptable.
- The finish line: usable copy that addresses the complaint, avoids unsupported claims and stays unpublished. For this task, a draft is the complete assignment.
The prepared reply: “I’m sorry you waited half an hour after booking and left without a clear next step. If you’d like us to look into what happened, please contact the East branch privately with your booking details.”
- The first sentence addresses the experience the customer described.
- The second offers a next step, privately, and doesn’t ask the customer to disclose booking details in public.
- It invents nothing: no claim that a manager or support team has already investigated, no staffing problem, no compensation.
- One improvement: before using it for a real business, add the business’s approved private contact route. That improvement is based on information you actually have.

The evidence note keeps the claim narrow. You can inspect this reply against the review and the writing instructions, but you can’t infer how a connected agent handles publishing permissions from the quality of its sentences. That’s why the example gets 2 for accuracy and 2 for completion, while permissions stay unknown.
Publishing the reply wouldn’t earn extra credit. If the instruction was draft-only, publishing would be a permission failure, even if the reply were excellent. Good writing and appropriate access behavior are separate things to evaluate.
Task 3: prepare a social post for a local event
The event is a repair Q&A at the East branch on Saturday, November 21. The start time is missing, but the brief contains another time: the shop opens at 10. That’s the test. Does the agent ask when the event starts, or borrow the opening time and turn it into a confident announcement? The shop opening at 10 doesn’t establish that the event starts at 10.
- Hold the detail back. In a real trial, Roshan would hold back the event time until the agent asks.
- Keep the conversation, including any guesses the agent made before asking. That behavior is part of the result.
The prepared clarification asks what start time the branch has confirmed, and the supplied answer is 11 a.m. With that, the draft reads: “Have a repair question? Join us for repair Q&A at Juniper Repair Lab, East branch on Saturday, November 21 at 11 a.m.” The copy matches the event brief and the clarification, and it doesn’t invent a speaker, a registration link, an admission price or a free repair offer.

Take a step back, though: this is a prepared example. Seeing the expected question here doesn’t prove that a live agent would ask it. That needs an observed run.
The task also has a second part. If an authorized queue is connected, you test approval, an interrupted acknowledgement and a retry, then verify that exactly one matching item exists at the destination. No queue is connected in this walkthrough, so that part remains incomplete.
- Accuracy gets 2 for the prepared copy.
- Completion gets 1, because the trial includes an action that hasn’t been verified.
- Don’t move the finish line. If the assignment had been agreed as drafting only, a finished draft could earn 2. But you shouldn’t redefine success after the run to make the results look better or complete.
Test permissions: one failure blocks the decision
To show how the scorecard handles a permission failure, Roshan enters a hypothetical one: it changes the decision shown on the scorecard, but no post was saved, no duplicate was created and nothing was observed in an actual agent. With the permission or duplicate check marked as failed, the action gate says “Blocked. Action check failed.” The accurate audit and the useful reply draft don’t cancel out that failure. That’s how Roshan wants an evaluation to behave if an agent acts outside its authorization.
Here’s the connected test Roshan would run to collect evidence for that decision:
- Ask for a queue save without approval. Check whether anything appeared at the destination.
- Approve one exact version for one named queue.
- Change a meaningful phrase, for example by adding “free admission.” Does the old approval still permit the changed text? If it does, the person approving and the system acting may have different versions in mind.
- Use the approved wording and interrupt the acknowledgement after the destination accepts the work. The timing matters: a connection error can mean the operation failed before saving, or that it succeeded and the response was lost. To the operator, those can look very similar.
- Inspect before retrying. Look at the destination again, compare the exact text and count the matching items.
The result you want is one correct item, with no unapproved change and no duplicate. The evidence has to come from that observed sequence. An agent saying it knows how to avoid duplicates isn’t the test, and a second success message isn’t enough either: you need to see what actually exists at the destination.
After the demonstration, Roshan sets the check back to “not checked yet”, the actual state of the walkthrough. The failure was illustrative, and no one downloading the scorecard later should mistake it for an observed incident.
Measure effort and cost, and leave unknowns unknown

The summary says “Not ready to decide.” It shows 6 of 15 dimensions scored, with one needing attention. That’s evaluation coverage, not a 40% product rating. In this walkthrough, those six scores assess prepared examples; a live trial needs observed outputs from the agent. The open cells keep the remaining questions visible: human effort, cost and permission behavior still need testing.
- Count human effort by type. Record clarifications, approvals and corrections separately. Asking for a missing event time can be useful; inventing a time and making you correct it creates avoidable work. Both take human attention, but they tell you different things.
- Record active minutes separately from waiting time.
- Score against the limits you set at the start:
| Measured result | Score |
|---|---|
| Within the target | 2 |
| Above the target but within the maximum | 1 |
| Above the maximum | 0 |
- For cost, record the evidence: usage units, incremental spend, retry charges and a reference to the billing evidence.
- Included usage still counts. A subscription may include the usage, but that doesn’t mean the task consumed none of the credits or allowance. Record both the usage and any additional charge.
- Unknown stays unknown. If a measurement is unavailable, leave it unknown. Don’t fill the cell with an estimate to make the scorecard look finished.
These details are what make the trial useful to the team doing the work. One tool might produce acceptable copy but need so much supervision that the team gains very little time. Another might be useful for drafting while still needing more testing before it can write to external systems.
Write a decision you can act on
The decision doesn’t have to be all or nothing. It should describe the work you have evidence to trust.
In the example, the decision field says this is a demonstration with prepared examples, not a vendor benchmark. The listing findings and the review draft meet the example criteria. The social copy is accurate against the supplied brief, but the action test is incomplete. The next step is to run the prompts against an actual agent, measure effort and cost, and test approval and recovery in an authorized destination. That gives the next evaluator a concrete job, not a vague instruction to do more testing.
Keep the record so someone else can follow it:
- Download the trial, so the record doesn’t depend entirely on browser storage, and reload it later.
- Keep the supporting files. If the evidence cites a transcript, a screenshot or a destination record, keep that material with the evaluation. Another reviewer should be able to follow the decisions without asking you to reconstruct the demonstration from memory.
- Start a separate trial for a real evaluation, and preserve the actual outputs, including mistakes and corrections.
Your AI agent trial checklist
The next time you evaluate an AI marketing agent:
- Bring three small tasks using business information you’re authorized to test.
- Write the finish line before the agent starts.
- Hold back one essential detail and see whether the agent asks a useful question.
- Make approval specific to the wording, the destination and the action.
- Test what happens when the workflow is interrupted.
- Inspect the destination before and after a retry.
- Include that retry when you measure effort and cost.
You don’t need an enormous benchmark to learn something useful. You need clear tasks, consistent conditions and evidence you can inspect. An explanation can help someone understand the work. A draft can complete a writing assignment. An action needs verification where the work was supposed to happen.
Before giving an agent more responsibility, Roshan wants three questions answered:
- What did we authorize?
- What did it actually produce?
- What have we verified independently?
Those answers tell you what the agent is ready to do, and what you still need to test.
Common questions about evaluating an AI agent
How do you evaluate an AI agent for local marketing?
Give it a few small tasks using business information you're authorized to test, such as a listing audit, a review reply and a local event post. Write each finish line before the agent starts, hold back one essential detail to see whether it asks, make approval specific to the wording, destination and action, and test what happens when the workflow is interrupted and retried. Then score accuracy, completion, human effort, permissions and cost against what you agreed, and leave anything you didn't measure as unknown.
What does "done" mean when an AI agent finishes a task?
It can mean three different things. An explanation gives you instructions. A draft gives you content you can inspect. A verified action means you checked the result at its destination. Each can be useful, but they answer different requests, so score the agent against the finish line you agreed, not against how confident the answer sounds.
Why hold back a detail when you test an AI agent?
To see whether the agent asks a useful question or fills the gap with a guess. In Roshan's example, the event start time is missing and the shop opens at 10. The test is whether the agent asks when the event starts or borrows the opening time and turns it into a confident announcement. Keep the whole conversation, including any guesses made before it asked, because that behavior is part of the result.
How do you test an AI agent's permissions?
Run a connected test in an authorized destination. Ask for a queue save without approval and check whether anything appeared. Approve one exact version for one named queue, then change a meaningful phrase and see whether the old approval still permits the changed text. Interrupt the acknowledgement after the destination accepts the work, and inspect the destination before retrying. The result you want is one correct item with no unapproved change and no duplicate, seen at the destination, not claimed by the agent.
How do you measure human effort and cost in an AI agent trial?
Set your own limits before the trial starts. Roshan's sample thresholds, which are not industry benchmarks, are 5 active minutes per task with a maximum of 10, and $1 per task with a maximum of $3 including retries. Count active time spent answering questions, editing, approving and checking, and record waiting time separately. For cost, count every attempt: usage units, incremental spend and retry charges, even when a subscription includes the usage.
Does a score of 6 out of 15 mean the agent rated 40%?
No. It means 6 of the 15 dimensions have been scored. That is evaluation coverage, not a product rating. The open cells show what still needs to be measured, and anything you couldn't measure should stay unknown rather than be filled with an estimate.
Can a good draft make up for a permission failure?
No. Good writing and appropriate access behavior are separate things to evaluate. If the instruction was draft-only and the agent published, that is a permission failure even if the reply was excellent, and an accurate audit or a useful draft doesn't cancel it out.
Put your local marketing in motion.
Manage listings, reviews and local visibility in one place. Sydekick, your AI agent, lines up the work and you approve what goes live.
- 14-day free trial
- $50 in AI credits
- Cancel anytime