How to Evaluate an AI Agent: 8-Question Checklist (2026)

This AI agent evaluation checklist is eight questions a buyer asks on the vendor call, each with a pass test. Reach: the agent makes a change you can verify in the target tool. Failure: errors surface in the channel, with no silent retries on data changes. Oversight: a named person approves irreversible actions. Evidence: an action log your admin can open. Cost: a worst-case monthly bill, a cap and a clear unit. Adoption: non-admins use it on day one. Learning: a corrected mistake stays corrected. Security hand-off: compliance documents and the OAuth scope list, ready on the call.

It’s written for a budget holder with no machine learning background and one hour with a salesperson.

Why this AI agent evaluation checklist is for buyers, not builders

Builders and buyers test the same agent with different tools. A builder runs a test set through three kinds of evaluator:

• Code-based, or deterministic, graders check what a script can verify: the right tool was called, the record exists, the output parses.
• LLM-as-a-judge graders score open-ended output against a rubric, such as whether a summary is faithful to its source.
• Human reviewers are the reference the other two are calibrated against, and the slowest.

A builder also picks a level. Component-level or trajectory evaluation scores each step the agent took: which tool it chose, what it passed in, whether a retry was justified. End-to-end evaluation scores only the finished outcome. Builders need both, because a right answer reached through a wrong step breaks on the next run.

A buyer can’t change the system, so the buyer’s evaluator is human review of end-to-end outcomes on their own workflows. The graders are the vendor’s job: ask which ones run before each release and ask to see one recent result. You can still spot-check a trajectory yourself: open the action log for one finished run and read the steps in order. A right result reached through a strange detour is a warning.

Then turn the builder metrics into rates you can count across your pilot runs:

• Task completion rate: runs where the change landed in the target tool, divided by runs attempted.
• Tool selection accuracy: runs where the action log shows the right tool, with no detour through the wrong one.
• Faithfulness: report numbers that trace to their source, out of numbers checked.
• Cost: what a typical request consumes in the billed unit.
• Latency: how long a multi-step request takes to finish.

Agents act. A wrong action sends the email, pauses the campaign or changes the record, and someone has to undo it.

Nothing below needs an eval harness, benchmark or scoring script. A vendor who offers a methodology walk-through instead of an answer is dodging; ask again in plain words.

Ask these 8 questions on the vendor call

Run the list in order. The first four establish whether the product can do real work safely; the last four, whether your team will still use it in six months. Write the answers down as given: vague ones sound fine in the moment and read badly a week later.

Question 1: What can it actually touch in my systems?

This is the cross-tool test, and every other question depends on it. An agent that can only read your tools can describe your work; one that can write to them can do it. Vendors call both “integration”.

Ask for a single request that crosses two systems, neither of which is the vendor’s own product. Something like: pull last month’s paid signups from Stripe, check which ones have no record in HubSpot, and post the list in this thread. Then watch what happens. If the answer is a finished list, the product acts where the work lives. If the answer is a project plan, a connector your team has to build, or a CSV export you then process yourself, you are buying a platform, and you should budget the build as well as the licence.

A pass looks like named systems, named actions, and a change you can verify in the target tool afterward: the deal stage moved, the ticket exists, the budget shifted. A fail looks like a summary in the chat window and an instruction for you to go do the thing.

See which tools an AI employee can act in →

Question 2: What happens when it is wrong?

Any agent is wrong sometimes. The model misreads the request, the data is stale, the integration times out halfway through a multi-step action. The product’s quality is decided by what happens next.

Ask for the three failure modes in plain terms. Wrong output: it produces a report with a bad number. Wrong action: it updates the wrong record or sends the wrong email. Looping: it retries a failed step over and over and burns credits or sends duplicates. For each one, ask how the product tells you, where it tells you, and whether it retries destructive actions on its own.

Then ask the vendor to name their own limits. “It does not do that, and here is what it does instead” means the product was tested against reality; “it can do that too” to every limit means it wasn’t. The post on what AI agents cannot do lists the categories where a straight answer is a good sign.

A pass is a concrete description of failure surfaced in the channel where the request was made, with enough detail to debug, and no silent retries on anything that changes data. A fail is “it is very accurate”, a percentage with no denominator, or a generic error message in the demo.

Question 3: Who approves an irreversible action?

Some actions cannot be undone: a customer email, a payment, a production deploy, a deleted record. The question is whether a named person sees them before they happen.

Ask the vendor to show a customer-facing email going out. The pass is a draft that waits for a named approver to click something, with the ability to edit or reject it. The fail is “it just sends”, or an approval setting that is global and off by default, or an answer that the product is autonomous and that is the point. Autonomy is the right setting for a Monday report and the wrong setting for anything that leaves the company.

Ask a second question underneath this one: can an admin decide which actions need approval, per tool, rather than per vendor policy? A product that lets you put CRM writes behind approval while leaving CRM reads free was designed by people who have run it inside a real company. The longer argument for review-first defaults is in why an agent should not act without asking.

Question 4: What evidence exists after the fact?

If a campaign gets paused on a Tuesday and nobody remembers doing it, you need to know whether the agent did it, who approved it and what it was told. Ask the vendor what a reviewer sees after an action has run.

The pass is an inspectable record for each action: what was done, in which tool, from what input, who approved it, and when. Ask to see the log for the last ten actions in the demo workspace, not a slide about logging. If the product writes to an audit log that your admin can open without a support ticket, the infrastructure exists. If the answer is “we keep logs” with no screen behind it, assume the log is for the vendor’s engineers and not for you.

Evidence also covers the output. A report that shows where each number came from is reviewable in minutes. One with no provenance has to be rebuilt to be trusted, which is the work you were trying to hand over. There is a practical routine for this in how to verify your AI employee’s work.

Question 5: What does the real cost look like at month six?

List price answers the wrong question. The right question is what the bill looks like once the product is doing real work for the whole team, because that number decides whether the contract renews.

Ask how pricing behaves as adoption grows. Per-seat pricing stays flat while usage climbs, so idle seats cost the same as busy ones. Usage-based pricing rises with the work, so your finance lead needs a cap. Hybrid models charge a platform fee plus usage. None of these is wrong. A vendor who cannot describe the worst-case monthly bill for your team size and your three heaviest workflows is.

Ask four follow-ups. What is the unit you are billed in, and what does one typical request consume? How long does a typical multi-step request take to finish? Is there a spend cap, and who sets it? What happens to unused allowance at the end of the month? Then do your own payback math on the workflows you actually intend to hand over, rather than the ones in the demo. The method is in how to tell if your AI employee is paying off, and it works for any vendor.

A pass is a worst-case number, a cap, and a clear unit. A fail is a quote that depends on a conversation with sales after you have signed.

Question 6: Who on my team can use it without IT?

Adoption decides the value, and friction decides adoption. A product that needs an admin rollout, a training session and a new tab gets used by the people who sat in the training, and not by the rest.

Ask where the product lives and who can start using it on day one. Does it work inside Slack or Microsoft Teams, where the requests already happen, or does it need its own app? Can an operations manager connect a tool and hand over a task, or does every connection go through IT? Can the whole team share one instance, or does each person set up their own?

The pass is usable by non-admins, in the place the team already talks, with admin controls layered on top rather than placed in front. The fail is a workflow canvas, a prompt library, and a rollout project. Those are fine products for a different buyer. Ask to see the first five minutes for a new user, and count how many of those minutes are setup.

See the jobs teams hand over →

Question 7: What did it learn, and does that carry over?

The difference between a tool and a hire is that a hire gets better. Ask whether the product retains anything between sessions, or starts blank each time.

The test: correct it once, then ask for the same work a week later. If the second run starts from the corrected version, the product compounds. If it repeats the mistake, it is an intern on their first day, every day. Ask what is stored, where it is stored, who can see it, and whether it can be exported or deleted when you leave.

Ask also whether that knowledge is shared across the team or trapped in one person’s chat history. The pass is retained knowledge, scoped to the workspace, that you can inspect and correct. The fail is “it remembers context”, with no demo of a second run.

Question 8: What do I check before I connect it?

This question is the hand-off. The first seven decide whether to buy at all. Once that answer is yes, a different review starts: what the product is allowed to access, which scopes it asks for, how credentials are stored, what the vendor’s audit status is, and what your admin can revoke in one click. That review belongs to your security owner, and it is a separate list, in what to check before you connect an AI tool.

On the sales call, ask only for the inputs that review will need: the vendor’s compliance documentation, the list of OAuth scopes the product requests, and a named contact for the security questionnaire. Also ask how the vendor tests for prompt injection, PII leaks and other adversarial inputs, and ask to see one test result. A vendor who has all of that ready has been through procurement before. A vendor who asks what SOC 2 is has not.

Read the security posture →

How one vendor answers these: Viktor

Viktor is an AI employee, not a scoring key. His answers to the eight questions are below so you can see what a complete set looks like and hold other vendors to the same shape.

What he can touch: the product page says he “lives in Slack, connects to 9,800+ tools, and actually does the work”, and the integrations page lists 27 native integrations plus 9,800+ tools through managed connectors, most of them one-click OAuth and some through API keys.

What happens when he is wrong: the integrations page states that “Sensitive actions wait for your approval. Everything is logged.” A wrong draft is rejected on the approval card and never runs.

Who approves: per the security page, “Money moves, code pushes, and customer emails wait for your explicit approval in Slack.” The approvals documentation describes how he prepares an action as a draft, posts a card with Approve and Reject buttons, and runs it only after someone clicks Approve. Admins can require approval for individual integration tools, and tools set to forbidden never run.

What evidence exists: each approval card records who approved or rejected the action, and on Enterprise workspaces admins get a customer audit log of access changes and every integration tool call, kept for 400 days and exportable through the API.

Real cost: pricing is usage-based credits per workspace with no per-seat fee, and the pricing page offers up to $100 in free credits with no credit card. On the Team plan unused credits roll over to next month and auto top-ups run under a monthly spend cap; Enterprise adds a per-user spend cap. Pricing checked October 2026.

Who can use him without IT: install him from the Slack App Directory or Microsoft Teams; teammates delegate by messaging him where they already work.

What he learns: the security page describes skills as “the persistent memory”, scoped to the workspace, encrypted at rest, never used to train models, and exportable or deletable on request.

Before you connect him: the security page lists SOC 2 Type 1 as certified with Type II in progress, ISO 27001 in progress, GDPR aligned, CCPA compliant and CASA Tier 3. It also states that API keys and tokens are injected at execution time by a tool gateway, so the model never sees them, and that admins can disconnect any integration, pause any user, or kill a running task in one click.

Start with up to $100 in free credits, no card required →

FAQ

How is this different from evaluating ChatGPT or another chat product?

A chat product is judged on its answers, because answers are all it produces. An agent is judged on its actions, because it changes records, sends messages and moves money in your systems. Questions 1, 3 and 4 do not apply to a chat product at all. If the vendor’s demo never leaves the chat window, you are evaluating a chat product, whatever the label says.

Should I pilot before buying?

Yes. Build the pilot from the three workflows you actually want to hand over, not the ones in the demo, and add edge cases such as a customer missing from HubSpot, stale data, or a Stripe call that times out halfway. Run each one with real data and approval on. Judge each run end to end, by whether the finished change landed in the target tool, and check whether the second run needed the same explanation as the first. A pilot that only touches the vendor’s sample data proves nothing, because sample data is what the demo was built on.

How do I compare two vendors on these 8 questions?

Write the eight questions in a column and each vendor’s answer next to it, in their own words. Mark each answer shown, described or dodged. “Shown” means you watched it happen in a demo workspace. “Described” means the vendor explained it. “Dodged” means the answer changed the subject. The vendor with more cells marked shown wins, regardless of feature count.

What if the vendor dodges a question?

Ask it again, more narrowly. “What happens when it is wrong” becomes “show me what the channel looks like when a Stripe call times out halfway through.” If the second answer is still general, mark it as dodged and move on. One dodge is a weak salesperson. Three dodges on the same list means the product does not do the thing.

How do I roll it out and keep checking it after launch?

Stage it. Launch one workflow with approval on, and treat its pilot set as the launch gate: the next workflow goes live only after the first passes. Confirm your admin can disconnect the integration in one click; that’s the rollback. After launch, the pilot set becomes your regression test: rerun it whenever the vendor changes the model, a connected tool changes, or your team hands over a new workflow. The action log is your observability; read a sample weekly and before each renewal.