The demo agent remembers the account, writes the follow-up, and finishes without a hitch. Your agent gets a conflicting note, an expired connection, and a colleague who changes the instructions halfway through.

Put those ordinary complications in the evaluation, so the trial shows whether the tool makes your team's work easier on inputs that look like your team's work.

Use the framework below to compare a new agent, a connection for an existing agent, or a workflow change. Download the evaluation scorecard and fill in the same cases for each option. No form is required.

What should an AI agent evaluation measure?#

Start with an accepted business result. For a renewal task, that means a draft saved on the correct account, based on current terms, ready for the account owner to review. A convincing paragraph does not count.

Separate five questions:

Question	What to observe
Did it finish the right work?	The result exists in the intended system and meets the acceptance criteria
What did a person have to fix?	Missing facts, wrong actions, repeated explanations, and manual completion
How long did the whole task take?	Elapsed time through review and final handoff, including waits
What did the task cost?	Model and tool charges plus the human work needed to finish it
What will the next task reuse?	A reviewed correction, procedure, or result with a clear scope

Keep these observations separate. A single score hides a fast answer that sends the wrong message, and an action outside the allowed scope fails the case however good the prose.

How do you choose representative tasks?#

Choose a small set of recent tasks with known outcomes: routine work, a common exception, and a handoff between systems.

Write the acceptance criteria before running the trial. Give the reviewer the expected result, allowed actions, sources to consult, and conditions that should cause the agent to ask for help. Use a test account or sanitized inputs when the work would otherwise contact customers or alter live records.

A useful renewal case: the account exists in the CRM, a recent email changes a contact, the previous meeting includes a promise, and the renewal terms require approval.

Give every option the same starting state, since an agent that sees the corrected document has a different task from one that sees the original. Record the model, instructions, permissions, and settings with each run.

What should you ask the vendor to demonstrate?#

Ask the vendor to complete one of your cases from start to finish:

Show where the agent found the account fact used in this draft.
Correct a fact, then run a related task and show whether the correction carries over within its intended scope.
Ask for an action the agent is not allowed to take and show what happens.
Approve a draft, change its recipient, and show whether the old approval still applies.
Interrupt the task after a tool call and show how the next attempt avoids doing completed work twice.
Show the finished record in the destination system and the history that explains how it got there.
Identify which parts require your current agent to be replaced.

If a tool cannot perform a step, record the workaround and its owner.

Which failure cases belong in the trial?#

Choose failures that resemble your team's work, and keep each case narrow so you can tell what caused the result.

Case	Change to the input or environment	Desired behavior to agree in advance
Conflicting information	Old note and current source disagree	Identify the conflict and use the agreed source or ask
Duplicate request	Submit the same business request twice	Avoid an unintended duplicate outcome or make the duplication clear
Uncertain tool result	A write times out before confirmation	Check what happened before repeating a consequential action
Permission change	Remove access before the next step	Stop the restricted action and explain what access is missing
Changed approval	Edit an approved action's target	Require a decision that applies to the changed action
Human correction	Fix an incorrect assumption	Use the correction within its intended scope
Handoff	Continue the work in another system	Carry the necessary task information and identify the next owner

For duplicate requests, distinguish repeated reads from repeated writes. A second search is harmless, or necessary when information changed; a second refund is a different outcome. Expose duplicate calls without teaching the agent to reuse stale answers.

Use the agent troubleshooting collection on repeated calls, duplicated messages, permissions, and missing documents to investigate a failed case.

How should you score the results?#

Use a rubric that reviewers can apply consistently:

Accepted: The result meets the case's criteria without a material correction.
Corrected: The result becomes acceptable after a person changes or completes it. Record that work.
Failed: The result is wrong, incomplete, or outside the agreed action boundary.
Not run: The case has not been tested. Keep it separate from success.

The CSV scorecard includes these fields alongside elapsed time, provider cost, reviewer minutes, and a link to the result. Add cases for your own workflow and repeat rows per option and run.

Have two reviewers independently assess a sample. If they disagree, refine the acceptance criteria before scoring the rest. "Looks good" is hard to repeat; "the draft names the current contact, uses the approved renewal terms, and is saved on the correct account" is reviewable.

Do not average away a critical failure. Set non-negotiable conditions in advance, such as no external send without approval, and compare only options that meet them.

How many runs do you need?#

Enough to see whether the result repeats across the types of work you intend to adopt. Start small to find setup problems, then repeat the important cases once configuration is stable. Record every run, including failures. Hold a few cases out of setup so the final evaluation includes tasks not used to tune the instructions.

If results swing between accepted and failed, check for changed source data, different permissions, or a different agent path before expanding.

For a high-volume workflow, sample its actual mix of tasks and exceptions. For an infrequent consequential workflow, spend the effort on how an action can be wrong and how a person stops or corrects it.

How do you compare cost and speed fairly?#

Use the same finish line: request to accepted result, including waiting for approval and human cleanup. Keep agent runtime visible separately so you can tell whether delay comes from the software or the surrounding workflow.

Record provider and tool charges separately from labor, and show your assumptions if you convert reviewer minutes into money.

Compare like task types, since a research summary and a CRM update cost differently. The finished-task cost guide explains how to keep retries and abandoned runs in the comparison instead of counting only successful output.

What should the decision meeting produce?#

Bring the completed scorecard, examples of accepted and failed results, and a proposed first rollout scope for one workflow, with a named owner and known limits.

Outcomes include adopting one option, fixing a connection and rerunning selected cases, narrowing the task, or keeping the current approach. A trial can show you already have a capable agent and need a better handoff.

If you are evaluating Deixic, bring your existing agent and one workflow. Start with connecting your agent, then run the same cases against the company's context, routed tool calls, approvals, and results. Dex is an optional agent if you also need one.

Once the team chooses a path, use the rollout checklist to turn the evaluation into an operable workflow.