AI failed 42.8% of Visa’s shopping test runs
Can the assistant walk away when nothing fits the budget?
- Visa benchmark
- 42.8% of runs failed
- Our local replay
- 1 of 3 cases passed
We test what AI actually gets done.
Independent tests and practical guides for choosing AI tools and checking the work they do.
Latest analysis
Latest tests
Claude Cowork Enterprise rollout checklist
Check whether the 16 November default change applies to your organization, then test one role’s file access and approvals.
Check your rollout settings
Can the support bot close this refund?
The bot says the full refund went through. The payment record shows only part of it. Decide what you’d do next.
Try the refund caseLatest tests
Can the agent finish the job?
Research-based guides to resolution, controls and reliable data.
Can a customer service agent resolve your problem?
Five questions to check what the agent can do when a case gets complicated.
The controls an AI agent needs in production
A guide to identity, permissions, approvals and the evidence your team can review.
Give AI agents the right data for the job
Design data access, retrieval and action so teams can verify the result.
Reference guides
Start with 25 picks
Which AI App Is Worth Paying For?
Choose an everyday AI app or compare the models behind it.
How to show up in AI search
Check whether crawlers can reach your pages and where AI systems cite your work.
How do you know an AI agent finished the job?
Build a record of what the agent did and how your team verified the outcome.
One useful AI pilot.
A plan you can actually follow.
Choose one task. Set the checks. Decide what deserves to scale.
Explore the free Playbook135 pages · Completed exampleFree guide with newsletter signup. Unsubscribe any time.
Read three excerpts before you sign up
Start with the work, set decision rules, then check readiness. These excerpts are from the full Playbook.
1 of 3
Start with the work.
Pick a task your team already does. Decide what should improve, how the work should change and which information you can trust.
See a completed pilot planQuestions from page 6 · The Digital Operating System
Ask these first
- Which business outcome is worth changing?
- How should work move tomorrow?
- Which inputs can the company trust?
Give AI clear limits.
Decide who checks the work, what AI may do and when someone must step in. Give it more freedom when those controls are ready.
Read the nine decision rulesRules from page 11 · Decision rules
Three rules to keep close
- Automate only after the process is understood.
- AI autonomy rises only when control rises.
- No owner means no transformation.
Scale what you can prove.
Name an owner, choose one measure and set a review date. At the review, decide whether to expand the pilot, change it or stop.
“Scale only what produced inspected before/after value.”
Check your pilot’s readinessReadiness test from page 41 · Quoted rule from page 11
Make the decision explicit
- Who owns it?
- One accountable owner
- What metric improves?
- One primary KPI
- What happens by day 90?
- Scale, stop, or redesign decision
Why I publish AI Vanguard
I want this publication to help you make sound decisions about AI. Start with the shopping replay or compare the refund records. When we run a test, we show the inputs, the result and what we learned.
Ehab Al Dissi
How we report