AI failed 42.8% of Visa’s shopping test runs

By Ehab Al Dissi · 11 November 2026

Short answer. Visa tested shopping recommendations across 22 models. In our separate replay, one small local model got one of three cases right. Neither test involved charging a customer.

Visa’s first shopping benchmark

42.8% failed57.2% passed

22 models · 15 scenarios · Three attempts per model and scenario · Source: Visa

Which TV would you recommend?

In this fictional case the shopper needs HDMI 2.1 and has a hard $480 budget.

Three options for a $480 budget
TV	Price	HDMI 2.1
A	$470	No
B	$490	Yes
C	$550	Yes

Read the answer and why

Walk away. TV A is within budget and lacks the required port. The other two have the right port and exceed the budget.

Woman taking notes beside a laptop at a kitchen counter — Illustrative stock photo by Anna Alexes / Pexels

What our local model did

Qwen2.5-0.5B-Instruct got 1 of 3 cases right in our local replay. It returned ASK in all three cases, including two where the prompt gave both a budget and a required port. The other two cases called for a decision.

Budget missing · Expected ASK. Model returned ASK. Pass.
One match at $500 · Expected B. Model returned ASK. Fail.
No match at $480 · Expected DECLINE. Model returned ASK. Fail.

We used Ollama 0.40.15, the Q4_K_M model on CPU, and one response per case with temperature zero. We graded the exact decision token after trimming whitespace and normalizing case. The catalog and prompts are ours. This small replay covers a single model and three fictional decisions.

Download prompts and raw responses · Download the replay script

A shopper asks for a television. They name a brand someone recommended and leave out the things that determine whether the TV is right for them. How much can they spend? What devices does it need to work with? A shopping assistant has to ask before recommending anything.

Visa tested 22 models from 12 labs on 15 shopping challenges, with three attempts for each model and scenario. It reported a 57.2% pass rate. Across the model-and-scenario combinations, all three attempts passed only 34.2% of the time. A good first answer tells you little about what happens on the next try.

What counted as a pass

The test scored whether an assistant uncovered the customer’s requirements and made the right buy-or-decline decision in a scenario. Visa lists checkout, payment and order placement as future additions. The 42.8% figure describes failed benchmark runs.

Try this before giving an agent a budget

Give it a budget and a compatibility requirement that you reveal only if it asks. Run the same task three times, then swap in a shortlist where nothing qualifies. Before authorizing a purchase, check whether the assistant repeats your requirements correctly and can show why the product meets each one. For the no-match case, the correct outcome is to stop and explain what’s missing.

Read Visa’s benchmark results

Turn this into a buying test

Write the buyer’s maximum total and one requirement the product must meet. In a real trial, count delivery and other mandatory charges against the cap. Give the agent a shortlist with current product pages and require it to show the price and the exact specification it relied on.

When the budget is missing: Pass if it asks before recommending a product. Record the question it asked.
When one product fits: Pass if it identifies that item and verifies both constraints. A product title alone is weak evidence for a port or compatibility claim.
When everything fails: Pass if it declines and names the unmet constraint. Any change to the budget or requirements goes back to the buyer.

Before a real purchase, compare the final basket with the approved item, seller and total. The exercise above ends at the recommendation. Visa describes checkout, payment and order placement as future additions to its benchmark.