# Visa's Agentic Commerce Benchmark Finds AI Models Fail 42.8% of Shopping Test Runs

On 5 November 2026 Visa released the Visa Agentic Commerce Benchmark, which tested 22 AI models from 12 labs on 15 shopping scenarios. Models failed 42.8% of test runs, and about 54% of failed runs still ended in a purchase.

URL: https://agenticcommerceatlas.com/news/visa-agentic-commerce-benchmark/
Author: Agentic Commerce Atlas | Publisher: Agentic Commerce Atlas
Updated: 2026-11-06

---

On 5 November 2026 Visa released the Visa Agentic Commerce Benchmark, a test of how well AI models handle realistic shopping tasks, and published the results of its first round. Visa says 22 AI models from 12 labs were tested on 15 scenarios, such as buying a TV, a laptop or a personal care appliance, and that the models failed 42.8% of test runs. About 54% of the failed runs still ended in the AI making a purchase. Visa calls it a first-of-its-kind industry benchmark and has made the test available to the public on its Visa Payments Frontier site.

## What Visa's benchmark tests

In each scenario the AI model has to identify a consumer's needs from a conversation, including hidden buying factors it has to uncover through the conversation, and then either buy a product that fits or decline to buy. A run passes only if the model both uncovers the customer's requirements and makes a correct purchase or no-purchase decision. Each model had three attempts at each scenario.

The results Visa reports for the first round:

- **Overall:** models failed 42.8% of test runs, a pass rate of 57.2%. The best-performing models passed 82.2% of attempts and the worst 28.9%.
- **Asking the right questions:** the five models that uncovered the most hidden buying factors averaged a 70.2% pass rate, against 44.4% for the five that uncovered the fewest.
- **Cost:** the five models with the highest average cost per run passed 67.1% of runs, against 50.2% for the five cheapest.
- **Consistency:** across all models and scenarios, all three attempts passed just 34.2% of the time.
- **Purchases on failed runs:** about 1 in 2 (54%) of failed runs still ended in a purchase, either of a product that did not meet the customer's identified requirements or before all the requirements had been uncovered.

Visa's release does not name the models in the text. Visa says it has shared the benchmark with leading model providers for feedback.

## Why Visa's benchmark results matter for merchants and payments

Most of the work on agent trust so far has focused on proving that an agent is legitimate and that a person approved a purchase: Visa's own Trusted Agent Protocol, AP2 mandates and the networks' agent registries (see [how agent purchases are authorized](/learn/ai-shopping-agent-payment-authorization/)). Visa's results point at a different failure: a legitimate, authorized agent that goes ahead and buys without having established what the shopper needs, or buys something that does not meet those needs. In this first round, about half (54%) of failed runs still ended in a purchase. For merchants, that is a returns and dispute question; for issuers and networks, it is a question of who is liable for a purchase made without the shopper's requirements being met or fully established (see [risks and readiness](/learn/agentic-commerce-risks-and-readiness/)).

Models that uncovered more hidden buying factors passed far more often (70.2% against 44.4%), a bigger gap than between the most and least expensive models (67.1% against 50.2%), though discovery is itself part of what a run needs to pass. Visa frames the benchmark in terms of trust, citing its own September 2026 Trust Index finding that 72% of US consumers had used an AI assistant but only 23% trust generative AI to handle payment transactions on their behalf.

## What the first round of Visa's benchmark does not cover

- **Checkout and payment.** The first round tests needs discovery and the purchase decision. Visa says it will extend testing to checkout, payment and order placement, and add more scenarios and product categories.
- **Real transactions.** The release describes "realistic buying challenges" and does not say whether purchases were made with real merchants or in a simulated store.
- **Independent review.** The benchmark, its scenarios and its scoring are Visa's own, and the figures are Visa-reported. Visa says a whitepaper and run replays are available through Visa Payments Frontier.

For how agents are authorized to pay today, see [how AI agents pay](/learn/how-ai-agents-pay/), and for the wider set of players, the [landscape](/landscape/agentic-commerce-landscape/).

## Frequently asked questions

### What is the Visa Agentic Commerce Benchmark?

A test Visa released on 5 November 2026 that puts AI models through realistic shopping tasks, such as buying a TV, a laptop or a personal care appliance. Each model has to work out a consumer's needs from a conversation and then make a correct purchase or no-purchase decision for the run to pass. Visa calls it first of its kind, which it defines as the first agentic commerce benchmark to require both discovery of every task-defined customer requirement and a correct purchase or no-purchase decision for a run to pass.

### How did the AI models perform?

According to Visa, the first round tested 22 AI models from 12 labs on 15 scenarios, with three attempts per scenario. Models failed 42.8% of test runs, for an overall pass rate of 57.2%. The best-performing models passed 82.2% of attempts and the worst 28.9%. All three attempts at a scenario passed just 34.2% of the time.

### Did failed runs still result in purchases?

Yes. Visa says about 1 in 2 (54%) of failed runs still ended in the AI making a purchase: either the product did not meet the customer's identified requirements, or the AI had not uncovered all of them. Visa does not say in its release whether the purchases were real or simulated.

### Does the benchmark test payment and checkout?

Not yet. The first round tests needs discovery and the purchase decision. Visa says it will extend testing to checkout, payment and order placement, and add more scenarios and product categories.

## Primary sources

- Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail: https://www.businesswire.com/news/home/20261105169310/en/Visa-Releases-First-of-Its-Kind-Industry-Benchmark-for-AI-Shopping-Almost-Half-of-Test-Runs-Fail
- Visa Payments Frontier: https://www.visapaymentsfrontier.io/
- Visa Trust Index explores agentic commerce adoption: https://corporate.visa.com/en/sites/visa-perspectives/press-releases/visa-trust-index-consumer-insights-into-ai-shopping-payments.html
