Visa benchmark finds AI agents fail 43% of shopping tasks
Visa has published what it describes as the first industry benchmark designed to measure how reliably AI agents can conduct real-world shopping on a consumer's behalf. The results, released on 5 November 2026, are sobering: across 22 models from 12 laboratories, the overall pass rate sat at just 57.2%, with consistency proving an even sharper problem than raw accuracy.
The payments network put each model through 15 buying challenges spanning categories such as televisions, laptops, and personal-care appliances. To pass, an agent had to do two things: surface every unstated requirement a consumer held (budget, format compatibility, brand preferences) through conversation, and then make a correct purchase or deliberate no-purchase decision. That pairing of discovery and decision-making is the benchmark's defining feature, and it is where most models fell short.
The numbers behind the failure rate
The spread between the strongest and weakest performers was wide. The top-performing model passed 82.2% of its attempts; the worst cleared only 28.9%. More telling still is the consistency figure: across all models and all scenarios, all three attempts at a given task passed just 34.2% of the time. A model that aced a TV-selection scenario in one run often failed the same scenario the next time around.
The benchmark also found a costly wrinkle in failed runs: roughly 54% of failures still ended with the AI recommending a purchase. The agent either bought a product that did not meet the customer's stated requirements, or it committed to a transaction before surfacing all the relevant constraints. For a consumer delegating spending decisions to an AI assistant, that is not a neutral miss.
Cost did correlate modestly with performance. The five highest-cost models per run passed 67.1% of attempts, against 50.2% for the five cheapest. But cost was no guarantee: expensive models still failed at scale, and cheaper models occasionally outperformed them on specific tasks.
Why Visa is building the scaffolding for agentic commerce
Visa's decision to publish this benchmark is as strategically significant as the results themselves. The company is not a model developer; it is the rails on which consumer spending moves. By establishing what "good" looks like for an AI purchasing agent, Visa is effectively positioning itself to set industry standards before regulators or competitors do.
That framing connects directly to the trust problem the benchmark surfaces. Visa's own Trust Index, published in September 2026, found that while 72% of US consumers had used an AI assistant, only 23% trusted AI to handle payment transactions autonomously. That gap between familiarity and financial delegation is the commercial obstacle Visa's Intelligent Commerce portfolio is designed to close.
The convergence angle here reaches well beyond payments. Retail and e-commerce platforms are already integrating agentic interfaces; model providers are racing to position their APIs as the preferred "shopping brain" for consumer applications. Whoever sets the benchmark for acceptable agent behaviour in a purchase flow will have outsized influence over which models get embedded in which platforms. That is a standards competition as much as a technology one, and Visa, sitting at the intersection of financial infrastructure and consumer trust, is well placed to lead it.
For capital allocators watching the agentic-AI space, the benchmark data points to a near-term product gap. Models that perform best in the Visa test tend to be the ones that ask more questions rather than move quickly to a decision. That design philosophy runs counter to the latency-minimising instincts baked into many commercial deployments. Bridging that tension, and doing so consistently across repeated interactions, is the technical and product challenge that the next generation of agentic commerce infrastructure will need to solve.
Visa says it plans to extend the benchmark to cover checkout flows, payment authorisation, and order placement, and has already shared the current version with leading model providers for feedback.