A/B Testing Isn’t Dead. Your Assumptions Are.

AI agents have taken over Conversion Rate Optimisation.

Every few months, someone stands on a stage and announces that AI agents will do all the shopping, your website will become irrelevant, and conversion rate optimisation is finished.
That’s not what the evidence says. It says something more interesting, and more useful.

The myth: “Agents will buy everything, so stop testing your site.”

The reality is that the way people find and buy things is changing, and most teams are measuring the new world with the old rulebook. That’s the real risk.

What’s actually happening (not what’s being sold to you)

Amazon says its Rufus shopping assistant helped more than 300 million customers in 2025. Adobe found that in March 2026, AI-referred traffic converted 42% better than non-AI traffic. A year earlier, it converted 38% worse.

That’s a big swing, and people are using AI to research and compare. But look closer:

Hot take: most of the “agent revenue” stories you’re reading are really stories about assisted humans. Nobody has published a trustworthy figure for how much of the market is bought end to end by an agent. Anyone quoting you a precise percentage is guessing.

The proof is in how AI helps humans

Here’s where the hard evidence sits, and it’s not about robots buying things.

Notice the pattern. The gains came from removing friction and uncertainty for a human, and mostly from more people buying, not bigger baskets. Fix confusion, and you win. It’s the oldest rule in CRO.

Agents are not rational robots

Another myth: “Agents will pick the objectively best product, so merchandising doesn’t matter.”

Researchers built a simulated store called ACES to test this. Agents showed position bias, penalised “Sponsored” labels, rewarded endorsement badges like “Overall Pick”, and preferred different products depending on which AI model was doing the shopping. Small changes to a product description could shift selection.
One finding should make every urgency-obsessed marketer uncomfortable: scarcity labels were weakly negative or no help at all. Your “Only 2 left!” banner may not work on a machine.

But hold on. That’s a simulation. It shows how agents behave in a lab, not what happens to your revenue. Use it to form hypotheses, then test them with real orders.

The story that should change your mindset

Anthropic ran an internal experiment called Project Deal. 69 employees let AI agents negotiate trades of their personal items. The real run produced 186 deals worth just over $4,000.

The stronger models got better deals. The people represented by weaker agents? Just as satisfied. They didn’t notice they’d lost out.

Satisfaction and outcome are not the same thing. If you only measure how happy people feel, you’ll miss when your system is quietly costing them, or you. It’s a small, self-selected sample, so don’t over-read it. The lesson still stands: measure real outcomes, not vibes.

Where the website still wins (and where it doesn’t)

Think in three journeys:

If a transaction runs purely through an API, your button colour never gets rendered. Your catalogue completeness, stock accuracy, total-cost clarity and offer rules do the selling. The same person can use all three journeys in one week.

So don’t stop testing your site. Measure how many journeys actually reach each part of it. A checkout test only tells you about people who saw that checkout.

Agents still struggle with real shopping

The tech is real, and it’s early.

Objections we keep hearing

“If agents are coming, why not go all-in now?”

 Because you’d be betting on a forecast. Nobody can honestly give you a date or a market share. Fix the shared foundations first. They help every shopper and every agent.

“Does mixed human and agent traffic ruin my A/B tests?”

No. Randomisation still works with mixed traffic. What breaks is sloppy measurement: counting every request as a customer, filtering traffic in ways your test changes, or confusing referral conversion with incrementality. These are old problems wearing new clothes.

“Can’t I just test with simulated agents?”

Use them to find broken things, like unreadable policies or wrong variants. Then validate against real orders. One experimentation platform documented an AI support experiment where the correctness checks passed but follow-up support queries got worse. Passing the test isn’t winning.

“Is ‘cart created’ a win?”

No. As the ShopGym researchers point out, a cart created successfully is not the right order, paid, delivered and kept.

What to do this week

• Stop calling all AI traffic “agent shoppers.” Split it into AI-referred visits, crawlers and verified delegated orders, and keep an “unclassified” bucket instead of forcing everything into “human.”
• Reconcile your test results with server-side paid orders. Don’t trust the front-end alone.
• Audit your facts. Do price, stock, delivery, returns and offer rules match across product pages, feeds and checkout?
• Randomise at the customer level, not the request level. One agent task can create dozens of requests, which fakes your sample size.
• Run an A/A test. If your setup can’t tell two identical experiences apart, it can’t be trusted with a real test.
• Measure what lasts: returns, support contacts, retained margin and repeat purchase, not just the first conversion.
• Keep the human work going. Customer research, accessibility, search and checkout fixes still pay.

The mindset

Here’s the contrarian bit: don’t optimise for agents, optimise for the customer. The agent is not a second customer. It’s an intermediary working for the person who is paying.

So the goal isn’t “get picked by the machine.” It’s helping a real person get the right product, on terms they understand, through whatever route they choose, at a price that works for your business.
Clear data, honest policies, reliable checkout and rigorous testing win in every scenario. They win when a human clicks, when an assistant recommends and when an agent buys.

The brands that panic and rebuild around hype will waste a year. The brands that ignore the shift will be blindsided. The ones who win will be calm, curious and very, very good at measuring what actually moves the needle.

So here’s your move: pick one thing this week, whether it’s the A/A test, the data audit or the traffic split, and do it. Then tell me what you found.

The sources synthesise findings from a diverse range of evidence, spanning rigorous academic theory to real-world commercial applications. The key areas of synthesis include:

• Allouah, A., Besbes, O., Figueroa, J. D., Kanoria, Y., & Kumar, A. (2026). What is your AI agent buying? Evaluation, biases, model dependence, & emerging implications for agentic e-commerce. Proceedings of the ACM Web Conference 2026 (WWW ’26), 8697–8700.
• Frank, D. A., & Otterbring, T. (2024). Autonomy, power and the special case of scarcity: Consumer adoption of highly autonomous artificial intelligence. British Journal of Management, 35(4), 1700–1723.
• Frank, D. A., Folwarczny, M., & Otterbring, T. (2025). Consumer acceptance of high-autonomy AI assistants is driven by perceived benefits in online shopping settings characterised by scarcity. Psychology & Marketing, 43(3), 538–555.
• Wadi, D., & Ma, Y. (2026). Does rank still matter? Position bias when AI agents shop on our behalf. Desautels Faculty of Management, McGill University Working Paper.
• Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., & Su, Y. (2023). Mind2Web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems (NeurIPS 2023), 36, 28091–28114.
• He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., & Yu, D. (2024). WebVoyager: Building an end-to-end web agent with large multimodal models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 6864–6890.
• Kim, S., Heo, R., Seo, Y., Yeo, J., & Lee, D. (2026). AgenticShop: Benchmarking agentic product curation for personalised web shopping. Proceedings of the ACM Web Conference 2026 (WWW ’26), 2489–2500.
• Li, A., Mao, K., Guo, Y., Xia, H., Yang, X., Luo, L., Wong, J., Yao, P., Xu, S., & Gu, S. (2026). ComboShoppingBench: Evaluating LLM agents for budget-constrained basket shopping with coupons. arXiv preprint, arXiv:2608.09282.
• Lyu, Y., Zhang, X., Yan, L., de Rijke, M., Ren, Z., & Chen, X. (2025). DeepShop: A benchmark for deep research shopping agents. arXiv preprint, arXiv:2506.02839.
• Peeters, R., Steiner, A., Schwarz, L., Caspary, J. Y., & Bizer, C. (2026). WebMall – A multi-shop benchmark for evaluating web agents. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26).
• Savadikar, C., Zhao, M., Zhu, Y., Li, H., Xie, S., Castelo, A., Wu, T., & Wang, L. (2026). ShopGym: An integrated framework for realistic simulation and scalable benchmarking of e-commerce web agents. arXiv preprint, arXiv:2605.16116.
• Tou, H., Zeng, Y., Li, Y., Ma, C., Li, M., Li, M., Yuan, W., Zhang, H., & Jia, K. (2026). ShoppingComp: Are LLMs really ready for your shopping cart? arXiv preprint, arXiv:2511.22978.
• Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022). WebShop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems (NeurIPS), 35, 20744–20757.
• Yu, Z., Xiao, K., Zhao, H., Luo, T., & Zeng, X. (2026). Shopping Companion: Benchmarking and training LLM agents for long-horizon preference-grounded e-commerce tasks. arXiv preprint, arXiv:2603.14864.
• Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., & Neubig, G. (2024). WebArena: A realistic web environment for building autonomous agents. International Conference on Learning Representations (ICLR 2024).
• Castelo, A., Foumani, Z. Z., Fan, A., Koay, K. Y., Malik, V., Zhu, Y., Li, H., Feghhi, M., Uliana, R., Xie, S., et al. (2026). SimGym: Traffic-grounded browser agents for offline A/B testing in e-commerce. arXiv preprint, arXiv:2602.01443.
• Fang, L., Yuan, Z., Zhang, K., Donati, D., & Sarvary, M. (2025–2026). Generative AI and sales productivity: Field experiments in online retail. arXiv preprint, arXiv:2510.12049.
• Luo, C., Choi, J., Dong, Z., Dua, R., Xu, C., Lei, X., Yan, Y., Zhang, X., Valvoda, J., Sinkar, G., Jha, B., Liu, Y., & Cheng, M. (2026). Building a production shopping agent at scale. Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), 1–5.
• Wang, L., Huang, N., He, Y., Liu, D., Guo, X., Sun, Y., & Chen, G. (2025). Artificial intelligence (AI) assistant in online shopping: A randomised field experiment on a livestream selling platform. Information Systems Research, 36(4), 2358–2374.
• Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.
• Larsen, N., Stallrich, J., Sengupta, S., Deng, A., Kohavi, R., & Stevens, N. T. (2024). Statistical challenges in online controlled experiments: A review of A/B testing methodology. The American Statistician, 78(2), 135–149.
• Balaskas, S. (2026). From recommendations to delegation: A systematic review mapping agentic AI in e-commerce and its consumer effects. Information, 17(3), 222.
• Balaskas, S., Komis, K., Yfantidou, I., & Skandali, D. (2026). When interfaces “act for you”: An eye-tracking experiment on delegation, transparency cues, and trust in agentic shopping assistants. Information, 18(3), 18.
• Restrepo Amariles, D., Charlotin, D., & He-Guelton, L. (2026). AI agents in payments: Applications, risks and regulations. European Journal of Risk Regulation, 1–19.

• 
  A/B Testing Isn’t Dead. Your Assumptions Are.
• 
  Why ‘Fast’ Websites Lose Money: The Counterintuitive Science of E-Commerce Speed
• 
  The Homepage Carousel Myth: Why Static Beats Spinning for E-commerce Conversions
• 
  Why 90% of Your A/B Tests Are Lying to You (And the Secret Top Tech Companies Use)
• 
  The Headline Myth That’s Costing You Conversions (And What Actually Works)
• 
  STOP Surveying Your Customers: The Controversial Truth Behind 30% More E-Commerce Sales