A/B Testing on Shopify: What to Test and How to Do It Right
A/B testing on Shopify works well when you have steady traffic and a clear question. It goes wrong when stores test tiny changes, stop tests early or measure the wrong thing. This guide covers what you can test, where Shopify gets in the way, and how to run tests you can trust.
It's part of the A/B testing guide for lead generation and ecommerce. If you capture leads instead of selling products, read A/B testing for lead generation.
If you haven't yet fixed the basics (mobile experience, speed, shipping clarity, product pages), start with Shopify conversion rate optimization. Testing works best after the obvious problems are gone.
Do you have enough traffic?
Ecommerce stores usually have an advantage over lead generation sites, because purchases tend to come faster. But a store with a few hundred orders a month still can't detect small changes. Use the A/B test win rate calculator with your real traffic and conversion rate to see how long a test would need. If it says several months, test a bigger change or improve the store using research instead.
What to test on a Shopify store
Focus on the pages where shoppers decide, in this rough order:
- Product pages. Product photos and video, the headline and description, price presentation, reviews placement, shipping and returns messaging, and the add-to-cart area. This is the page most stores benefit from testing first.
- Collection pages. Filters, sorting defaults, product card information and how many products appear above the fold.
- Cart. Free shipping progress bars, upsells, trust badges and the clarity of the checkout button.
- Offers and pricing. Bundles, quantity discounts, free shipping thresholds and subscription options. These often move revenue more than layout changes do.
- Homepage and navigation. Hero messaging and how easily shoppers reach products.
- Checkout. High-value but more restricted than the rest of the store, as explained below.
Shopify's built-in testing: Rollouts
Shopify now has a native testing feature called Rollouts, found in the admin under Markets. It lets you split traffic between two complete versions of your store, such as two themes, and recent reports say it can also test checkout and customer account configurations. Because it's built into Shopify, it's an easy first step if you've never tested before.
Its limits matter, though:
- It tests whole versions, not single elements. You compare one theme against another, rather than changing only a headline or a button, so you can't isolate exactly which change made the difference.
- Reporting is basic. Checkout results in particular focus on conversion rate, with little detail on revenue per visitor or order value, and the variant isn't always passed to your analytics for independent checking.
- Plan access varies. Which tests you can run (and whether checkout testing needs Shopify Plus) depends on your plan and has changed over time, so confirm in Shopify's current documentation before you plan a test.
Use Rollouts for big, deliberate changes such as a redesigned theme or a new checkout setup. For element-level tests on product pages, pricing or shipping offers, most stores still add a dedicated tool.
Choosing a testing tool
If you need element-level tests or deeper reporting than Rollouts offers, Shopify-focused tools such as Intelligems and Shoplift, and general platforms such as VWO and Convert, are common choices. Whichever you pick, check:
- Whether it works with your theme and apps without slowing the page or causing a flash of the original content.
- Whether it can test pricing and shipping offers, not only layout.
- How it reports revenue, so you aren't judging tests on conversion rate alone.
- Whether it integrates with your analytics, so you can verify its numbers independently.
Google Optimize, once the free default, was shut down in 2023, so older advice that recommends it is out of date.
Measure revenue, not just conversion rate
A variant can win on conversion rate and lose on money. A discount can raise the conversion rate and lower profit per order. Judge ecommerce tests on revenue per visitor as the primary metric where your tool supports it, and watch average order value and profit margin as guardrails. If a test increases orders but reduces order value, calculate the net effect before declaring a winner.
Mistakes that ruin Shopify tests
- Testing during unusual periods. Sales, holidays and big promotions change shopper behavior. Run tests across normal weeks, or plan the test to cover a full event consistently for both versions.
- Stopping early. Wait for the planned sample, and cover at least one or two full weeks so weekday and weekend behavior are both included.
- Running overlapping tests on the same page. The tests can interact and muddy the result.
- Ignoring mobile. Most Shopify traffic is on phones, so check that the variant works well on small screens and look at mobile results separately.
- Broken tracking. Confirm purchases are counted once and match your Shopify orders. The tracking and attribution checklist covers the common problems.
- Testing without a hypothesis. "Let's try a green button" isn't a plan. Base every test on something you observed.
A starter testing roadmap
- Verify tracking and compare tool numbers with Shopify orders.
- Use the conversion diagnostic to find where shoppers drop off.
- Test the product page first, starting with shipping and returns messaging, the offer and the photos.
- Move to cart and offer tests (shipping threshold, bundles).
- Review revenue per visitor and average order value for every test, then document the result.
If you want a second opinion on what to test first in your store, the CRO & Funnel Audit reviews your analytics, user behavior and conversion path, including Shopify storefronts, and returns a prioritized list of fixes for $99.