A/B Testing with Claude Code: Simulate Experiments Before You Spend Real Traffic
The most useful conversion insight of my career came from a dashboard nobody asked me to open. At Namu, acquisition ran entirely on Facebook Ads, and when I compared segments, organic visitors were converting noticeably better than paid ones. That single comparison ended up reshaping the whole content strategy, focusing on blog content with CTAs instead of paid landing pages.
Claude Code can now run that kind of comparison before a single real visitor sees your page. You hand it the business context plus a hypothesis, approve a test plan, and it simulates an A/B test between two variants and writes up the projected winner. The simulation works as a triage step, and real users still make the final call.
What is a Claude Code A/B test simulation?
Basically, it’s language model role-playing your target user against two versions of a page, then estimating which version wins on a metric you defined. Claude Code reads both variants from your repo, applies the persona and business context you gave it, and produces a report with projected conversion numbers.
Those numbers are synthetic. They come from the model's understanding of how people behave, which is broad but generic. What makes the exercise useful anyway is that most losing variants lose for boring, predictable reasons: unclear value proposition, a form asking for too much too early, a CTA below the fold, no trust signal near the payment step. A model catches those instantly.
The biggest difference is that a real user panel costs money and weeks. A simulation costs just a few prompts.
Why bother, when real A/B testing exists?
Because most sites can't run a real A/B test that means anything. Run the numbers through any sample size calculator: detecting a 10% relative lift on a 3% conversion baseline, at standard significance and power settings, needs roughly 50,000 sessions per variant. A client store doing 20,000 sessions a month would need five months per test, assuming nothing else changes in that window.
So, or you have volume, or any decision is biased towards your own preferences. In practice, teams either skip testing entirely or run underpowered tests and ship whichever variant was ahead when someone got impatient. I've written before about why AI demos fail in production, and underpowered A/B tests fail for the same root cause: the gap between what was validated and what ships. A simulation at least stops you from wasting your limited real traffic on a variant that was never going to work anyway.
Step 1: build the context before asking anything
Claude Code simulates users only as well as it understands who those users are. Before touching the test itself, feed it the business model, the target persona, the goal of the specific screen, and read access to the implementation itself.
The pattern that works is a short inline description paired with @file references to deeper docs:
This is a WooCommerce store selling wellness supplements in Canada.
Business model details: @docs/business.md
Target customer profile: @docs/persona.md
Explore the codebase and analyze how the checkout flow is implemented.
Do not change any files.
I want to A/B test the "create account" step, so build an
understanding of how it currently works.
If anything is unclear, ask me questions.
The @file mentions matter because Claude Code opens those files on its own when it needs depth, so you keep the prompt short without starving it of context. The "ask me questions" line matters just as much, and it's the same principle behind the constraint-heavy prompting I covered in PACREF Claude Code prompts: the model performs better when the boundaries are explicit.
Step 2: hypothesis first, plan second
A test without a written hypothesis produces a report you can't act on. Switch Claude Code into Plan Mode (Shift+Tab cycles permission modes), which locks it into read-only analysis, and give it a structured hypothesis before asking for the plan:
Hypothesis:
If the account step explains what the address data is used for,
more users will complete it, because the value becomes clear
before the commitment.
Current flow:
User is asked to create an account with a full address at checkout step 2.
Business goal: conversion
Primary metric: step completion rate
Guardrail metrics: page load time, CTA click latency
Two rules keep this honest. Use exactly one primary metric, because a simulation asked to optimize four numbers at once starts hallucinating trade-offs between them. And define guardrail metrics, the operational numbers that expose fake wins. A variant that lifts completion by loading a heavier script has failed, and without a guardrail in the plan, the report will call it a winner.
With the hypothesis approved, ask for the actual plan: reference both variant files, state their differences in one line, and request the assignment logic, the test plan itself, the analytics events, and the conditions for rolling back. Review it like you'd review a junior dev's migration plan, because that's functionally what it is.
Step 3: run the simulation and fence it in
The execution prompt has one job beyond running the test: forcing the report to separate fact from fiction. This is the prompt shape I use, adapted from Babich's original:
Use the approved A/B testing plan as the source of truth.
Simulate the test described in that plan.
Do not modify product code. Do not change the plan.
The report must clearly separate:
- what came from the original plan
- what was assumed for the simulation
- what the simulated results suggest
- what still needs validation in a real experiment
Confirm that no product code was modified.
When the report says variant B wins by 4 points, you need to see, in writing, that the traffic numbers were assumed and the significance calculation ran on synthetic data. I've watched enough AI output get pasted into client decks as fact to insist on this. Repeat the "do not modify code" instruction even though Plan Mode already restricted the planning phase, because the execution run happens outside it.
If the simulated winner survives your skepticism, the follow-up is mechanical: promote the winning variant to default, remove the branching logic and experiment config, update the tests, run lint and typecheck. Claude Code handles that cleanup well, the same way it handles the migration and refactoring work in my Claude Code WordPress workflows.
What to do with a simulated result
Treat the simulation as a filter, and the decision tree stays simple. If the simulation shows one variant clearly losing for a concrete, explainable reason, kill it and don't spend traffic on it. If both variants look close, ship the simpler one and move on. If the simulation shows a strong winner and the page matters commercially, that's the variant that earns your one real test, with real analytics events, on real users.
If your store is making conversion decisions on gut feeling because real A/B testing was never feasible at your traffic level, setting up this kind of AI-assisted validation loop is exactly the consulting work I do. Get in touch.