Understanding A/B Test Results in Web Personalization

This guide explains A/B testing in plain language: when to use it, and how to read the results table in the AB Testing tool. It doesn't require any technical or statistics background.

What is A/B testing, in short?

A/B testing lets you show two (or more) different versions of a web personalization to different groups of visitors, and then measure which version performs better.

• Variant A (usually the "control" / baseline) — what visitors normally see, or a first version

• Variant B (the "treatment") — the version you want to test, e.g. a different layout, a different recommendation strategy, or personalization turned on vs. off

Visitors are randomly assigned to a variant. We then track what they do (clicks, add-to-carts, purchases, etc.) and calculate which variant is actually driving better results — not just which one looks better.

When should you use A/B testing?

A/B testing is most useful when you have a genuine choice to make — e.g. "should we launch this new homepage widget for everyone?" — and you have enough website traffic to get a reliable answer within a reasonable time frame. It's less useful for very low-traffic pages, since it can take a long time to gather enough data to draw a confident conclusion.

Reading the results table

Once an experiment has been running for a while, the AB Testing tool shows a results table like this (example numbers):

Here's what each column means:

• Variant set — Which version of the personalization this row describes. One row is always the control (baseline) — the version everything else is compared against.

• Users — How many unique visitors were counted in this variant. This matters because a result based on 50 users is much less reliable than one based on 5,000 — always check this before trusting a result.

• Metric value — The actual result for this variant, in the unit of whatever you're measuring. For example: the average number of clicks per user, the percentage of users who converted, or the average revenue per user. This is the "what happened" number.

• Uplift vs Baseline — How much better (or worse) this variant did compared to the control, as a percentage. +62% means this variant's metric value was 62% higher than the control's.

• P (Beating baseline) — The probability that this variant is genuinely better than the control, not just better by chance. This is a percentage from 0–100%. The higher it is, the more confident you can be that the improvement is real and not random noise.

• P (Best) — Only relevant when testing 3+ variants at once. The probability that this specific variant is the single best performer out of all the variants being tested (not just better than the control).

So... when is a result "significant"?

Unlike some A/B testing tools, this tool doesn't use a fixed pass/fail cutoff (like the classic "95% significant, or it doesn't count" rule). Instead, P (Beating baseline) is a probability you interpret on a sliding scale — the closer to 100%, the more confident you can be.

A rough rule of thumb:

A few important caveats:

• Check the Users column first. A 97% probability based on 40 users per variant is far less trustworthy than 97% based on 5,000 users. Small sample sizes can swing probabilities wildly from day to day.

• Let it run before deciding. Don't act on results after just a day or two — probabilities can fluctuate early on and settle down as more data comes in. As a rule of thumb, give a test at least a couple of full weeks (to smooth out day-of-week effects) before drawing conclusions, longer if traffic is low.

• A high uplift with a low P (Beating baseline) is not yet a win. A variant can show "+62% uplift" while the probability is only 60% — that's still essentially a coin flip; the underlying data is just too noisy so far.

• If you're testing more than 2 variants, use P (Best) to see which single variant is winning overall — a variant can have a decent P (Beating baseline) but still not be the best of the group.

Example: how to read the table above

Control converts at 2.1%. Personalized recs converts at 3.4% — a 62% relative uplift. There's a 97% probability this improvement over the control is real (not chance), based on roughly 4,800–4,900 users per variant — a solid sample size. Conclusion: this is a strong, trustworthy result — safe to consider rolling out the personalized recommendations.

What can be measured? (metric types)

Behind the scenes, every metric is one of three types — you don't need to configure this yourself (Building Blocks sets this up per experiment), but it helps to know what you're looking at:

• Count-based (e.g. number of page views, number of clicks) — "Metric value" is the average count per user.

• Revenue-based (e.g. average order value) — "Metric value" is the average revenue per user, including visitors who didn't buy anything (counted as €0).

• Yes/no based (e.g. did the user check out? did the user click?) — "Metric value" is the conversion rate, i.e. the percentage of users who did the thing at all.

A note on personalization clicks

Something worth knowing if you're setting metrics for a personalization: web personalization can now send a "personalization clicked" event to the CDP whenever click tracking is enabled on a widget. This is a relatively recent addition — until it was introduced, there was no way to measure clicks on a personalization at all, only downstream outcomes like purchases.

This means you can now set up a metric that measures engagement with a personalization directly (e.g. "did the visitor click on this recommendation widget?"), rather than only measuring whether it eventually led to a sale. This is especially useful for personalizations where the goal is discovery or engagement rather than an immediate purchase.

One current limitation: today, this only tells you that a click happened (and how many), not which specific product inside the personalization was clicked. That more granular, product-level click data does exist in the CDP data, so it can be very helpful to also look at these events!

Quick checklist before trusting a result

• Does the Users column show a reasonably large number for both variants (not a handful)?

• Has the experiment run for at least a couple of weeks?

• Is P (Beating baseline) at or above ~95%?

• If testing more than 2 variants, does P (Best) agree that this variant is the overall winner?

If all four are true, you're looking at a result you can act on with confidence.