AB Testing

AB Testing

Subscription: PRO

The AB Testing module provides experiment management and statistical evaluation capabilities. It processes experiment assignments, aggregates user-level metrics, and runs Bayesian statistical analysis to determine which variant performs best.

Note: AB testing is currently only applicable to web personalization. AB testing for email campaigns is not yet supported.

‌

Key Requirements

Before setting up the AB Testing module, ensure that:

1. The PRO subscription is active.

2. The AB Testing service is configured and operational.

3. Web personalization is configured — AB testing currently only applies to web personalization experiments.

4. CDP events are flowing for metrics calculation (the module uses CDP events to measure experiment outcomes).

‌

How It Works

The module orchestrates the complete A/B testing evaluation workflow:

1. Experiment retrieval — Fetches active experiments from the AB Testing configuration service

2. User assignment processing — Determines which users are assigned to which experiment variants

3. Entry condition evaluation — Checks whether assigned users meet the conditions to enter the experiment (e.g., must trigger a specific event after assignment)

4. Metrics aggregation — Aggregates user-level metrics for each variant based on CDP events

5. Bayesian analysis — Runs statistical evaluation to determine variant performance and winning probability

6. Results publishing — Sends evaluation results back to the AB Testing service

‌

Metric Types

The module supports three types of metrics:

COUNT

Counts the number of events of a specific type per user per day.

• Example: Number of page views, number of add-to-cart actions

REVENUE

Sums monetary amounts from events per user per day.

• Example: Total purchase amount, average order value

BINARY

Tracks whether a user triggered a specific event (yes/no).

• Example: Did the user complete a checkout? Did the user click the campaign?

‌

Evaluation Output

For each metric and variant combination, the module calculates:

A probability of beating baseline of 0.95 means there is 95% confidence that the variant performs better than the control group.

‌

Entry Conditions

Users can enter an experiment in two ways:

• Immediate entry — Users enter the experiment as soon as they are assigned to a variant. Metrics are counted from the assignment date.

• Conditional entry — Users must trigger a specific CDP event after being assigned before they enter the experiment. Only events after the entry condition is met count towards metrics.

This allows you to measure the impact of a personalization only for users who actually encounter it.

‌

Configuration

Experiment setup (variants, metrics, entry conditions) is configured through the AB Testing service, not through the module's cockpit properties.

‌

Edge Cases

• Multiple variant assignments — If a user is assigned to multiple variants within the same experiment, they are excluded from results entirely (indicates a data quality issue)

• Entry condition not met — Users who never trigger the required entry event are excluded from metric aggregation but still tracked in assignment data

• No CDP events — Users with no matching events have no metrics calculated but are still counted in daily active user counts

‌

Use Cases

• Web personalization testing — Measure the impact of personalized vs. non-personalized recommendations on your website

• Feature rollout — Gradually roll out new web personalization features and measure their impact before full deployment

• Content testing — Test different content variants (product ordering, layout, recommendation strategies) on engagement metrics