Run an A/B test

An A/B test sends two or more versions of a campaign to different parts of your audience and tells you which did better.

Two things have to be true, and both are decided before you start:

• You picked A/B test rather than Regular when creating the campaign. This can't be changed afterwards.
• Email is one of the channels. The A/B section only appears for email.

The variants

The first variant is the Control — the version you're measuring the others against.

For each one:

• Variant name — for you, not the recipient. *Short subject* beats *Variant B*.
• Template — the content this variant sends.
• Subject line override — optional, and replaces the campaign's subject for this variant only. Testing only the subject means leaving the template the same on both and changing this.
• Traffic split — the share of the audience that gets it.

Change one thing at a time. If two variants differ in both subject and template, you'll know which won but not what made the difference.

How people are split

Random assigns each recipient by weighted chance as the send runs. Fine for a one-off.

Stable (by contact) always gives the same contact the same variant, worked out from who they are. Use this if you'll run the test again or send related campaigns to the same people, so nobody gets version A one month and version B the next.

What counts as winning

• Open rate — best for testing subject lines.
• Click rate — best for testing the call to action.
• Conversion rate — best when you care about the money rather than the interest.

Pick the one that matches what you changed. A new subject line can lift opens without selling anything more, so measure clicks or conversions if that's what you're after.

Letting Qcom decide

Auto-select winner picks the winner when the result is statistically significant, rather than when it merely looks better. Three settings shape it:

• Minimum sample size — how many people must be in the test first. The lowest you can set is 100. Raise it if the two versions are close, since a small difference needs a bigger group to show up.
• Confidence level — 90%, 95% or 99%. 95% is the sensible default. 99% is stricter and often won't declare a winner at all on a normal-sized list.
• Test duration — how long to wait before deciding, in hours. Give it at least a full day when you're measuring opens, since people read email at different times.

Leave auto-select off and nothing is chosen for you — you read the results and decide.

Reading the result

The campaign report shows each variant side by side. See Read your campaign results.

If no winner is declared, the two versions performed about the same. That's a result too — try a bigger change rather than a longer test.

Related

• Write your campaign message
• Choose who gets a campaign
• Read your campaign results
• Schedule and send a campaign
• Create a campaign