Why FLYDE
Platform How it works Integrations Security ★ FLYDE Brain Core Customer 360 Market Trends Analytics
Solutions FLYDE Brain Omnichannel attributionIncrementality Omnichannel customer journey B2B + B2C identity Predictive models See all solutions Retail Automotive NGOs Banking Restaurants EnergyFitness
Resources Blog FLYDE Talks Whitepapers & ebooks Self-assessment ROI Calculator Glossary
Login
Glossary · Measurement

A/B testing: what it is, how to do it and common mistakes

FLYDE Glossary · November 19, 2026 · 3 min read

A/B testing compares two versions of something (an email, a page, an offer) by showing each to a randomly chosen group of people, to find out with data which one works better.

How to run an A/B test

  1. State a hypothesis: "a subject line with the customer's name will increase opens".
  2. Change one thing only between version A and B, so you know what caused the difference.
  3. Choose the metric before you start: open, click, conversion or, better still, sales.
  4. Split people at random between the two versions.
  5. Wait for enough sample and don't stop the test as soon as one version is ahead.
  6. Check the difference is significant and not due to chance.

What sample size you need

It depends on the baseline rate and the difference you want to detect. Detecting a small lift on a low conversion rate takes thousands of people per version; large differences need only hundreds. With a small base, test bolder changes.

Common mistakes

  • Checking results daily and stopping at the first favorable number.
  • Changing several things at once.
  • Optimizing for clicks when what matters is sales.
  • Running the test in an unusual week, such as Black Friday.

A/B test vs control group

An A/B test tells you which of two versions works better. A control group answers a different question: how much the campaign sells compared with doing nothing. Version B can beat A and still sell nothing that wouldn't have sold anyway. To know, you need to measure incrementality.

Testing with FLYDE

FLYDE lets you split random groups on unified profiles and measure the result in real sales, online and in store, not just opens or clicks. Its incrementality module calculates how much each campaign really sells against its control group.

Frequently asked questions

What is A/B testing? An experiment that compares two versions of an element by showing each to a randomly chosen group, to find out which works better.

How long should an A/B test run? Long enough to reach the planned sample size and cover at least one full purchase cycle, usually one to two weeks at minimum.

What is the difference between A/B and multivariate testing? An A/B test changes one element between two versions; a multivariate test tries several combinations of several elements at once and needs far more sample.

What is the difference between an A/B test and a control group? An A/B test compares two versions with each other; a control group compares the campaign with doing nothing, which measures incremental sales.

Related terms

← Back to the glossary

Free self-assessment

How much revenue is sleeping in your customer data?

Twelve questions, two minutes, no sign-up. Get your customer-data maturity level and the three priorities to start with, instantly.

Take the self-assessment

Shall we start?

Give us 15 minutes and see how much FLYDE can untangle your day-to-day.

Let's talk