A holdout group is a share of an audience deliberately excluded from a campaign or automation so its behavior can be compared with the contacts who received it. It’s how a program measures what a message caused rather than what happened around it. An A/B test compares two versions; a holdout compares sending with not sending.
Category
Metrics
Also known as
global holdout; holdout; holdout audience
In practice
Keep a small permanent holdout on always-on automations (a few percent is usually enough once the audience is large; below that, hold out more) and read it quarterly. It’s the number that survives an argument about attribution, because the contacts in it saw everything else the business did except the automation. If the holdout and the treated group convert at the same rate, the automation is a cost, whatever the click reports say.
Something out of date?