How to Test Webinar Reminder Calls: The Holdout Design

A holdout test on a webinar list means splitting your registrants into two groups at random, running your reminder calls or texts on one group only, leaving the other group entirely alone, and comparing the two show up rates. It is the only design that tells you whether the calls caused anything, because everything else about the two groups - the ads, the offer, the week, the weather - is identical by construction. This page gives you the design, and then does the thing nobody else publishes: it works through the sample-size arithmetic so you can see, before you spend anything, whether your own webinar is big enough for the answer to mean anything.

What is a holdout test on a webinar list?

Take the registrant list for one webinar. Before any reminder goes out, assign each registrant at random to one of two groups. Group A gets the full treatment - the calls, the texts, whatever you are testing. Group B gets nothing beyond whatever your baseline already was. When the session runs, you count how many people from each group attended, and the difference between the two rates is your estimate of what the treatment did.

The reason this design is worth the discomfort is that it removes every alternative explanation at once. If you instead compare this month's webinar (with calls) against last month's (without), the difference could be the calls, or it could be that you changed the ad creative, or that it is now July, or that the topic was better. A random split within one event holds all of that constant by construction, which is exactly why it is the test a serious buyer asks for.

One clarification before the arithmetic, because it changes what "the treatment" means. A holdout measures the whole package you applied to the treated arm, not any single component inside it. If the treated arm gets a text, a call, and a different email sequence, you have measured all three together. That is fine if what you want to know is "does buying this thing help me". It is useless if what you want to know is "which of the three did it", which needs a much larger test with more arms and is out of reach for almost every operator running this.

How many registrants do you need to detect a real lift?

This is the section that decides whether the rest of the page is worth acting on, and it is arithmetic rather than a claim. We are going to work it through in full so you can redo it with your own numbers and disagree with us if the disagreement is real.

Worked example, step by step

Take the base case: a 30% baseline lifted to 38%. Substitute and the whole thing is four lines of school arithmetic.

• p₁(1 − p₁) = 0.30 × 0.70 = 0.21
• p₂(1 − p₂) = 0.38 × 0.62 = 0.2356
• Sum = 0.4456. Difference squared = (0.38 − 0.30)² = 0.0064
• n per arm = 7.84 × 0.4456 / 0.0064 = 7.84 × 69.625 = 545.9, round up to 546

So 1,092 registrants in total, 546 in each arm, to have a four-in-five chance of detecting an eight-point lift if that lift is genuinely there. Here is the same calculation across the range of effects worth caring about.

Where the "realistic effect" number comes from

We did not pick 34% to make our own product look hard to prove. It is what happens when you take the strongest transferable reminder evidence and apply it to a 30% baseline.

Apply RR 1.14 to a 30% baseline and you get 34.2%. That is a four-point lift, and four points is what the 4,258-registrant row in the table above is priced for. The Cochrane review is Gurol-Urganci and colleagues, 2013, and it covers healthcare appointment attendance in Australia, China, Kenya, Malaysia and the UK - not webinars. We use it because it is the best-designed reminder evidence that exists, and we label the transfer as an assumption rather than a finding.

The other anchor is Nickerson and Rogers (2010), whose abstract states that facilitating a voting plan "can increase turnout by 4.1 percentage points among those contacted, but a standard encouragement call and self-prediction have no significant impact". Two things travel with that number every time we use it. The 4.1 points is the effect among people the callers actually reached, not among everyone assigned to be called. And the setting is the 2008 US presidential election, with live human callers - whether the same effect survives when the caller is a disclosed AI is, as far as we can establish, unpublished. Nobody has tested it.

What can a single webinar actually detect?

Run the formula in the other direction. Fix the number of registrants you actually have, and solve for the smallest effect the test could reliably find. This is the table that should decide whether you run the test at all.

So the plain answer to "can I settle this with my next webinar" is: almost certainly not. A 400 to 600 registrant event split down the middle can only see effects roughly two to three times larger than the ones the published reminder research supports. If you run it anyway and get a null, you have learned nothing about the treatment - you have learned that your test was too small, which you already knew from this table.

Should you split 50/50 or 90/10?

Most operators want a 90/10 split. The instinct is understandable: treat almost everyone, hold back a token slice, lose almost nothing. The arithmetic says this choice is expensive, and the reason is worth understanding in plain language rather than taking on trust.

The smaller arm limits you. The precision of a comparison is governed by whichever side has fewer people, because a rate measured on 60 people is a wobbly number no matter how precisely you have measured the other 540. Adding more people to the already-large arm buys you very little; the uncertainty is coming from the small side. A 50/50 split is the allocation that makes both arms as precise as they can be for a fixed total.

The published correction for this is simple. Suresh and Chandrashekara (2012) state it directly: "If r = n1/n2 is the ratio of sample size in 2 groups, then the required sample size is N1 = N(1+r)²/4r", where N is the total you calculated assuming equal groups.

Read the last two columns together, because the trade is not the one people expect. Going lopsided shrinks your holdout arm only modestly - from 546 at an even split down to about 303 at 90/10 - while the total list you must accumulate nearly triples. You are buying a slightly smaller holdout at the cost of almost three times the registrants. And there is a floor: no matter how enormous you make the treated arm, the holdout arm for this effect cannot go below about 257, because at that point the uncertainty is coming entirely from the small side and adding to the other side buys nothing at all.

What should you measure, and where is the trap?

Show up rate is the obvious metric and it is the wrong one to measure alone. The objection that gets raised in every serious conversation about this is: show rate is a vanity metric, where are the sales? That objection is correct, and it has a specific mechanism behind it rather than just cynicism.

Measure the whole chain in both arms, and keep the arm label attached to every downstream record so you can trace a sale back to the group it came from. That last part is the piece people forget, and without it the test is unrecoverable after the fact.

How do you actually run the test?

The design is not complicated. The discipline is where these fall apart, and every step below exists because it is a way we have seen a test become unreadable.

What contaminates a webinar holdout?

Randomisation protects you from confounding, not from leakage. These are the failure modes specific to running this on a registrant list rather than in a lab, and most of them do not announce themselves.

What if you will not withhold it from half your list?

Some operators refuse to run a holdout because they believe the treatment helps, and deliberately withholding help from half their audience feels wrong. That is a legitimate ethical position and not an objection to argue past. If you genuinely believe the calls help the registrant - not just you - then randomising who gets helped is a real cost, and the fact that it produces a cleaner number does not automatically settle the question.

There is a weaker design available that most people find ethically comfortable: alternate the policy across consecutive events. Run the system on event one, not on event two, on event three, not on event four, and compare the pooled treated events against the pooled untreated ones. Everyone at a given event is treated the same, which is what the objection was actually about.

If you alternate, alternate rather than switching once. Treated, untreated, treated, untreated, so that a drift in your ad costs or your list fatigue lands roughly equally on both arms instead of entirely on the second half of the experiment. And write down the traffic mix per event, because if it moves, you will want to know whether it moved before or after you saw the result.

How long before you know anything?

Several events, not one. That is the whole answer, and the tables above are why. If your webinars run 500 registrants and you are looking for the realistic four-point effect, you need roughly nine events before the arithmetic gives you the power you assumed you had from the beginning. At a monthly cadence that is most of a year.

There is a shorter honest version. If you are prepared to settle for "is this pointing the right way and is the interval consistent with the published evidence" rather than "is this proven", three or four events will tell you something worth knowing, and you will have a confidence interval you can report without embarrassment. What three or four events will not give you is a number to put in a case study, and if a vendor produces one from a sample that size, the arithmetic on this page is how you check it.

Write the rules down before you start

Pre-registration is a formal habit borrowed from clinical research and it takes fifteen minutes. Write down, before the first registrant is assigned: the primary metric, the secondary metrics, the arm sizes, the number of events you will pool, the analysis you will run, and the stopping rule. Then do not change any of it while the test is running.

When should you not bother testing at all?

Testing is not free. It costs list size, operational discipline, and the option of just getting on with it. There are cases where the correct decision is to skip the test and decide on judgement instead, and it is worth naming them rather than implying everyone should run an experiment.

What this means if you are thinking of buying this from us

We sell a system that calls and texts webinar registrants, and this page hands you the tool to check whether it works, including the arithmetic showing that most single tests cannot prove it either way. That is deliberate. We would rather you arrive at a conversation knowing what a real answer would cost than sign on the strength of a number we made comfortable.

One consequence follows for anyone shopping this category. If a vendor tries to talk you out of a holdout, you have learned what you needed to know. The design is not exotic, it costs the vendor nothing but the risk of a null result, and the only reason to resist it is not wanting the number. The same test applies to us - and if we ever ask you to skip it, hold us to this paragraph.

For what the numbers underneath a test like this actually look like, our teardown of every published webinar attendance study gives the sources and the samples. The two questions most operators should settle before spending money on reminders are covered in does charging for a webinar increase attendance - the largest reported lever in the whole corpus, and one nobody has controlled either - and should you offer the webinar replay, which is the most-stated reason people skip. Both of those pages end by telling you to test it on your own list; this is the page that tells you how. The operational side of what we build sits on AI webinar attendance, the evidence review on webinar reminder calls, and the after-the-event work on post-webinar follow-up calls. If you are in the EU or the UK, check what your registration form has to say before anyone may call those registrants on webinar registrant consent, and if you would rather work through the design with us, our contact page books a scoped conversation.

Frequently Asked Questions