Free Tool

A/B Test Sample Size Calculator

Calculate how many visitors you need for statistically significant A/B test results. Never run underpowered tests again.

%

Your current conversion rate (control variant)

%

Smallest improvement you want to detect (e.g., 0.5% absolute increase)

How confident you want to be in your results

Probability of detecting an effect if it exists

What Is Sample Size?

Sample size is the number of visitors or observations needed in each variant (control and treatment) of your A/B test to achieve statistical significance. It's a critical calculation that determines whether your test results are reliable or just due to random chance.

Without proper sample size, you risk two costly mistakes: (1) declaring a "winning" variant that's actually just lucky variation (Type I error), or (2) missing a real improvement because you didn't collect enough data (Type II error). The correct sample size balances these risks based on your specific test parameters.

Why Sample Size Matters for A/B Tests

  • Avoid False Positives: Adequate sample size reduces the risk of declaring a winner when the difference is just random fluctuation. At 95% confidence, you accept only a 5% chance of being wrong.
  • Detect Real Improvements: Proper power (typically 80%) ensures you have a high probability of catching real improvements when they exist, avoiding missed opportunities.
  • Business Confidence: Statistically significant results give you confidence to implement changes with real revenue impact, rather than relying on gut feeling.
  • Efficient Resource Allocation: Knowing your sample size upfront prevents wasting time and resources running tests longer than needed or stopping early out of impatience.

Understanding Statistical Significance

Statistical significance answers the question: "Is this result real, or just random chance?" A result is statistically significant when the probability of observing it by chance alone (if there were truly no difference) is very small - typically less than 5%.

  • P-Value:The p-value is the probability of seeing your results if there's actually no difference. A p-value of 0.03 (3%) means there's only a 3% chance this difference occurred randomly. With 95% confidence, you need p < 0.05.
  • Confidence Interval: A 95% confidence interval around your observed effect means if you repeated the test 100 times, the true effect would fall within this range 95 times. Wider intervals indicate less certainty; narrower ones indicate more precision.
  • Statistical vs. Practical Significance: A result can be statistically significant (real difference exists) but practically insignificant (too small to matter). Conversely, a practically significant result might not reach statistical significance without enough data. Both matter for good business decisions.

Confidence Levels and Statistical Power

Confidence Level (Type I Error Protection)

  • 90% - 10% false positive risk: More lenient, detects effects with fewer visitors. Use when speed matters and you can tolerate higher false positive risk.
  • 95% - 5% false positive risk: Industry standard. Recommended for most A/B tests. Provides good balance between safety and efficiency.
  • 99% - 1% false positive risk: Very strict, requires more visitors. Use only when false positives are extremely costly.

Statistical Power (Type II Error Protection)

  • 80% - 20% false negative risk: Standard in A/B testing. Recommended for most tests. Good balance of detecting real effects while managing sample size.
  • 90% - 10% false negative risk: More strict, requires more visitors. Use when missing improvements is very costly.

Minimum Detectable Effect (MDE)

Minimum detectable effect is the smallest change in conversion rate (or other metric) that your test is powered to detect reliably. It's expressed as an absolute percentage point change.

  • 0.1% MDE: 2.5% → 2.6% conversion - Very large sample size (hundreds of thousands)
  • 0.5% MDE: 2.5% → 3.0% conversion - Large sample size (tens of thousands)
  • 1.0% MDE: 2.5% → 3.5% conversion - Moderate sample size (thousands)
  • 5.0% MDE: 2.5% → 7.5% conversion - Small sample size (hundreds)

Smaller MDEs require exponentially larger sample sizes. Define your MDE based on business impact: what's the smallest improvement worth implementing? If a 0.5% improvement would save €100K/year, that's worth detecting; if a 0.1% improvement would only save €5K, you might not need to detect it.

How to Use This Calculator

  1. Enter Your Baseline Conversion Rate: Get this from your analytics (Google Analytics, BuddyStat, etc.). If 100 visitors and 3 convert, your baseline is 3%. Use your most recent representative data.
  2. Define Your Minimum Detectable Effect: What's the smallest improvement worth detecting? If your baseline is 2%, a 0.5% MDE means you want to reliably detect moving to 2.5% or higher. Base it on business impact rather than percentage points alone.
  3. Choose Your Confidence Level (Default: 95%): 95% is standard and recommended. This means you accept a 5% chance that your result is a false positive. Only use 90% if time is critical, or 99% if false positives are extremely costly.
  4. Select Your Statistical Power (Default: 80%): 80% power is standard, meaning you accept a 20% chance of missing a real improvement. Use 90% if missing improvements is very costly (requires larger sample size).
  5. Calculate and Plan Your Test: The calculator shows the required sample size per variant, total visitors needed, and estimated test duration. Use BuddyStat Analytics to monitor progress and track when you reach statistical significance.

Frequently Asked Questions

Sample size is the number of visitors needed in each test variant (control and treatment) to reach statistical significance. It's calculated based on your baseline conversion rate, the minimum detectable effect you want to find, your desired confidence level, and statistical power. Proper sample size ensures your A/B test results are reliable and not due to random chance.

Minimum detectable effect (MDE) defines the smallest improvement you want to reliably detect. Smaller MDEs require exponentially larger sample sizes. For example, detecting a 0.5% improvement might require 10 times more visitors than detecting a 5% improvement. Set your MDE based on business impact: what's the smallest improvement worth implementing? This helps balance statistical rigor with practical feasibility.

Statistical significance means the difference between your variants is unlikely to be due to random chance. At 95% confidence, a statistically significant result means there's less than a 5% probability the difference occurred randomly. Without sufficient sample size, you might see a "winning" variant that's just lucky variation. Reaching calculated sample size is critical - don't stop tests early even if you see early winners.

Confidence level (typically 95%) measures Type I error - the risk of claiming a difference exists when it doesn't (false positive). At 95% confidence, you accept a 5% chance of being wrong. Statistical power (typically 80%) measures Type II error - the risk of missing a real difference when it exists (false negative). At 80% power, you have a 20% chance of missing a real effect. Both matter for reliable A/B testing.

Run your test until you reach the calculated sample size, even if you see apparent "winners" mid-test. Stopping early increases false positive risk. The duration depends on your daily visitor volume - if you need 10,000 visitors and get 1,000/day, run it about 10 days. Use BuddyStat Analytics to track progress and know when you reach statistical significance.

Your baseline is your current control variant's conversion rate. Get this from your analytics tools. If 2.5% of visitors currently convert, use 2.5%. Use your most recent representative data. Seasonal variations and traffic changes can affect your baseline, so update it if conditions change. More accurate baselines lead to more realistic sample size calculations and test planning.

Related tools

Keep working with a nearby calculator or utility.

View all tools

Run better A/B tests with BuddyStat

Track conversions, variants, and statistical significance in real-time. Know exactly when you've reached statistical significance and avoid costly false positives. Get started for free with up to 3,000 pageviews per month.

Start tracking for free