Hypothesis setup, sample size and power, p-values, confidence intervals, novelty effects, multiple testing and Bayesian approaches.
40 cards · basic cards · AI-written, checked twice. Edit anything.
- What is the null hypothesis in an A/B test?
Show answer
The statement that there is no difference between the control and treatment groups.- What is the alternative hypothesis?
Show answer
The statement that there is a difference between the control and treatment groups.- What does 'one-tailed test' mean in A/B testing?
Show answer
A hypothesis test that specifies the direction of the effect (e.g., treatment is better, not just different).- What does 'two-tailed test' mean in A/B testing?
Show answer
A hypothesis test that does not specify direction, testing only whether groups differ.- Define statistical power in an experiment.
Show answer
The probability of detecting a real effect if one exists (1 minus beta).- What is the relationship between alpha and Type I error?
Show answer
Alpha is the significance level and equals the probability of a Type I error.- What is beta in statistical testing?
Show answer
The probability of a Type II error (failing to reject a false null hypothesis).- What does a p-value of 0.03 mean?
Show answer
If the null hypothesis is true, there is a 3% probability of observing data this extreme or more extreme by chance.- What is a common significance level used in A/B testing?
Show answer
0.05 (5%), meaning a 5% acceptable Type I error rate.- How does sample size affect statistical power?
Show answer
Larger sample size increases power, holding effect size and alpha constant.- Define effect size in A/B testing.
Show answer
The magnitude of the difference between the control and treatment groups.- What is a Type I error?
Show answer
Rejecting the null hypothesis when it is actually true (false positive).- What is a Type II error?
Show answer
Failing to reject the null hypothesis when it is actually false (false negative).- What does statistically significant at 0.05 mean?
Show answer
The p-value is less than 0.05, so we reject the null hypothesis at the 5% significance level.- Define confidence interval.
Show answer
A range of values that likely contains the true population parameter with a specified probability.
FlashKeepers