A/B test design, significance, power, metrics selection, cohort and funnel analysis.
40 cards · basic cards · AI-written, checked twice. Edit anything.
- What is an A/B test?
Show answer
A controlled experiment comparing two variants to determine which performs better.- Define statistical significance.
Show answer
The probability that an observed difference between groups is real and not due to random chance.- What is a p-value?
Show answer
The probability of observing a test statistic as extreme or more extreme, given the null hypothesis is true.- What does p < 0.05 mean?
Show answer
Less than 5 percent probability the observed difference occurred by random chance; the standard significance threshold.- Define Type I error.
Show answer
Rejecting the null hypothesis when it is true, resulting in a false positive.- Define Type II error.
Show answer
Failing to reject the null hypothesis when it is false, resulting in a false negative.- What is statistical power?
Show answer
The probability of correctly rejecting the null hypothesis when the alternative hypothesis is true.- Express the relationship between Type II error and power.
Show answer
Power equals 1 minus the probability of Type II error.- Name four factors affecting statistical power.
Show answer
Sample size, effect size, significance level, and variance.- Why do A/B tests use randomization?
Show answer
To ensure treatment and control groups are equivalent on average, isolating the effect of the treatment.- What is a control group in A/B testing?
Show answer
The group that does not receive the treatment, used as a baseline for comparison.- Define a cohort.
Show answer
A group of users or items sharing a common characteristic or experience within a specific time period.- What is cohort analysis?
Show answer
Analyzing behavior of cohorts over time to understand user lifecycle and retention patterns.- Define a funnel.
Show answer
A series of sequential steps users take toward a goal, such as sign up, activate, and purchase.- What is funnel analysis?
Show answer
Examining drop-off rates at each step of a user journey to identify optimization opportunities.
FlashKeepers