← Back to all products
$39
A/B Testing Statistical Framework
Complete A/B testing toolkit: sample size calculator, statistical significance tests, Bayesian analysis, and results reporting templates.
PythonYAMLMarkdownJSON
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 39 files
ab-testing-framework/
├── LICENSE
├── README.md
├── configs/
│ └── experiment_config.yaml
├── free-sample.zip
├── guide/
│ ├── experiment_design.md
│ └── interpreting_results.md
├── guides/
│ ├── experiment_design.md
│ └── interpreting_results.md
├── index.html
├── interactive.html
├── requirements.txt
├── src/
│ └── abtest/
│ ├── __init__.py
│ ├── __pycache__/
│ │ ├── __init__.cpython-312.pyc
│ │ ├── bayesian.cpython-312.pyc
│ │ ├── cuped.cpython-312.pyc
│ │ ├── frequentist.cpython-312.pyc
│ │ ├── main.cpython-312.pyc
│ │ ├── reporter.cpython-312.pyc
│ │ ├── sample_size.cpython-312.pyc
│ │ └── sequential.cpython-312.pyc
│ ├── bayesian.py
│ ├── cuped.py
│ ├── frequentist.py
│ ├── main.py
│ ├── reporter.py
│ ├── sample_size.py
│ └── sequential.py
└── tests/
├── __init__.py
├── __pycache__/
│ ├── __init__.cpython-312.pyc
│ ├── test_bayesian.cpython-312-pytest-9.0.2.pyc
│ ├── test_bayesian.cpython-312.pyc
│ ├── test_frequentist.cpython-312-pytest-9.0.2.pyc
│ ├── test_frequentist.cpython-312.pyc
│ ├── test_sample_size.cpython-312-pytest-9.0.2.pyc
│ └── test_sample_size.cpython-312.pyc
├── test_bayesian.py
├── test_frequentist.py
└── test_sample_size.py
Documentation Preview README excerpt
A/B Testing Statistical Framework
A complete A/B testing toolkit implementing real statistical methods from scratch. Includes sample size calculation, frequentist tests, sequential testing, Bayesian analysis, and CUPED variance reduction — all runnable with Python's standard library + scipy.
Features
- Sample Size Calculator — Power analysis for proportions and continuous metrics
- Frequentist Tests — Z-test for proportions, Welch's t-test, chi-square goodness-of-fit
- Sequential Testing — O'Brien-Fleming spending function for early stopping
- Bayesian A/B — Beta-Binomial model with posterior probability of winning
- CUPED Variance Reduction — Pre-experiment covariate adjustment to reduce required sample size
- Results Reporter — Generate structured experiment summaries with confidence intervals
Quick Start
pip install -r requirements.txt
python -m src.abtest.main
Structure
src/abtest/
├── __init__.py # Package exports
├── main.py # Demo runner
├── sample_size.py # Power analysis and sample size calculation
├── frequentist.py # Z-test, t-test, chi-square implementations
├── sequential.py # Group sequential testing with alpha spending
├── bayesian.py # Beta-Binomial Bayesian A/B testing
├── cuped.py # CUPED variance reduction
├── reporter.py # Structured results reporting
tests/
├── test_sample_size.py # Validates power calculations against known values
├── test_frequentist.py # Validates test statistics
├── test_bayesian.py # Validates posterior calculations
configs/
├── experiment_config.yaml # Example experiment configuration
guides/
├── experiment_design.md # How to design a rigorous experiment
├── interpreting_results.md # How to read and act on test results
Usage Examples
Calculate Required Sample Size
from src.abtest.sample_size import required_sample_size_proportions
n = required_sample_size_proportions(
baseline_rate=0.12, # Current conversion rate: 12%
minimum_detectable_effect=0.02, # Want to detect 2pp lift (to 14%)
alpha=0.05, # 5% significance level
power=0.80, # 80% power
)
print(f"Need {n:,} users per variant")
Run a Frequentist Test
... continues with setup instructions, usage examples, and more.
Code Sample .py preview
src/abtest/bayesian.py
"""
Bayesian A/B testing using the Beta-Binomial conjugate model.
Computes posterior distributions for conversion rates and the
probability that one variant beats another. No MCMC needed —
the Beta-Binomial model has a closed-form posterior.
"""
from __future__ import annotations
import math
from dataclasses import dataclass
@dataclass
class BayesianResult:
"""Result of a Bayesian A/B test."""
control_posterior_alpha: float
control_posterior_beta: float
treatment_posterior_alpha: float
treatment_posterior_beta: float
control_posterior_mean: float
treatment_posterior_mean: float
prob_treatment_wins: float
expected_lift: float
credible_interval_lift: tuple[float, float]
risk_of_choosing_treatment: float
interpretation: str
def bayesian_ab_test(
control_successes: int,
control_total: int,
treatment_successes: int,
treatment_total: int,
prior_alpha: float = 1.0,
prior_beta: float = 1.0,
num_simulations: int = 100_000,
) -> BayesianResult:
"""Run a Bayesian A/B test using Beta-Binomial model.
The Beta-Binomial is the conjugate prior for binomial data:
- Prior: Beta(alpha, beta) — default is uniform Beta(1,1)
- Posterior: Beta(alpha + successes, beta + failures)
The probability that treatment > control is computed via
Monte Carlo sampling from both posteriors.
Args:
control_successes: Conversions in control.
# ... 149 more lines ...