← Back to all products

A/B Testing Statistical Framework

$39

Complete A/B testing toolkit: sample size calculator, statistical significance tests, Bayesian analysis, and results reporting templates.

39 files
PythonYAMLMarkdownJSON

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 39 files

ab-testing-framework/ ├── LICENSE ├── README.md ├── configs/ │ └── experiment_config.yaml ├── free-sample.zip ├── guide/ │ ├── experiment_design.md │ └── interpreting_results.md ├── guides/ │ ├── experiment_design.md │ └── interpreting_results.md ├── index.html ├── interactive.html ├── requirements.txt ├── src/ │ └── abtest/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── bayesian.cpython-312.pyc │ │ ├── cuped.cpython-312.pyc │ │ ├── frequentist.cpython-312.pyc │ │ ├── main.cpython-312.pyc │ │ ├── reporter.cpython-312.pyc │ │ ├── sample_size.cpython-312.pyc │ │ └── sequential.cpython-312.pyc │ ├── bayesian.py │ ├── cuped.py │ ├── frequentist.py │ ├── main.py │ ├── reporter.py │ ├── sample_size.py │ └── sequential.py └── tests/ ├── __init__.py ├── __pycache__/ │ ├── __init__.cpython-312.pyc │ ├── test_bayesian.cpython-312-pytest-9.0.2.pyc │ ├── test_bayesian.cpython-312.pyc │ ├── test_frequentist.cpython-312-pytest-9.0.2.pyc │ ├── test_frequentist.cpython-312.pyc │ ├── test_sample_size.cpython-312-pytest-9.0.2.pyc │ └── test_sample_size.cpython-312.pyc ├── test_bayesian.py ├── test_frequentist.py └── test_sample_size.py

Documentation Preview README excerpt

A/B Testing Statistical Framework

A complete A/B testing toolkit implementing real statistical methods from scratch. Includes sample size calculation, frequentist tests, sequential testing, Bayesian analysis, and CUPED variance reduction — all runnable with Python's standard library + scipy.

Features

  • Sample Size Calculator — Power analysis for proportions and continuous metrics
  • Frequentist Tests — Z-test for proportions, Welch's t-test, chi-square goodness-of-fit
  • Sequential Testing — O'Brien-Fleming spending function for early stopping
  • Bayesian A/B — Beta-Binomial model with posterior probability of winning
  • CUPED Variance Reduction — Pre-experiment covariate adjustment to reduce required sample size
  • Results Reporter — Generate structured experiment summaries with confidence intervals

Quick Start


pip install -r requirements.txt
python -m src.abtest.main

Structure


src/abtest/
├── __init__.py           # Package exports
├── main.py               # Demo runner
├── sample_size.py        # Power analysis and sample size calculation
├── frequentist.py        # Z-test, t-test, chi-square implementations
├── sequential.py         # Group sequential testing with alpha spending
├── bayesian.py           # Beta-Binomial Bayesian A/B testing
├── cuped.py              # CUPED variance reduction
├── reporter.py           # Structured results reporting
tests/
├── test_sample_size.py   # Validates power calculations against known values
├── test_frequentist.py   # Validates test statistics
├── test_bayesian.py      # Validates posterior calculations
configs/
├── experiment_config.yaml  # Example experiment configuration
guides/
├── experiment_design.md    # How to design a rigorous experiment
├── interpreting_results.md # How to read and act on test results

Usage Examples

Calculate Required Sample Size


from src.abtest.sample_size import required_sample_size_proportions

n = required_sample_size_proportions(
    baseline_rate=0.12,          # Current conversion rate: 12%
    minimum_detectable_effect=0.02,  # Want to detect 2pp lift (to 14%)
    alpha=0.05,                  # 5% significance level
    power=0.80,                  # 80% power
)
print(f"Need {n:,} users per variant")

Run a Frequentist Test

... continues with setup instructions, usage examples, and more.

Code Sample .py preview

src/abtest/bayesian.py """ Bayesian A/B testing using the Beta-Binomial conjugate model. Computes posterior distributions for conversion rates and the probability that one variant beats another. No MCMC needed — the Beta-Binomial model has a closed-form posterior. """ from __future__ import annotations import math from dataclasses import dataclass @dataclass class BayesianResult: """Result of a Bayesian A/B test.""" control_posterior_alpha: float control_posterior_beta: float treatment_posterior_alpha: float treatment_posterior_beta: float control_posterior_mean: float treatment_posterior_mean: float prob_treatment_wins: float expected_lift: float credible_interval_lift: tuple[float, float] risk_of_choosing_treatment: float interpretation: str def bayesian_ab_test( control_successes: int, control_total: int, treatment_successes: int, treatment_total: int, prior_alpha: float = 1.0, prior_beta: float = 1.0, num_simulations: int = 100_000, ) -> BayesianResult: """Run a Bayesian A/B test using Beta-Binomial model. The Beta-Binomial is the conjugate prior for binomial data: - Prior: Beta(alpha, beta) — default is uniform Beta(1,1) - Posterior: Beta(alpha + successes, beta + failures) The probability that treatment > control is computed via Monte Carlo sampling from both posteriors. Args: control_successes: Conversions in control. # ... 149 more lines ...