← Back to all products

A/B Test Framework

$29

Statistical A/B testing framework with experiment setup, significance testing, and result analysis.

15 files
PythonMarkdownJSON

Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free Sample

File Structure 15 files

ab-test-framework/ ├── LICENSE ├── README.md ├── examples/ │ └── sample_experiment.json ├── free-sample.zip ├── guide/ │ ├── 01_features.md │ ├── 02_quick-start.md │ ├── 03_output.md │ └── 04_faq.md ├── index.html ├── interactive.html ├── src/ │ ├── __pycache__/ │ │ └── ab_test_framework.cpython-312.pyc │ └── ab_test_framework.py └── tests/ ├── __pycache__/ │ └── test_ab_test_framework.cpython-312-pytest-9.0.2.pyc └── test_ab_test_framework.py

Documentation Preview README excerpt

A/B Test Framework

Statistical A/B testing framework: experiment setup, significance testing, sample size calculation, and result analysis. Know when a winner is real, not noise.

Features

  • Z-test for proportions — Industry-standard two-proportion z-test for conversion rate comparisons
  • Chi-squared test — Alternative significance test (2×2 contingency table)
  • Sample size calculator — Know how many visitors you need before starting
  • Confidence levels — 90%, 95%, 99% with correct z-score thresholds
  • Simulation mode — Generate synthetic experiments to test your setup
  • Relative & absolute lift — Both metrics reported for full context
  • Pretty terminal output — Clear, formatted results with winner declaration
  • JSON export — Machine-readable results for automation pipelines

Requirements

  • Python 3.10+
  • No external dependencies (stdlib only)

Quick Start


# Analyze experiment results from a JSON file
python src/ab_test_framework.py analyze --results examples/sample_experiment.json

# Analyze with 99% confidence level
python src/ab_test_framework.py analyze --results examples/sample_experiment.json --confidence 0.99

# Simulate an experiment (5000 visitors, 3.2% vs 4.1% conversion)
python src/ab_test_framework.py simulate --visitors 5000 --rate-a 0.032 --rate-b 0.041

# Calculate minimum sample size
python src/ab_test_framework.py sample-size --baseline 0.05 --mde 0.10

# Save results to JSON
python src/ab_test_framework.py analyze --results data.json --output report.json

Input Format

JSON


{
    "control": {
        "name": "Original Checkout",
        "visitors": 5000,
        "conversions": 160
    },
    "treatment": {
        "name": "New Checkout",
        "visitors": 5000,
        "conversions": 205
    }
}

CSV

... continues with setup instructions, usage examples, and more.

Code Sample .py preview

src/ab_test_framework.py #!/usr/bin/env python3 """ A/B Test Framework — Analytics Hub (DataNest) Statistical A/B testing framework: experiment setup, visitor assignment, chi-squared and z-test significance analysis. Know when a winner is real, not noise. Usage: python ab_test_framework.py --results experiment.json --analyze python ab_test_framework.py --results experiment.csv --confidence 0.99 python ab_test_framework.py --simulate --visitors 5000 --rate-a 0.032 --rate-b 0.041 Dependencies: Python 3.10+ stdlib only (no pip packages) License: MIT """ from __future__ import annotations import argparse import csv import hashlib import json import logging import math import sys from dataclasses import dataclass, field from pathlib import Path from typing import Any logger = logging.getLogger("ab_test_framework") # --------------------------------------------------------------------------- # Constants — common significance thresholds # --------------------------------------------------------------------------- # Standard confidence levels and their z-scores for two-tailed tests. # 90% is exploratory, 95% is the industry standard, 99% is conservative. CONFIDENCE_Z: dict[float, float] = { 0.90: 1.645, 0.95: 1.960, 0.99: 2.576, } # --------------------------------------------------------------------------- # Data models # --------------------------------------------------------------------------- @dataclass class Variant: """One arm of an A/B experiment (control or treatment).""" # ... 492 more lines ...