← Back to all products
$29
A/B Test Framework
Statistical A/B testing framework with experiment setup, significance testing, and result analysis.
PythonMarkdownJSON
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 15 files
ab-test-framework/
├── LICENSE
├── README.md
├── examples/
│ └── sample_experiment.json
├── free-sample.zip
├── guide/
│ ├── 01_features.md
│ ├── 02_quick-start.md
│ ├── 03_output.md
│ └── 04_faq.md
├── index.html
├── interactive.html
├── src/
│ ├── __pycache__/
│ │ └── ab_test_framework.cpython-312.pyc
│ └── ab_test_framework.py
└── tests/
├── __pycache__/
│ └── test_ab_test_framework.cpython-312-pytest-9.0.2.pyc
└── test_ab_test_framework.py
Documentation Preview README excerpt
A/B Test Framework
Statistical A/B testing framework: experiment setup, significance testing, sample size calculation, and result analysis. Know when a winner is real, not noise.
Features
- Z-test for proportions — Industry-standard two-proportion z-test for conversion rate comparisons
- Chi-squared test — Alternative significance test (2×2 contingency table)
- Sample size calculator — Know how many visitors you need before starting
- Confidence levels — 90%, 95%, 99% with correct z-score thresholds
- Simulation mode — Generate synthetic experiments to test your setup
- Relative & absolute lift — Both metrics reported for full context
- Pretty terminal output — Clear, formatted results with winner declaration
- JSON export — Machine-readable results for automation pipelines
Requirements
- Python 3.10+
- No external dependencies (stdlib only)
Quick Start
# Analyze experiment results from a JSON file
python src/ab_test_framework.py analyze --results examples/sample_experiment.json
# Analyze with 99% confidence level
python src/ab_test_framework.py analyze --results examples/sample_experiment.json --confidence 0.99
# Simulate an experiment (5000 visitors, 3.2% vs 4.1% conversion)
python src/ab_test_framework.py simulate --visitors 5000 --rate-a 0.032 --rate-b 0.041
# Calculate minimum sample size
python src/ab_test_framework.py sample-size --baseline 0.05 --mde 0.10
# Save results to JSON
python src/ab_test_framework.py analyze --results data.json --output report.json
Input Format
JSON
{
"control": {
"name": "Original Checkout",
"visitors": 5000,
"conversions": 160
},
"treatment": {
"name": "New Checkout",
"visitors": 5000,
"conversions": 205
}
}
CSV
... continues with setup instructions, usage examples, and more.
Code Sample .py preview
src/ab_test_framework.py
#!/usr/bin/env python3
"""
A/B Test Framework — Analytics Hub (DataNest)
Statistical A/B testing framework: experiment setup, visitor assignment,
chi-squared and z-test significance analysis. Know when a winner is real,
not noise.
Usage:
python ab_test_framework.py --results experiment.json --analyze
python ab_test_framework.py --results experiment.csv --confidence 0.99
python ab_test_framework.py --simulate --visitors 5000 --rate-a 0.032 --rate-b 0.041
Dependencies: Python 3.10+ stdlib only (no pip packages)
License: MIT
"""
from __future__ import annotations
import argparse
import csv
import hashlib
import json
import logging
import math
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
logger = logging.getLogger("ab_test_framework")
# ---------------------------------------------------------------------------
# Constants — common significance thresholds
# ---------------------------------------------------------------------------
# Standard confidence levels and their z-scores for two-tailed tests.
# 90% is exploratory, 95% is the industry standard, 99% is conservative.
CONFIDENCE_Z: dict[float, float] = {
0.90: 1.645,
0.95: 1.960,
0.99: 2.576,
}
# ---------------------------------------------------------------------------
# Data models
# ---------------------------------------------------------------------------
@dataclass
class Variant:
"""One arm of an A/B experiment (control or treatment)."""
# ... 492 more lines ...