ML Pipeline Templates
End-to-end ML pipelines: data ingestion, preprocessing, training, evaluation, and deployment orchestration.
Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
Interactive Reader (Free Preview) ⚙ Try Demo Tools Download Free SampleFile Structure 18 files
Documentation Preview README excerpt
ML Pipeline Templates
End-to-end, dependency-light templates for building tabular machine-learning
pipelines: data ingestion -> preprocessing -> training -> evaluation, chained
together by a small DAG orchestrator you can actually read.
The orchestration core is pure standard library, so it is fast to import and
trivial to test. The ML stages use pandas/numpy and prefer scikit-learn when
it is installed -- but every stage has a numpy fallback, so the whole pipeline
(and the test suite) runs end-to-end even with nothing but pandas + numpy.
Features
- DAG orchestrator (
Pipeline/Step) -- declare dependencies, get
deterministic topological execution, per-step timing/status, cycle and
missing-dependency detection, and fail_fast handling. No third-party deps.
- Ingestion into one
Datasetinterface from CSV, Parquet, or SQL (or an
in-memory DataFrame), so downstream stages never touch a file path.
- Preprocessing built on a scikit-learn
ColumnTransformer(impute + scale +
one-hot encode), with a dependency-free PandasFeaturePipeline fallback and a
leakage-safe train/test split (optionally stratified).
- Training with backend-agnostic K-fold cross-validation that works with any
fit/predict estimator -- scikit-learn models or the bundled numpy
baselines (RidgeRegressor, MeanRegressor, MajorityClassifier).
- Evaluation with pure-numpy regression and classification metrics plus a
renderable EvaluationReport (Markdown, including a confusion matrix).
- Config-driven -- build the standard four-stage pipeline from
configs/pipeline.yaml; retarget your data without editing Python.
- Tested -- a pytest suite covers DAG ordering, context passing, error
handling, and the preprocessing transforms.
Requirements
- Python 3.10+
pandas,numpy,PyYAML(required by the ML stages)scikit-learn(recommended; auto-used when present),pyarrow(for Parquet) --
both optional
pip install -r requirements.txt
Quick start
Everything runs straight from the product folder -- the examples and tests add
src/ to the path, so no install step is needed.
# 1) Run the full pipeline on a generated toy dataset
python examples/run_pipeline.py
# 2) Run it from the YAML config instead of an inline config
python examples/run_pipeline.py --config configs/pipeline.yaml
# 3) See the scikit-learn ColumnTransformer + RandomForest path
python examples/sklearn_pipeline.py # prints an install hint if sklearn is absent
# 4) Run the test suite
pip install pytest
python -m pytest tests/ -v
*... continues with setup instructions, usage examples, and more.*