Getting Started

Prerequisites

• Python 3.10+
• pip

Installation

pip install -r requirements.txt

Your First Pipeline Run

Run the included classification pipeline:

python -m src.pipeline_runner --config pipelines/classification_pipeline.yaml --verbose

This will:

1. Generate a demo dataset (2000 rows with age, income, region, etc.)

2. Apply feature engineering (scaling, encoding, imputation)

3. Train a Random Forest classifier

4. Evaluate with accuracy, F1, and AUC threshold gates

5. Attempt a canary deployment (dry-run — no actual infrastructure)

6. Save all artifacts to ./artifacts//

Inspect the Output

After the run completes, check:

• ./artifacts//run_summary.json — full run metadata
• ./artifacts//snapshot.json — reproducibility snapshot

Customize for Your Data

1. Copy pipelines/classification_pipeline.yaml to pipelines/my_pipeline.yaml

2. Update the feature_engineering stage to match your columns

3. Adjust the training hyperparameters

4. Set meaningful threshold_gates in the evaluation stage

5. Configure your deployment endpoint in the deployment stage

Adding Custom Stages

Create a function with this signature and register it:

from src.pipeline_runner import register_stage

def my_custom_stage(params: dict, context: dict) -> dict:
    # Do your work here
    result = {"my_output": "some_value"}
    return result

register_stage("my_custom_stage", my_custom_stage)

Then reference it in your pipeline YAML:

stages:
  - name: my_custom_stage
    depends_on: [feature_engineering]
    params:
      my_param: my_value