cursus.pipeline_catalog.core.builders

Common Pipeline Builders

Generates MODS-compatible pipeline classes from declarative config. Eliminates the per-project boilerplate (97-285 lines → 1 function call).

Usage:

# Generate a MODS pipeline class: MungedAddressPipelineNA = build_mods_pipeline(

author=”bjjin”, version=”0.0.5”, description=”Munged Address Detection DistilBERT Training Pipeline”, dag_path=”pipeline_config/dag_NA.json”, config_path=”pipeline_config/config_NA.json”,

)

# Or build and run directly (SAIS notebook): pipeline, report = build_and_compile(

dag_path=”pipeline_config/dag_training_NA.json”, config_path=”pipeline_config/config_training_NA.json”, sagemaker_session=pipeline_session, role=role,

)

MODSTemplate(author, description, version)[source]
build_and_compile(dag_path, config_path, sagemaker_session=None, role=None, project_root=None)[source]

Build and compile a pipeline from DAG + config paths. No class needed.

Parameters:

  • dag_path (str) – Path to .dag.json file

  • config_path (str) – Path to config JSON

  • sagemaker_session (Session | None) – SageMaker session

  • role (str | None) – IAM role ARN

  • project_root (str | Path | None) – Optional explicit project folder for docker source_dir resolution (the caller hook). When omitted, the compiler infers it from config_path.

Returns:

(Pipeline, CompilationReport)

Return type:

Tuple[Pipeline, any]

build_mods_pipeline(author, version, description, dag_path, config_path, class_name=None)[source]

Generate a @MODSTemplate-decorated pipeline class from declarative config.

The generated class has the standard MODS interface: - __init__(sagemaker_session, execution_role, regional_alias) - generate_pipeline() -> Pipeline

Parameters:

  • author (str) – Pipeline author alias

  • version (str) – Pipeline version string

  • description (str) – Pipeline description

  • dag_path (str) – Relative path to DAG JSON (relative to the calling module’s directory)

  • config_path (str) – Relative path to config JSON

  • class_name (str | None) – Optional class name (default: derived from description)

Returns:

A @MODSTemplate decorated class ready for MODS Lambda

Return type:

Type