cursus.steps.configs.config_redshift_data_loading_step¶

Redshift Data Loading Step Configuration.

Flattens the SAIS SDK’s 3-specification JSON structure (clusterSpecification, querySpecification, outputSpecification) into Pydantic fields. The computed property step_config_json reassembles the 3-spec dict for the container script.

class RedshiftDataLoadingConfig(*, author, bucket, role, region, service_name, pipeline_version, model_class='xgboost', current_date=<factory>, framework_version='2.1.0', py_version='py310', source_dir=None, enable_caching=False, use_secure_pypi=False, max_runtime_seconds=172800, project_root_folder, cluster_id, db_name, role_arn, cluster_endpoint, query, connector_type='redshift_connector', port=5439, is_using_andes3=True, output_data_source_type='S3', edx_arn=None, drop_header=False, job_type=None, max_runtime_in_seconds=43200, **extra_data)[source]¶

Bases: BasePipelineConfig

Configuration for RedshiftDataLoading step.

Source node — executes SQL against Redshift, writes CSV to S3. Optionally uploads to EDX as side effect.

cluster_id: str¶

db_name: str¶

role_arn: str¶

cluster_endpoint: str¶

query: str¶

connector_type: str¶

port: int¶

is_using_andes3: bool¶

output_data_source_type: str¶

edx_arn: str | None¶

drop_header: bool¶

job_type: str | None¶

max_runtime_in_seconds: int¶

validate_edx_config()[source]¶

Validate EDX ARN is provided when output type is EDX.

property step_config_json: Dict[str, Any]¶: Generate the 3-spec JSON config that the container script reads from /opt/ml/processing/config/config.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'protected_namespaces': (), 'validate_assignment': True}¶: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)¶

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

self (BaseModel) – The BaseModel instance.
context (Any) – The context.