cursus.steps.configs.config_dummy_data_loading_step

Dummy Data Loading Step Configuration

This module implements the configuration class for the Dummy Data Loading step, which processes user-provided data instead of calling internal Cradle services.

class DummyDataLoadingConfig(*, author, bucket, role, region, service_name, pipeline_version, model_class='xgboost', current_date=<factory>, framework_version='2.1.0', py_version='py310', source_dir=None, enable_caching=False, use_secure_pypi=False, max_runtime_seconds=172800, project_root_folder, processing_instance_count=1, processing_volume_size=500, processing_instance_type_large='ml.m5.4xlarge', processing_instance_type_small='ml.m5.2xlarge', use_large_processing_instance=False, skip_volume_kms=None, processing_source_dir=None, processing_entry_point='dummy_data_loading.py', processing_script_arguments=None, processing_framework_version='1.2-1', data_source, job_type='training', max_file_size_mb=1000, supported_formats=['csv', 'parquet', 'json', 'jsonl'], write_data_shards=False, shard_size=10000, output_format='CSV', max_workers=0, batch_size=5, optimize_memory=False, streaming_batch_size=0, enable_true_streaming=False, metadata_format='JSON', **extra_data)

Bases: ProcessingStepConfigBase

Configuration for a dummy data loading step.

This configuration follows the three-tier field categorization: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that users can override 3. Tier 3: Derived Fields - fields calculated from other fields, stored in private attributes

data_source: str | Path

processing_entry_point: str

job_type: str

max_file_size_mb: int

supported_formats: list[str]

write_data_shards: bool

shard_size: int

output_format: str

max_workers: int

batch_size: int

optimize_memory: bool

streaming_batch_size: int

enable_true_streaming: bool

metadata_format: str

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'protected_namespaces': (), 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod validate_job_type(v)

Ensure job_type is one of the allowed values.

classmethod validate_output_format(v)

Ensure output_format is one of the allowed values (case-insensitive).

Matching is case-insensitive and the stored value is normalized to the canonical uppercase allowed value.

classmethod validate_metadata_format(v)

Ensure metadata_format is one of the allowed values (case-insensitive).

Matching is case-insensitive and the stored value is normalized to the canonical uppercase allowed value.

validate_config()

Validate configuration and ensure defaults are set.

This validator ensures that: 1. Data source is provided and properly formatted 2. Entry point is provided 3. Script contract is available and valid 4. Required input/output paths are defined in the script contract

is_s3_source()

Check if the data source is an S3 URI.

Returns:

True if data source is S3, False if local path

Return type:

bool

get_data_source_uri()

Get the data source as a string URI.

Returns:

Data source URI (S3 or local path)

Return type:

str

get_supported_extensions()

Get supported file extensions based on configured formats.

Returns:

Set of supported file extensions (with dots)

Return type:

set[str]

get_environment_variables()

Get environment variables for the processing script.

Returns:

Dictionary of environment variables to be passed to the processing script.

Return type:

dict

model_dump(**kwargs)

Override model_dump to include derived properties.

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

processing_instance_count: int

processing_volume_size: int

processing_instance_type_large: str

processing_instance_type_small: str

use_large_processing_instance: bool

skip_volume_kms: bool | None

processing_source_dir: str | None

processing_script_arguments: List[str] | None

processing_framework_version: str

author: str

bucket: str

role: str

region: str

service_name: str

pipeline_version: str

model_class: str

current_date: str

framework_version: str

py_version: str

source_dir: str | None

enable_caching: bool

use_secure_pypi: bool

max_runtime_seconds: int

project_root_folder: str