cursus.steps.configs.config_cradle_data_loading_step

get_flattened_fields(config_obj, prefix='')

Recursively gather all fields from a config object and its nested objects, flattening them into a single list with dot notation to indicate hierarchy.

Parameters:

• config_obj – Configuration object to analyze

• prefix – String prefix for nested fields (for recursive calls)

Returns:

Dict with keys ‘essential’, ‘system’, and ‘derived’ mapping to lists of field names

Return type:

Dict[str, List[str]]

class BaseCradleComponentConfig(**extra_data)

Bases: BaseModel

Base class for Cradle configuration components with three-tier field classification support.

Implements common functionality for categorizing fields and supporting inheritance.

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

categorize_fields()

Categorize all fields into three tiers: 1. Tier 1: Essential User Inputs - fields with no defaults (required) 2. Tier 2: System Inputs - fields with defaults (optional) 3. Tier 3: Derived Fields - properties that access private attributes

Returns:

Dict with keys ‘essential’, ‘system’, and ‘derived’ mapping to lists of field names

Return type:

Dict[str, List[str]]

get_public_init_fields()

Get fields suitable for initializing a child config. Only includes fields that should be passed to child class constructors.

Returns:

Dictionary of field names to values for child initialization

Return type:

Dict[str, Any]

class MdsDataSourceConfig(*, service_name, region, output_schema, org_id=0, use_hourly_edx_data_set=False, **extra_data)

Bases: BaseCradleComponentConfig

Configuration for MDS data source with three-tier field classification.

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

service_name: str

region: str

output_schema: List[Dict[str, Any]]

org_id: int

use_hourly_edx_data_set: bool

classmethod validate_region(v)

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class EdxDataSourceConfig(*, schema_overrides=None, edx_arn=None, edx_provider=None, edx_subject=None, edx_dataset=None, edx_manifest_key=None, **extra_data)

Bases: BaseCradleComponentConfig

Configuration for EDX data source with three-tier field classification.

Supports two input modes: 1. Direct ARN input: Provide edx_arn directly 2. Component-based input: Provide edx_provider, edx_subject, edx_dataset, edx_manifest_key

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

schema_overrides: List[Dict[str, Any]] | None

edx_arn: str | None

edx_provider: str | None

edx_subject: str | None

edx_dataset: str | None

edx_manifest_key: str | None

property edx_manifest: str

Get EDX manifest ARN from direct input or built from components.

categorize_fields()

Dynamic field categorization based on edx_arn presence.

classmethod validate_edx_arn_format(v)

Validate EDX ARN format if provided.

classmethod validate_manifest_key_format(v)

Validate that edx_manifest_key is in the format ‘[…]’ if provided.

validate_edx_input_mode()

Ensure either edx_arn OR all component fields are provided.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

class AndesDataSourceConfig(*, provider, table_name, andes3_enabled=True, **extra_data)

Bases: BaseCradleComponentConfig

Configuration for Andes data source with three-tier field classification.

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

provider: str

table_name: str

andes3_enabled: bool

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'str_strip_whitespace': True, 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod validate_provider(v)

Validate that the provider is either: 1. A valid 32-character UUID 2. The special case ‘booker’

classmethod validate_table_name(v)

Validate that the table name is not empty and follows valid format.

class DataSourceConfig(*, data_source_name, data_source_type, mds_data_source_properties=None, edx_data_source_properties=None, andes_data_source_properties=None, **extra_data)

Bases: BaseCradleComponentConfig

Configuration for data sources with three-tier field classification.

Corresponds to com.amazon.secureaisandboxproxyservice.models.datasource.DataSource

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

data_source_name: str

data_source_type: str

mds_data_source_properties: MdsDataSourceConfig | None

edx_data_source_properties: EdxDataSourceConfig | None

andes_data_source_properties: AndesDataSourceConfig | None

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'frozen': True, 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod validate_type(v)

classmethod check_properties(model)

Ensure the appropriate properties are set based on data_source_type and that only one set of properties is provided.

class DataSourcesSpecificationConfig(*, start_date, end_date, data_sources, **extra_data)

Bases: BaseCradleComponentConfig

Configuration for data sources specification with three-tier field classification.

Corresponds to com.amazon.secureaisandboxproxyservice.models.datasourcesspecification.DataSourcesSpecification

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

start_date: str

end_date: str

data_sources: List[DataSourceConfig]

classmethod validate_exact_datetime_format(v, field)

Must match exactly “%Y-%m-%dT%H:%M:%S”

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class JobSplitOptionsConfig(*, merge_sql=None, split_job=False, days_per_split=7, **extra_data)

Bases: BaseCradleComponentConfig

Corresponds to com.amazon.secureaisandboxproxyservice.models.jobsplitoptions.JobSplitOptions:

• split_job: bool

• days_per_split: int

• merge_sql: str

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

merge_sql: str | None

split_job: bool

days_per_split: int

classmethod days_must_be_positive(v)

classmethod require_merge_sql_if_split(model)

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class TransformSpecificationConfig(*, transform_sql, job_split_options=<factory>, **extra_data)

Bases: BaseCradleComponentConfig

Corresponds to com.amazon.secureaisandboxproxyservice.models.transformspecification.TransformSpecification:

• transform_sql: str

• job_split_options: JobSplitOptionsConfig

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

transform_sql: str

job_split_options: JobSplitOptionsConfig

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class OutputSpecificationConfig(*, output_schema, job_type='training', pipeline_s3_loc=None, output_format='PARQUET', output_save_mode='ERRORIFEXISTS', output_file_count=0, keep_dot_in_output_schema=False, include_header_in_s3_output=True, **extra_data)

Bases: BaseCradleComponentConfig

Corresponds to com.amazon.secureaisandboxproxyservice.models.outputspecification.OutputSpecification:

• output_schema: List[str]

• output_format: str (e.g. ‘PARQUET’, ‘CSV’, etc.)

• output_save_mode: str (e.g. ‘ERRORIFEXISTS’, ‘OVERWRITE’, ‘APPEND’, ‘IGNORE’)

• output_file_count: int (0 means “auto”)

• keep_dot_in_output_schema: bool

• include_header_in_s3_output: bool

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

output_schema: List[str]

job_type: str

pipeline_s3_loc: str | None

output_format: str

output_save_mode: str

output_file_count: int

keep_dot_in_output_schema: bool

include_header_in_s3_output: bool

property output_path: str

Get output path derived from pipeline_s3_loc and job_type.

validate_output_path()

Validate that output_path is a valid S3 URI.

classmethod validate_job_type(v)

classmethod validate_format(v)

classmethod validate_save_mode(v)

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

class CradleJobSpecificationConfig(*, cradle_account, cluster_type='STANDARD', extra_spark_job_arguments='', job_retry_count=1, **extra_data)

Bases: BaseCradleComponentConfig

Corresponds to com.amazon.secureaisandboxproxyservice.models.cradlejobspecification.CradleJobSpecification:

• cluster_type: str (e.g. ‘SMALL’, ‘MEDIUM’, ‘LARGE’)

• cradle_account: str

• extra_spark_job_arguments: Optional[str]

• job_retry_count: int

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

cradle_account: str

cluster_type: str

extra_spark_job_arguments: str | None

job_retry_count: int

classmethod validate_cluster_type(v)

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class CradleDataLoadingConfig(*, author, bucket, role, region, service_name, pipeline_version, model_class='xgboost', current_date=<factory>, framework_version='2.1.0', py_version='py310', source_dir=None, enable_caching=False, use_secure_pypi=False, max_runtime_seconds=172800, project_root_folder, job_type, data_sources_spec, transform_spec, output_spec, cradle_job_spec, s3_input_override=None, **extra_data)

Bases: BasePipelineConfig

Top‐level configuration for creating a CreateCradleDataLoadJobRequest with three-tier field classification.

Fields are organized into three tiers: 1. Tier 1: Essential User Inputs - fields that users must explicitly provide 2. Tier 2: System Inputs with Defaults - fields with reasonable defaults that can be overridden 3. Tier 3: Derived Fields - fields calculated from other fields (private attributes with properties)

This class inherits from BasePipelineConfig (not BaseCradleComponentConfig) to maintain consistency with other pipeline configurations.

job_type: str

data_sources_spec: DataSourcesSpecificationConfig

transform_spec: TransformSpecificationConfig

output_spec: OutputSpecificationConfig

cradle_job_spec: CradleJobSpecificationConfig

s3_input_override: str | None

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'protected_namespaces': (), 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

classmethod validate_job_type(v)

initialize_derived_fields()

Initialize all derived fields once after validation.

categorize_fields()

Categorize all fields into three tiers: 1. Tier 1: Essential User Inputs - fields with no defaults (required) 2. Tier 2: System Inputs - fields with defaults (optional) 3. Tier 3: Derived Fields - properties that access private attributes

Returns:

Dict with keys ‘essential’, ‘system’, and ‘derived’ mapping to lists of field names

Return type:

Dict[str, List[str]]

get_all_tiered_fields()

Get a flattened list of all fields (including nested fields) organized by tier.

Returns:

Dict with keys ‘essential’, ‘system’, and ‘derived’ mapping to lists of field names with dot notation for nesting

Return type:

Dict[str, List[str]]

check_split_and_override()

Check consistency of split settings and overrides.

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

author: str

bucket: str

role: str

region: str

service_name: str

pipeline_version: str

model_class: str

current_date: str

framework_version: str

py_version: str

source_dir: str | None

enable_caching: bool

use_secure_pypi: bool

max_runtime_seconds: int

project_root_folder: str