cursus.steps.configs.config_bedrock_batch_processing_step

Bedrock Batch Processing Step Configuration

This module implements the configuration class for the Bedrock Batch Processing step using the three-tier design pattern for optimal user experience and maintainability.

class BedrockBatchProcessingConfig(*, author, bucket, role, region, service_name, pipeline_version, model_class='xgboost', current_date=<factory>, framework_version='2.1.2', py_version='py310', source_dir=None, enable_caching=False, use_secure_pypi=False, max_runtime_seconds=172800, project_root_folder, processing_instance_count=1, processing_volume_size=500, processing_instance_type_large='ml.m5.4xlarge', processing_instance_type_small='ml.m5.2xlarge', use_large_processing_instance=False, skip_volume_kms=None, processing_source_dir=None, processing_entry_point='bedrock_batch_processing.py', processing_script_arguments=None, processing_framework_version='1.2-1', bedrock_batch_role_arn, job_type='training', bedrock_primary_model_id='anthropic.claude-sonnet-4-5-20250929-v1:0', bedrock_fallback_model_id='anthropic.claude-sonnet-4-20250514-v1:0', bedrock_inference_profile_arn=None, bedrock_inference_profile_required_models=<factory>, bedrock_max_tokens=32768, bedrock_temperature=1.0, bedrock_top_p=0.999, bedrock_batch_size=10, bedrock_max_retries=3, bedrock_output_column_prefix='llm_', bedrock_skip_error_records=False, bedrock_concurrency_mode='sequential', bedrock_max_concurrent_workers=5, bedrock_rate_limit_per_second=10, bedrock_batch_mode='auto', bedrock_batch_threshold=1000, bedrock_batch_timeout_hours=24, bedrock_max_records_per_job=45000, bedrock_max_concurrent_batch_jobs=20, bedrock_max_input_field_length=400000, bedrock_truncation_enabled=True, bedrock_log_truncations=True, **extra_data)

Bases: ProcessingStepConfigBase

Configuration for Bedrock Batch Processing step using three-tier design.

This step processes input data through AWS Bedrock models using batch inference capabilities with automatic fallback to real-time processing. Integrates with generated prompt templates and validation schemas from the Bedrock Prompt Template Generation step. Provides cost-efficient processing for large datasets.

Tier 1: Essential user inputs (required) Tier 2: System inputs with defaults (optional) Tier 3: Derived fields (private with property access)

bedrock_batch_role_arn: str

job_type: str

bedrock_primary_model_id: str

bedrock_fallback_model_id: str | None

bedrock_inference_profile_arn: str | None

bedrock_inference_profile_required_models: List[str]

bedrock_max_tokens: int

bedrock_temperature: float

bedrock_top_p: float

bedrock_batch_size: int

bedrock_max_retries: int

bedrock_output_column_prefix: str

bedrock_skip_error_records: bool

bedrock_concurrency_mode: str

bedrock_max_concurrent_workers: int

bedrock_rate_limit_per_second: int

bedrock_batch_mode: str

bedrock_batch_threshold: int

bedrock_batch_timeout_hours: int

bedrock_max_records_per_job: int

bedrock_max_concurrent_batch_jobs: int

bedrock_max_input_field_length: int

bedrock_truncation_enabled: bool

bedrock_log_truncations: bool

processing_entry_point: str

framework_version: str

py_version: str

property effective_inference_profile_required_models: List[str]

Get effective list of models requiring inference profiles with auto-detection.

property bedrock_environment_variables: Dict[str, str]

Get environment variables for the Bedrock batch processing step.

property processing_metadata: Dict[str, Any]

Get processing step metadata.

property batch_configuration: Dict[str, Any]

Get batch processing configuration details.

property cost_optimization_info: Dict[str, Any]

Get cost optimization information.

classmethod validate_job_type(v)

Validate job_type is one of the allowed values.

classmethod validate_primary_model_id(v)

Validate primary model ID format.

classmethod validate_batch_mode(v)

Validate batch processing mode.

classmethod validate_batch_role_arn(v)

Validate batch role ARN format.

classmethod validate_concurrency_mode(v)

Validate concurrency mode.

classmethod validate_inference_profile_models(v)

Validate inference profile required models list.

model_dump(**kwargs)

Override model_dump to include derived properties.

initialize_derived_fields()

Initialize all derived fields once after validation.

validate_production_readiness()

Validate configuration for production readiness.

get_script_path(default_path=None)

Get script path for the Bedrock batch processing step.

Parameters:

default_path (str | None) – Default script path to use if not found via other methods

Returns:

Script path resolved from processing_entry_point and source directories

Return type:

str | None

get_public_init_fields()

Override get_public_init_fields to include Bedrock batch-specific fields. Gets a dictionary of public fields suitable for initializing a child config.

Returns:

Dictionary of field names to values for child initialization

Return type:

Dict[str, Any]

get_environment_variables()

Get all environment variables for the step builder.

Returns:

Complete environment variables dictionary

Return type:

Dict[str, str]

is_production_ready()

Check if configuration is production-ready.

Returns:

True if configuration has production-ready settings

Return type:

bool

get_performance_estimate()

Get estimated performance characteristics including batch processing.

Returns:

Performance estimates and recommendations

Return type:

Dict[str, Any]

get_batch_processing_info()

Get detailed batch processing configuration information.

Returns:

Batch processing details and recommendations

Return type:

Dict[str, Any]

get_job_arguments()

CLI args — config is the single source (FZ 31e1d3h). job_type + batch/retry overrides.

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'protected_namespaces': (), 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.

processing_instance_count: int

processing_volume_size: int

processing_instance_type_large: str

processing_instance_type_small: str

use_large_processing_instance: bool

skip_volume_kms: bool | None

processing_source_dir: str | None

processing_script_arguments: List[str] | None

processing_framework_version: str

author: str

bucket: str

role: str

region: str

service_name: str

pipeline_version: str

model_class: str

current_date: str

source_dir: str | None

enable_caching: bool

use_secure_pypi: bool

max_runtime_seconds: int

project_root_folder: str