cursus.steps.configs.config_tokenizer_training_step

Configuration for Tokenizer Training Processing Step.

This module defines the configuration class for the tokenizer training processing step, which trains a BPE tokenizer optimized for customer name data with automatic vocabulary size tuning to achieve target compression ratio.

class TokenizerTrainingConfig(*, author, bucket, role, region, service_name, pipeline_version, model_class='xgboost', current_date=<factory>, framework_version='2.1.2', py_version='py310', source_dir=None, enable_caching=False, use_secure_pypi=False, max_runtime_seconds=172800, project_root_folder, processing_instance_count=1, processing_volume_size=500, processing_instance_type_large='ml.m5.4xlarge', processing_instance_type_small='ml.m5.2xlarge', use_large_processing_instance=False, skip_volume_kms=None, processing_source_dir=None, processing_entry_point='tokenizer_training.py', processing_script_arguments=None, processing_framework_version='1.2-1', text_field, job_type='training', target_compression=2.5, min_frequency=25, max_vocab_size=50000, **extra_data)

Bases: ProcessingStepConfigBase

Configuration for Tokenizer Training Processing Step.

This class extends ProcessingStepConfigBase to include specific fields for training a BPE tokenizer on text data with compression tuning.

The tokenizer training script uses CompressionBPETokenizer from cursus.processing.tokenizers module to train a tokenizer that matches the legacy OrderTextTokenizer implementation with improved compression tuning capabilities.

text_field: str

processing_entry_point: str

job_type: str

framework_version: str

py_version: str

target_compression: float

min_frequency: int

max_vocab_size: int

property environment_variables: Dict[str, str]

Get environment variables for the tokenizer training script.

Returns:

Dictionary of environment variables required by the script

classmethod validate_job_type(v)

Validate job type is one of the allowed values.

classmethod validate_text_field(v)

Validate text_field is not empty.

validate_tokenizer_config()

Validate tokenizer training configuration.

Ensures all tokenizer parameters are within valid ranges and the configuration is consistent.

get_job_arguments()

CLI args — config is the single source (FZ 31e1d3h).

model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'extra': 'allow', 'protected_namespaces': (), 'validate_assignment': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init(context, /)

This function is meant to behave like a BaseModel method to initialize private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.

• context (Any) – The context.