cursus.steps.scripts.payload

MIMS Payload Generation Processing Script

This script reads field information from hyperparameters extracted from model.tar.gz, extracts configuration from environment variables, and creates payload files for model inference.

class VariableType(*values)

Bases: str, Enum

Type of variable in model input/output

NUMERIC = 'NUMERIC'

TEXT = 'TEXT'

detect_model_type(hyperparams)

Detect model type from hyperparameters.

Detection logic: 1. Check for trimodal indicators (primary_text_name + secondary_text_name) 2. Check for bimodal indicators (text_name field) 3. Default to tabular (traditional XGBoost/LightGBM)

Parameters:

hyperparams (Dict) – Dictionary loaded from hyperparameters.json

Returns:

‘trimodal’, ‘bimodal’, or ‘tabular’

Return type:

str

get_field_defaults(environ_vars)

Load field default values from environment.

Priority (highest to lowest): 1. SPECIAL_FIELD_* prefix (per-field overrides, highest priority for backward compatibility) 2. FIELD_DEFAULTS (JSON dict, base defaults) 3. Empty dict (use auto-generated intelligent defaults)

Parameters:

environ_vars (Dict[str, str]) – Environment variables dictionary

Returns:

Dictionary mapping field names to default values

Return type:

Dict[str, str]

generate_text_sample(field_name, field_defaults, default_text_value='Sample text for inference testing')

Generate sample text for a text field with 3-tier priority.

Priority (highest to lowest): 1. User-provided value from field_defaults (exact or case-insensitive match) 2. Intelligent default based on field name pattern 3. Generic default from DEFAULT_TEXT_VALUE

Parameters:

• field_name (str) – Name of the text field

• field_defaults (Dict[str, str]) – User-provided field defaults dictionary

• default_text_value (str) – Generic fallback default

Returns:

Sample text string for the field

Return type:

str

load_custom_payload(custom_path, content_type='application/json')

Load user-provided custom payload sample.

Supports: - JSON file: Load and return as dict - CSV file: Load first row as dict - Parquet file: Load first row as dict - Directory: Search for JSON/CSV/Parquet files

Parameters:

• custom_path (Path) – Path to custom payload file or directory

• content_type (str) – Expected content type (‘application/json’ or ‘text/csv’)

Returns:

Dictionary with payload data if successful, None otherwise

Return type:

Dict | None

get_required_fields_from_model(model_dir, hyperparams, var_type_list)

Get required fields using the SAME logic as inference handlers.

This ensures validation matches what the actual inference handler expects.

Priority: 1. feature_columns.txt (if exists) - for XGBoost/LightGBM models 2. hyperparameters.json - for PyTorch models or fallback

Parameters:

• model_dir (Path) – Directory containing model artifacts

• hyperparams (Dict) – Model hyperparameters from hyperparameters.json

• var_type_list (List[List[str]]) – List of [field_name, field_type] pairs

Returns:

• tabular_fields: List of required tabular feature names

• id_field: Optional ID field name

• text_fields: Dict of text field names by type

• model_type: ‘tabular’, ‘bimodal’, or ‘trimodal’

• field_order: Ordered list of all fields (for CSV)

• source: ‘feature_columns.txt’ or ‘hyperparameters.json’

Return type:

Dictionary with

validate_payload_completeness(payload, hyperparams, var_type_list, model_dir=None)

Validate payload contains all required fields for any model type.

Uses the SAME source of truth as inference handlers: - feature_columns.txt for XGBoost/LightGBM - hyperparameters.json for PyTorch

Parameters:

• payload (Dict) – Generated payload dictionary

• hyperparams (Dict) – Model hyperparameters

• var_type_list (List[List[str]]) – List of [field_name, field_type] pairs

• model_dir (Path | None) – Optional model directory to check for feature_columns.txt

Returns:

(is_valid, missing_fields)

Return type:

Tuple[bool, List[str]]

log_payload_field_mapping(payload, hyperparams, var_type_list)

Log comprehensive field mapping for payload validation and debugging.

Parameters:

• payload (Dict) – Generated payload dictionary

• hyperparams (Dict) – Model hyperparameters

• var_type_list (List[List[str]]) – List of [field_name, field_type] pairs

ensure_directory(directory_path)

Ensure a directory exists, creating it if necessary.

create_model_variable_list(full_field_list, tab_field_list, cat_field_list, label_name='label', id_name='id')

Creates a list of [variable_name, variable_type] pairs.

Parameters:

• full_field_list (List[str]) – List of all field names

• tab_field_list (List[str]) – List of numeric/tabular field names

• cat_field_list (List[str]) – List of categorical field names

• label_name (str) – Name of the label column (default: “label”)

• id_name (str) – Name of the ID column (default: “id”)

Returns:

List of [variable_name, type] pairs where type is ‘NUMERIC’ or ‘TEXT’

Return type:

List[List[str]]

extract_hyperparameters_from_tarball(input_model_dir, working_directory)

Extract and load hyperparameters from model artifacts

get_environment_content_types(environ_vars)

Get content types from environment variables.

get_environment_default_numeric_value(environ_vars)

Get default numeric value from environment variables.

get_environment_default_text_value(environ_vars)

Get default text value from environment variables.

generate_csv_payload(input_vars, default_numeric_value, default_text_value, hyperparams=None, field_defaults=None, model_dir=None)

Generate CSV format payload with multi-modal support.

CRITICAL: For XGBoost/LightGBM models, field order MUST match feature_columns.txt for inference to work correctly. This function uses get_required_fields_from_model() to ensure correct ordering.

Handles: - Tabular: Only numeric/categorical fields - Bimodal: text_name + numeric/categorical fields - Trimodal: primary_text_name + secondary_text_name + numeric/categorical fields

Parameters:

• input_vars – List of [field_name, var_type] pairs for tabular features

• default_numeric_value (float) – Default for numeric fields

• default_text_value (str) – Generic default for text fields

• hyperparams (Dict | None) – Full hyperparameters dict from model (for multi-modal detection)

• field_defaults (Dict[str, str] | None) – User-provided field defaults dictionary

• model_dir (Path | None) – Model directory to check for feature_columns.txt (CRITICAL for correct ordering)

Returns:

Comma-separated string of values (no header) in CORRECT field order

Return type:

str

generate_json_payload(input_vars, default_numeric_value, default_text_value, hyperparams=None, field_defaults=None)

Generate JSON format payload with multi-modal support.

Handles: - Tabular: Only numeric/categorical fields - Bimodal: text_name + numeric/categorical fields - Trimodal: primary_text_name + secondary_text_name + numeric/categorical fields

Parameters:

• input_vars – List of [field_name, var_type] pairs for tabular features

• default_numeric_value (float) – Default for numeric fields

• default_text_value (str) – Generic default for text fields

• hyperparams (Dict | None) – Full hyperparameters dict from model (for multi-modal detection)

• field_defaults (Dict[str, str] | None) – User-provided field defaults dictionary

Returns:

JSON string with complete payload

Return type:

str

generate_sample_payloads(input_vars, content_types, default_numeric_value, default_text_value, hyperparams=None, field_defaults=None, model_dir=None)

Generate sample payloads for each content type with multi-modal support.

Parameters:

• input_vars – List of [field_name, var_type] pairs for tabular features

• content_types (List[str]) – List of content types to generate

• default_numeric_value (float) – Default for numeric fields

• default_text_value (str) – Generic default for text fields

• hyperparams (Dict | None) – Full hyperparameters dict (for multi-modal detection)

• field_defaults (Dict[str, str] | None) – User-provided field defaults dictionary

• model_dir (Path | None) – Model directory for correct CSV field ordering (CRITICAL for XGBoost/LightGBM)

Returns:

List of dictionaries containing content type and payload

Return type:

List[Dict[str, str | dict]]

save_payloads(output_dir, input_vars, content_types, default_numeric_value, default_text_value, hyperparams=None, field_defaults=None, model_dir=None)

Save payloads to files with multi-modal support.

Parameters:

• output_dir (str) – Directory to save payload files

• input_vars – Source model inference input variable list

• content_types (List[str]) – List of content types to generate payloads for

• default_numeric_value (float) – Default value for numeric fields

• default_text_value (str) – Default value for text fields

• hyperparams (Dict | None) – Full hyperparameters dict (for multi-modal detection)

• field_defaults (Dict[str, str] | None) – User-provided field defaults dictionary

• model_dir (Path | None) – Model directory for correct CSV field ordering (CRITICAL for XGBoost/LightGBM)

Returns:

List of paths to created payload files

Return type:

List[str]

create_payload_archive(payload_files, output_dir=None)

Create a tar.gz archive containing only payload files (not metadata).

Parameters:

• payload_files (List[str]) – List of paths to payload files

• output_dir (Path) – Output directory path (defaults to DEFAULT_OUTPUT_DIR)

Returns:

Path to the created archive

Return type:

str

main(input_paths, output_paths, environ_vars, job_args=None)

Main entry point for the MIMS payload generation script.

Parameters:

• input_paths (Dict[str, str]) – Dictionary of input paths with logical names

• output_paths (Dict[str, str]) – Dictionary of output paths with logical names

• environ_vars (Dict[str, str]) – Dictionary of environment variables

• job_args (Namespace | None) – Command line arguments (optional)

Returns:

Path to the generated payload archive file

Return type:

str