cursus.steps.scripts.label_ruleset_execution

Label Ruleset Execution Script

Applies validated rulesets to processed data to generate classification labels. Supports train/val/test splits and provides comprehensive execution statistics.

Key Features: - Field availability validation at execution time - Priority-based rule evaluation (first match wins) - Comprehensive statistics tracking - Fail-safe error handling - Support for multiple job types (training, validation, testing, calibration)

Usage:

python label_ruleset_execution.py –job-type training

class RulesetFieldValidator[source]

Bases: object

Validates field availability in actual data at execution time.

validate_fields(ruleset, data_df)[source]

Validates all field references exist in actual data.

This is an EXECUTION-TIME validator that checks: - All required fields exist in DataFrame - All fields used in rules exist in DataFrame - Field null percentages (data quality check)

Parameters:

  • ruleset (dict) – Validated ruleset configuration

  • data_df (DataFrame) – Actual DataFrame to check

Returns:

  • valid: bool

  • missing_fields: List[str]

  • warnings: List[str]

Return type:

Dictionary with validation results

class RuleEngine(validated_ruleset)[source]

Bases: object

Evaluates validated rules against data rows to produce labels. Extended for multilabel support.

Optimized for: - Batch processing (vectorized where possible) - Priority-based evaluation (first match wins) - Efficient condition checking - Minimal memory footprint - Multilabel sparse representation

evaluate_row(row)[source]

Evaluate rules against a single row.

Returns:

int or str (label value) - Multilabel mode: Dict[str, Any] (column → value mapping)

Return type:

  • Single-label mode

evaluate_batch(df)[source]

Evaluate rules for entire DataFrame.

Returns:

DataFrame with label column(s) added

Return type:

DataFrame

get_statistics()[source]

Get execution statistics with multilabel support.

apply_field_types(df, field_config)[source]

Apply field type conversions based on ruleset field_config.

This ensures data types match rule expectations, avoiding type mismatch issues (e.g., string ‘1’ vs float 1.0) that can cause rule evaluation failures.

Parameters:

  • df (DataFrame) – Input DataFrame

  • field_config (dict) – Field configuration from validated ruleset containing field_types

Returns:

DataFrame with corrected types

Return type:

DataFrame

main(input_paths, output_paths, environ_vars, job_args, logger=None)[source]

Main logic for ruleset execution.

Supports multiple file formats: CSV, TSV, Parquet (auto-detected).

Parameters:

  • input_paths (Dict[str, str]) – Dictionary with keys: - “validated_ruleset”: Path to validated ruleset JSON - “input_data”: Directory with train/val/test splits

  • output_paths (Dict[str, str]) – Dictionary with keys: - “processed_data”: Directory for output with labels - “execution_report”: Path for execution statistics - “rule_match_statistics”: Optional path for detailed statistics

  • environ_vars (Dict[str, str]) – Environment variables

  • job_args (Namespace) – Command line arguments (job_type)

  • logger (Callable[[str], None] | None) – Optional logger function

Returns:

Dictionary of processed DataFrames by split name

Return type:

Dict[str, DataFrame]