cursus.steps.scripts.pseudo_label_merge

Pseudo Label Merge Script

Intelligently merges original labeled training data with pseudo-labeled or augmented samples for Semi-Supervised Learning (SSL) and Active Learning workflows.

Key Features: - Split-aware merge for training jobs (maintains train/test/val boundaries) - Auto-inferred split ratios (adapts to base data proportions) - Simple merge for validation/testing jobs - Data format preservation (CSV/TSV/Parquet) - Schema alignment and provenance tracking

Design: slipbox/1_design/pseudo_label_merge_script_design.md

load_dataframe_with_format(file_path)

Load DataFrame and detect its format.

Parameters:

file_path (Path) – Path to the file

Returns:

Tuple of (DataFrame, format_string)

Return type:

Tuple[DataFrame, str]

save_dataframe_with_format(df, output_path, format_str)

Save DataFrame in specified format.

Parameters:

• df (DataFrame) – DataFrame to save

• output_path (Path) – Base output path (without extension)

• format_str (str) – Format to save in (‘csv’, ‘tsv’, or ‘parquet’)

Returns:

Path to saved file

Return type:

Path

load_base_data(base_data_dir, job_type)

Load base training data, detecting split structure automatically.

Parameters:

• base_data_dir (str) – Path to base data directory

• job_type (str) – Job type (training, validation, testing, calibration)

Returns:

Dictionary mapping split names to DataFrames - Training job: {“train”: df, “test”: df, “val”: df} - Other jobs: {job_type: df}

Return type:

Dict[str, DataFrame]

load_augmentation_data(aug_data_dir)

Load augmentation data (always single dataset).

Parameters:

aug_data_dir (str) – Path to augmentation data directory

Returns:

DataFrame with augmentation samples

Return type:

DataFrame

detect_merge_strategy(base_splits, job_type)

Determine merge strategy based on input structure.

Parameters:

• base_splits (Dict[str, DataFrame]) – Dictionary of base data splits

• job_type (str) – Job type

Returns:

“split_aware” or “simple”

Return type:

str

extract_split_ratios(base_splits)

Calculate split proportions from base data.

Parameters:

base_splits (Dict[str, DataFrame]) – Dictionary with train/test/val splits

Returns:

Dictionary with split proportions summing to 1.0

Return type:

Dict[str, float]

align_schemas(base_df, aug_df, label_field, pseudo_label_column='pseudo_label', id_field='id')

Align schemas between base and augmentation data.

Handles: - Label column conversion (pseudo_label → label) - Common columns extraction - Data type compatibility

Parameters:

• base_df (DataFrame) – Base training data

• aug_df (DataFrame) – Augmentation data

• label_field (str) – Label column name

• pseudo_label_column (str) – Pseudo-label column name in augmentation

• id_field (str) – ID column name

Returns:

Tuple of (aligned_base_df, aligned_aug_df)

Return type:

Tuple[DataFrame, DataFrame]

merge_with_splits(base_splits, augmentation_df, label_field, use_auto_split_ratios=True, train_ratio=None, test_val_ratio=None, stratify=True, random_seed=42, preserve_confidence=True)

Merge with proportional augmentation distribution across splits.

Strategy: 1. Auto-infer split ratios from base data (or use manual ratios if provided) 2. Split augmentation data using calculated proportions 3. Add provenance to all datasets 4. Merge corresponding splits 5. Return merged splits maintaining structure

Parameters:

• base_splits (Dict[str, DataFrame]) – Dictionary with train/test/val DataFrames

• augmentation_df (DataFrame) – Augmentation data to distribute

• label_field (str) – Label column name for stratification

• use_auto_split_ratios (bool) – Auto-infer ratios from base data (recommended)

• train_ratio (float | None) – Optional proportion for train split (None = auto-infer from base)

• test_val_ratio (float | None) – Optional test vs val proportion of holdout (None = auto-infer)

• stratify (bool) – Use stratified splits if True

• random_seed (int) – Random seed for reproducibility

• preserve_confidence (bool) – Keep confidence scores if present

Returns:

Dictionary with merged train/test/val DataFrames

Return type:

Dict[str, DataFrame]

merge_simple(base_df, augmentation_df, preserve_confidence=True)

Simple merge for non-training job types.

Parameters:

• base_df (DataFrame) – Base dataset

• augmentation_df (DataFrame) – Augmentation dataset

• preserve_confidence (bool) – Keep confidence scores if present

Returns:

Merged DataFrame with provenance

Return type:

DataFrame

validate_provenance(merged_df, expected_sources={'original', 'pseudo_labeled'})

Validate provenance column in merged data.

save_merged_data(merged_splits, output_dir, output_format='csv', job_type='training')

Save merged data maintaining input structure.

Parameters:

• merged_splits (Dict[str, DataFrame]) – Dictionary of merged DataFrames by split

• output_dir (str) – Output directory path

• output_format (str) – “csv”, “tsv”, or “parquet”

• job_type (str) – Job type for file naming

Returns:

Dictionary mapping split names to output file paths

Return type:

Dict[str, str]

save_merge_metadata(output_dir, metadata)

Save merge operation metadata.

Parameters:

• output_dir (str) – Output directory

• metadata (Dict[str, Any]) – Metadata dictionary

Returns:

Path to metadata file

Return type:

str

main(input_paths, output_paths, environ_vars, job_args)

Main function for pseudo label merge.

Parameters:

• input_paths (Dict[str, str]) – Dictionary with keys: - base_data: Path to base labeled data - augmentation_data: Path to augmentation data

• output_paths (Dict[str, str]) – Dictionary with keys: - merged_data: Path for merged output

• environ_vars (Dict[str, str]) – Dictionary with environment variables: - LABEL_FIELD: Label column name (REQUIRED) - ADD_PROVENANCE: Track data source (default: “true”) - OUTPUT_FORMAT: Output format (default: “csv”) - USE_AUTO_SPLIT_RATIOS: Auto-infer split ratios (default: “true”) - TRAIN_RATIO: Train split proportion (default: None) - TEST_VAL_RATIO: Test vs val proportion (default: None) - PSEUDO_LABEL_COLUMN: Pseudo-label column name (default: “pseudo_label”) - ID_FIELD: ID column name (default: “id”) - PRESERVE_CONFIDENCE: Keep confidence scores (default: “true”) - STRATIFY: Use stratified splits (default: “true”) - RANDOM_SEED: Random seed (default: “42”)

• job_args (Namespace) – Command-line arguments: - job_type: Type of merge job (training, validation, testing, calibration)

Returns:

Dictionary of merged DataFrames by split name

Return type:

Dict[str, DataFrame]