cursus.steps.scripts.feature_selection¶

Feature Selection Script

A comprehensive feature selection script that implements multiple statistical and machine learning-based feature selection methods. Follows the cursus framework’s testability patterns and provides a self-contained solution.

Author: Cursus Framework Date: 2025-10-25

check_call(cmd, *a, **k)¶

install_packages_from_public_pypi(packages)[source]¶

Install packages from standard public PyPI.

Parameters:: packages (list) – List of package specifications (e.g., [“pandas==1.5.0”, “numpy”])

install_packages_from_secure_pypi(packages)[source]¶

Install packages from secure CodeArtifact PyPI.

Parameters:: packages (list) – List of package specifications (e.g., [“pandas==1.5.0”, “numpy”])

install_packages(packages, use_secure=False)[source]¶

Install packages from PyPI source based on configuration.

This is the main installation function that delegates to either public or secure PyPI based on the USE_SECURE_PYPI environment variable.

Parameters:

packages (list) – List of package specifications (e.g., [“pandas==1.5.0”, “numpy”])
use_secure (bool) – If True, use secure CodeArtifact PyPI; if False, use public PyPI. Defaults to USE_SECURE_PYPI environment variable.

Environment Variables:: USE_SECURE_PYPI: Set to “true” to use secure PyPI, “false” for public PyPI

Example

# Install from public PyPI (default) install_packages([“xgboost==1.7.3”])

# Install from secure PyPI os.environ[“USE_SECURE_PYPI”] = “true” install_packages([“xgboost==1.7.3”])

setup_logging()[source]¶

Configure logging for CloudWatch compatibility

copy_existing_artifacts(src_dir, dst_dir)[source]¶

Copy all existing model artifacts from previous processing steps.

This enables the parameter accumulator pattern where each step: 1. Copies artifacts from previous steps 2. Adds its own artifacts 3. Passes all artifacts to the next step

Parameters:

src_dir (str) – Source directory containing existing artifacts
dst_dir (str) – Destination directory to copy artifacts to

load_selected_features(model_artifacts_dir)[source]¶

Load pre-computed selected features from training job.

Parameters:: model_artifacts_dir (str) – Directory containing selected_features.json
Returns:: List of selected feature names
Return type:: List[str]

load_single_split_data(input_data_dir, job_type)[source]¶

Load single split data for non-training job types with format detection.

Parameters:

input_data_dir (str) – Directory containing job_type subdirectory
job_type (str) – Type of job (validation, testing, etc.)

Returns:

Dictionary with single split DataFrame and format metadata

Return type:

Dict[str, DataFrame]

load_preprocessed_data(input_data_dir)[source]¶

Load train/val/test splits from tabular preprocessing output with format detection.

Parameters:: input_data_dir (str) – Directory containing train/val/test subdirectories
Returns:: Dictionary with ‘train’, ‘val’, ‘test’ DataFrames and format metadata
Return type:: Dict[str, DataFrame]

save_selected_data(splits, selected_features, target_variable, output_dir)[source]¶

Save feature-selected splits preserving input format.

Parameters:

splits (Dict[str, DataFrame]) – Dictionary of DataFrames by split name (includes “_format” key)
selected_features (List[str]) – List of selected feature names
target_variable (str) – Target column name
output_dir (str) – Output directory path

variance_threshold_selection(X, threshold=0.01)[source]¶

Remove features with low variance.

Parameters:

X (DataFrame) – Feature matrix
threshold (float) – Variance threshold

Returns:

Dictionary with selected features and scores

Return type:

Dict[str, Any]

correlation_based_selection(X, y, threshold=0.95)[source]¶

Remove highly correlated features, keeping those with higher target correlation.

Parameters:

X (DataFrame) – Feature matrix
y (Series) – Target variable
threshold (float) – Correlation threshold

Returns:

Dictionary with selected features and scores

Return type:

Dict[str, Any]

mutual_info_selection(X, y, k=10, random_state=42)[source]¶

Select features based on mutual information with target.

Parameters:

X (DataFrame) – Feature matrix
y (Series) – Target variable
k (int) – Number of features to select
random_state (int) – Random seed

Returns:

Dictionary with selected features and scores

Return type:

Dict[str, Any]

chi2_selection(X, y, k=10)[source]¶

Select features using chi-square test (for non-negative features).

Parameters:

X (DataFrame) – Feature matrix (non-negative values)
y (Series) – Target variable
k (int) – Number of features to select

Returns:

Dictionary with selected features and scores

Return type:

Dict[str, Any]

f_classif_selection(X, y, k=10)[source]¶

Select features using ANOVA F-test.

Parameters:

X (DataFrame) – Feature matrix
y (Series) – Target variable
k (int) – Number of features to select

Returns:

Dictionary with selected features and scores

Return type:

Dict[str, Any]

rfe_selection(X, y, estimator_type='rf', n_features=10, random_state=42)[source]¶

Recursive Feature Elimination with various estimators.

Parameters:

X (DataFrame) – Feature matrix
y (Series) – Target variable
estimator_type (str) – Type of estimator (‘rf’, ‘svm’, ‘linear’)
n_features (int) – Number of features to select
random_state (int) – Random seed

Returns:

Dictionary with selected features and scores

Return type:

Dict[str, Any]

feature_importance_selection(X, y, method='random_forest', n_features=10, random_state=42)[source]¶

Select features based on model feature importance.

Parameters:

X (DataFrame) – Feature matrix
y (Series) – Target variable
method (str) – Method for importance (‘random_forest’, ‘xgboost’, ‘extra_trees’)
n_features (int) – Number of features to select
random_state (int) – Random seed

Returns:

Dictionary with selected features and scores

Return type:

Dict[str, Any]

lasso_selection(X, y, alpha=0.01, random_state=42)[source]¶

Select features using LASSO regularization.

Parameters:

X (DataFrame) – Feature matrix
y (Series) – Target variable
alpha (float | str) – Regularization strength
random_state (int) – Random seed

Returns:

Dictionary with selected features and scores

Return type:

Dict[str, Any]

permutation_importance_selection(X, y, estimator_type='rf', n_features=10, random_state=42)[source]¶

Select features using permutation importance.

Parameters:

X (DataFrame) – Feature matrix
y (Series) – Target variable
estimator_type (str) – Type of estimator
n_features (int) – Number of features to select
random_state (int) – Random seed

Returns:

Dictionary with selected features and scores

Return type:

Dict[str, Any]

combine_selection_results(method_results, combination_strategy='voting', final_k=10)[source]¶

Combine results from multiple feature selection methods.

Parameters:

method_results (List[Dict[str, Any]]) – List of results from different methods
combination_strategy (str) – Strategy for combination (‘voting’, ‘ranking’, ‘scoring’)
final_k (int) – Final number of features to select

Returns:

Combined selection results

Return type:

Dict[str, Any]

apply_feature_selection_pipeline(splits, target_variable, methods, method_configs)[source]¶

Apply feature selection pipeline using training data, then apply to all splits.

Parameters:

splits (Dict[str, DataFrame]) – Dictionary of train/val/test DataFrames
target_variable (str) – Target column name
methods (List[str]) – List of feature selection methods to apply
method_configs (Dict[str, Dict]) – Configuration for each method

Returns:

Dictionary with selection results and metadata

Return type:

Dict[str, Any]

save_selection_results(selection_results, model_artifacts_dir)[source]¶

Save feature selection results and metadata to model artifacts directory.

Parameters:

selection_results (Dict[str, Any]) – Results from feature selection pipeline
model_artifacts_dir (str) – Directory for all feature selection output files

main(input_paths, output_paths, environ_vars, job_args)[source]¶

Main function for feature selection processing.

Parameters:

input_paths (Dict[str, str]) – Dictionary of input paths with logical names - “input_data”: Directory containing train/val/test splits from tabular preprocessing - “model_artifacts_input”: Model artifacts from previous steps (standardized)
output_paths (Dict[str, str]) – Dictionary of output paths with logical names - “processed_data”: Directory for feature-selected train/val/test splits (XGBoost input format) - “model_artifacts_output”: Model artifacts output for next steps (standardized)
environ_vars (Dict[str, str]) – Dictionary of environment variables - “FEATURE_SELECTION_METHODS”: Comma-separated list of methods - “LABEL_FIELD”: Target column name (standard across framework) - “N_FEATURES_TO_SELECT”: Number/percentage of features to select - “CORRELATION_THRESHOLD”: Threshold for correlation filtering - “VARIANCE_THRESHOLD”: Threshold for variance filtering - “RANDOM_STATE”: Random seed for reproducibility - “COMBINATION_STRATEGY”: Method combination strategy
job_args (Namespace) – Command line arguments - job_type: Must be “training” to process all splits