cursus.step_catalog¶
Unified Step Catalog System
This module provides a unified interface for discovering and retrieving step-related components (scripts, contracts, specifications, builders, configs) across multiple workspaces.
The system consolidates 16+ fragmented discovery mechanisms into a single, efficient StepCatalog class that provides O(1) lookups and intelligent component discovery.
- class StepCatalog(workspace_dirs=None)[source]¶
Bases:
objectUnified step catalog addressing all validated user stories (US1-US5).
This single class consolidates the functionality of 16+ discovery systems while maintaining simple, efficient O(1) lookups through dictionary-based indexing.
PHASE 1 ENHANCEMENT: Now includes StepBuilderRegistry functionality: - Config-to-builder resolution - Legacy alias support - Pipeline construction interface - Enhanced registry integration
- LEGACY_ALIASES = {'MIMSPackaging': 'Package', 'MIMSPayload': 'Payload', 'ModelRegistration': 'Registration', 'PytorchModel': 'PyTorchModel', 'PytorchTraining': 'PyTorchTraining'}¶
- build_complete_config_classes(project_id=None)[source]¶
Build complete mapping integrating manual registration with auto-discovery.
This addresses the TODO in the existing build_complete_config_classes() function.
- create_unified_specification(contract_name)[source]¶
Create unified specification from multiple variants using smart selection.
This method delegates to SpecAutoDiscovery for smart specification handling, integrating SmartSpecificationSelector functionality: - Multi-variant specification discovery - Union of dependencies and outputs from all variants - Smart validation logic with detailed feedback - Primary specification selection (training > generic > first available)
- discover_config_classes(project_id=None)[source]¶
Auto-discover configuration classes from core and workspace directories.
- discover_contracts_with_scripts()[source]¶
DISCOVERY: Find all steps that have both contract and script components.
- discover_cross_workspace_components(workspace_ids=None)[source]¶
DISCOVERY: Find components across multiple workspaces.
- discover_script_files(project_id=None)[source]¶
Discover script files from package and workspaces with prioritization.
This method enables script discovery for interactive runtime testing by finding scripts referenced in config and contract entry points.
- discover_scripts_from_dag(dag)[source]¶
Discover scripts referenced in a DAG with intelligent node-to-script mapping.
This method enables DAG-guided script discovery for interactive runtime testing by mapping DAG nodes to actual script files using step catalog intelligence.
- extract_parent_values_for_inheritance(target_config_class_name, completed_configs)[source]¶
Extract parent values for inheritance from completed configs.
This method enables Smart Default Value Inheritance by extracting field values from the immediate parent config for pre-populating child config forms. It implements cascading inheritance to eliminate redundant user input.
- Parameters:
- Returns:
Dictionary of field values from immediate parent config
- Return type:
Example
- completed_configs = {
“BasePipelineConfig”: base_config_instance, “ProcessingStepConfigBase”: processing_config_instance
}
- parent_values = step_catalog.extract_parent_values_for_inheritance(
“TabularPreprocessingConfig”, completed_configs
) # Returns: ALL field values from ProcessingStepConfigBase # (which includes cascaded values from BasePipelineConfig) # Result: {“author”: “lukexie”, “bucket”: “my-bucket”, # “processing_instance_type”: “ml.m5.2xlarge”, …}
- find_contracts_by_entry_point(entry_point)[source]¶
Find contracts that reference a specific script entry point.
- find_specs_by_contract(contract_name)[source]¶
Find all specifications that reference a specific contract.
This method enables contract-specification alignment validation by finding specifications that are associated with a given contract name.
- get_all_builders()[source]¶
Get all available builders with canonical names.
This method provides comprehensive builder discovery for dynamic testing without requiring hard-coded maintenance.
- get_builder_class_path(step_name)[source]¶
Get builder class path for a step using BuilderAutoDiscovery component.
- get_builder_for_config(config, node_name=None)[source]¶
Map a config instance to a builder PROVIDER (the thing the assembler calls with its 5-kwarg signature to get a StepBuilderBase).
Returns a
BuilderProvider(FZ 31e1d3g1 Phase 1): TODAY this is the per-step builder CLASS (a class is already such a callable), so behavior is unchanged. The annotation is provider-based so the classless Design-B end-state can return a non-class factory here without breaking the contract — callers must invoke the result, never assume it is a class.
- get_builder_for_step_type(step_type)[source]¶
Get the builder PROVIDER for a step type, with legacy alias support.
Returns a
BuilderProvider(FZ 31e1d3g1 Phase 1) — currently the builder class; seeget_builder_for_configfor the dual-mode contract.
- get_builders_by_step_type(step_type)[source]¶
Get builders filtered by SageMaker step type.
This method enables step-type-specific testing by filtering builders based on their registered SageMaker step type.
- get_immediate_parent_config_class(config_class_name)[source]¶
Get the immediate parent config class name for inheritance.
This method enables Smart Default Value Inheritance by identifying which parent config class a specific config should inherit from. It implements cascading inheritance where TabularPreprocessingConfig inherits from ProcessingStepConfigBase (not BasePipelineConfig directly) to get all cascaded values.
- Parameters:
config_class_name (str) – Target config class name (e.g., “TabularPreprocessingConfig”)
- Returns:
Immediate parent class name (e.g., “ProcessingStepConfigBase”) or None if not found
- Return type:
str | None
Example
parent = step_catalog.get_immediate_parent_config_class(“TabularPreprocessingConfig”) # Returns: “ProcessingStepConfigBase” (not “BasePipelineConfig”)
parent = step_catalog.get_immediate_parent_config_class(“CradleDataLoadConfig”) # Returns: “BasePipelineConfig” (direct inheritance)
- get_script_discovery_stats()[source]¶
Get script discovery statistics.
This method provides discovery statistics for interactive runtime testing to help users understand the script discovery process.
- get_script_info(script_name)[source]¶
Get information about a script in dictionary format.
This method provides script information for interactive runtime testing in a user-friendly dictionary format.
- get_sdk_output_location(built_step, output_type=None)[source]¶
Read the output S3 location(s) off a BUILT SDK-delegation step (FZ 31e1d3i).
A generic, DUCK-TYPED accessor for the MODSPredefinedProcessingStep family: any built SDK step that exposes
get_output_locations(the SAIS contract) is supported — CradleDataLoadingStep returns a{DATA, SIGNATURE, METADATA}dict (or one entry whenoutput_typeis given), DataUploadingStep returnsNone(a SINK — output goes to Andes/EDX), and a step lacking the method (e.g. RedshiftDataLoadingStep) returnsNone. This REPLACES the per-stepget_output_location/get_step_outputshelpers the cradle builder used to carry (so that builder becomes a pure shell). It reads a method off an ALREADY-CONSTRUCTED step instance, so it introduces NO offline SAIS import — the SAIS class was necessarily importable to build the step.- Parameters:
- Returns:
The step’s output location(s) as the SDK reports them, or
Noneif the step does not exposeget_output_locations.- Return type:
- get_spec_job_type_variants(base_step_name)[source]¶
Get all job type variants for a base step name from specifications.
This method discovers different job type variants (training, validation, testing, etc.) for a given base step name by examining specification file naming patterns.
- get_step_builder_suggestions(config_class_name)[source]¶
Get suggestions for step builders based on config class name.
- get_step_info(step_name, job_type=None)[source]¶
Get complete information about a step, optionally with job_type variant.
- get_step_interface(step_name, job_type=None)[source]¶
Load a step’s unified
.step.yamlinterface — the single canonical accessor.This is THE one load function for interface data (FZ 31e1d3g3 follow-up): both
load_contract_class(→iface.contract) andload_spec_class(→iface) are thin views over it, so the job_type-suffix fallback below is written once and every consumer inherits it.On the first (exact) load miss, if
job_typewas not supplied, we resolve the node’s base step name ROBUSTLY (naming.resolve_base_step_nameagainst the registry — validates the base against actual step names, not a hardcoded suffix list) and retry with the trailing segment as the job_type (ModelCalibration_calibration→ModelCalibration+ job_typecalibration;CradleDataLoading_munged→CradleDataLoading+munged). Because the base is validated against the registry, ANY suffix resolves (job_type is open now) while a base likeXGBoostModelis never mis-stripped (noXGBooststep exists to strip to).
- has_builder_provider(step_name)[source]¶
Whether a step has a buildable builder, WITHOUT importing/instantiating it (FZ 31e1d3g3 D4).
This is a ROUTABILITY check answered from declarative data — the registry’s
sagemaker_step_type(+ the.step.yamlpatterns.step_assembly) must resolve to a realPatternHandler. It does NOT callload_builder_class(which would import the builder module — and fail for an SDK-bound builder offline even though the step is perfectly routable). So it stays True for a fileless-but-routable step (the factory-shell end-state) and for the SDK steps offline, and False only for genuinely-absent or non-routable rows (Base/Lambda/unknown). Used by the DAG resolver’s component-availability check, which must not encode the deleted-builder_*.py-file assumption.
- is_step_type_supported(step_type)[source]¶
Check if step type is supported (including legacy aliases).
- list_available_scripts()[source]¶
List all available script names.
This method enables script enumeration for interactive runtime testing by listing all discovered script names.
- list_available_steps(workspace_id=None, job_type=None)[source]¶
List all available concrete pipeline steps with deduplication.
Excludes base configuration steps (‘Base’, ‘Processing’) and applies canonical name deduplication following standardization rules.
- list_steps_with_scripts(workspace_id=None, job_type=None)[source]¶
List all steps that have script components.
This method filters available steps to only return those that have script file components, which is useful for alignment validation and script-based testing frameworks.
- list_steps_with_specs(workspace_id=None, job_type=None)[source]¶
List all steps that have specification components.
This method filters available steps to only return those that have specification file components, which is useful for validation frameworks that need to work specifically with steps that have specifications.
- load_all_specifications()[source]¶
Load all specification instances from both package and workspace directories.
This method provides comprehensive specification loading for validation frameworks and dependency analysis tools. It discovers and loads all available specifications, serializing them to dictionary format for easy consumption.
- load_builder_class(step_name)[source]¶
Load builder class for a step with job type variant fallback support.
- load_contract_class(step_name)[source]¶
Load the contract for a step — a VIEW onto its
.step.yamlinterface.
- load_script_info(script_name)[source]¶
Load script information for a specific script with workspace-aware discovery.
This method enables script information retrieval for interactive runtime testing with workspace prioritization support.
- load_spec_class(step_name)[source]¶
Load the specification for a step — the
.step.yamlinterface itself.The StepInterface is a drop-in for the legacy StepSpecification (exposes
dependencies/outputs/…), so it IS the spec view.
- resolve_pipeline_node(node_name)[source]¶
Resolve PipelineDAG node name to StepInfo (handles job_type variants).
- search_steps(query, job_type=None)[source]¶
Search steps by name with basic fuzzy matching.
- Parameters:
- Returns:
List of search results sorted by relevance
- Return type:
- serialize_contract(contract_instance)[source]¶
Convert contract instance to dictionary format.
This method provides standardized serialization of ScriptContract objects for use in script-contract alignment validation.
- serialize_spec(spec_instance)[source]¶
Convert specification instance to dictionary format.
This method provides standardized serialization of StepSpecification objects for use in validation and alignment testing.
- validate_builder_availability(step_types)[source]¶
Validate that builders are available for step types.
- validate_contract_script_mapping()[source]¶
Validate contract-script relationships across the system.
- validate_logical_names_smart(contract, contract_name)[source]¶
Smart validation using multi-variant specification logic.
This method delegates to SpecAutoDiscovery for smart validation, implementing the core Smart Specification Selection validation: - Contract input is valid if it exists in ANY variant - Contract must cover intersection of REQUIRED dependencies - Provides detailed feedback about which variants need what
- class StepInfo(*, step_name, workspace_id, registry_data=<factory>, file_components=<factory>)[source]¶
Bases:
BaseModelEssential step information combining registry data with file metadata.
- model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True}¶
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- file_components: Dict[str, FileMetadata | None]¶
- class FileMetadata(*, path, file_type, modified_time)[source]¶
Bases:
BaseModelSimple metadata for component files.
- model_config: ClassVar[ConfigDict] = {'arbitrary_types_allowed': True, 'frozen': True}¶
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class StepSearchResult(*, step_name, workspace_id, match_score, match_reason, components_available=<factory>)[source]¶
Bases:
BaseModelSimple search result for step queries.
- model_config: ClassVar[ConfigDict] = {'frozen': True}¶
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- class ConfigAutoDiscovery(package_root, workspace_dirs)[source]¶
Bases:
objectSimple configuration class auto-discovery.
- build_complete_config_classes(project_id=None)[source]¶
Build complete mapping using pure auto-discovery. Includes both config and hyperparameter classes for comprehensive discovery.
ConfigClassStore was removed as part of the unified step catalog refactoring. This method now uses pure AST-based auto-discovery which is deployment-agnostic and works in all environments (installed, submodule, Lambda, Docker, etc.).
- discover_config_classes(project_id=None)[source]¶
Auto-discover configuration classes from package and workspace directories.
- create_step_catalog(workspace_root, use_unified=None)[source]¶
Factory function for step catalog with feature flag support.
- class ContractDiscoveryEngineAdapter(workspace_root)[source]¶
Bases:
objectModernized adapter using unified StepCatalog for all discovery operations.
Replaces: src/cursus/validation/alignment/discovery/contract_discovery.py All methods now use the step catalog’s built-in discovery capabilities.
- build_entry_point_mapping()[source]¶
MODERNIZED: Build entry point mapping from step names to script paths.
Used by tests and legacy systems that expect entry point mappings.
- discover_all_contracts()[source]¶
MODERNIZED: Discover all steps that have contracts using step catalog.
Used by alignment validation system.
- discover_contracts_with_scripts()[source]¶
MODERNIZED: Use step catalog’s built-in method.
Historically the primary contract-discovery entry for the Level-2 alignment tester (now removed — contract<->spec alignment is a construction-time invariant; FZ 31e1d3h/D5); the method stays as a general script-backed-contract discovery helper.
- class ContractDiscoveryManagerAdapter(test_data_dir=None, workspace_root=None)[source]¶
Bases:
objectModernized adapter using unified StepCatalog with minimal business logic.
Replaces: src/cursus/validation/script_testing (formerly runtime/contract_discovery.py) Focuses on test-specific functionality while leveraging step catalog for discovery.
- discover_contract(step_name, canonical_name=None)[source]¶
MODERNIZED: Use StepCatalog’s load_contract_class for discovery.
- class ContractDiscoveryResult(contract=None, contract_name=None, discovery_method='step_catalog', error_message=None)[source]¶
Bases:
objectLegacy result class for contract discovery operations.
Maintains backward compatibility with existing tests and code.
- class FlexibleFileResolverAdapter(workspace_root)[source]¶
Bases:
objectModernized file resolver using unified step catalog system.
Replaces: src/cursus/validation/alignment/file_resolver.py
This adapter leverages the step catalog’s superior discovery capabilities while maintaining backward compatibility with legacy FlexibleFileResolver APIs.
- class DeveloperWorkspaceFileResolverAdapter(workspace_root=None, project_id=None, developer_id=None, enable_shared_fallback=True, **kwargs)[source]¶
Bases:
FlexibleFileResolverAdapterAdapter maintaining backward compatibility with DeveloperWorkspaceFileResolver.
Replaces: src/cursus/workspace/validation/workspace_file_resolver.py
- find_builder_file(step_name)[source]¶
Workspace-aware builder file discovery. Returns string path like legacy method.
- find_config_file(step_name)[source]¶
Workspace-aware config file discovery. Returns string path like legacy method.
- find_contract_file(step_name)[source]¶
Workspace-aware contract file discovery. Returns string path like legacy method.
- find_script_file(step_name, script_name=None)[source]¶
Workspace-aware script file discovery. Returns string path like legacy method.
- class WorkspaceDiscoveryManagerAdapter(workspace_root)[source]¶
Bases:
objectAdapter maintaining backward compatibility with WorkspaceDiscoveryManager.
Replaces: src/cursus/workspace/core/discovery.py
- discover_components(workspace_ids=None, developer_id=None)[source]¶
Enhanced method: discover ALL component types with workspace-specific functionality.
When workspace_ids is None: focuses on shared package (cursus/steps - “core” workspace) When workspace_ids specified: discovers from those specific workspaces
Returns comprehensive inventory of all 5 component types: - builders, configs, contracts, specs, scripts
- discover_workspaces(workspace_root)[source]¶
Legacy method: discover available workspaces with simplified structure.
- get_workspace_info(workspace_id=None, developer_id=None)[source]¶
Legacy method: get workspace info with simplified structure.
- class HybridFileResolverAdapter(workspace_root)[source]¶
Bases:
objectAdapter maintaining backward compatibility with HybridFileResolver.
Replaces: src/cursus/validation/alignment/patterns/file_resolver.py
- class LegacyDiscoveryWrapper(workspace_root)[source]¶
Bases:
objectWrapper providing legacy discovery interfaces during migration period.
This class provides a unified interface that can be used as a drop-in replacement for legacy discovery systems during the migration phase. It delegates all StepCatalog methods to the underlying catalog while also providing access to legacy adapters.
- discover_cross_workspace_components(workspace_ids=None)[source]¶
Delegate to underlying StepCatalog.
Modules
Step catalog adapters package. |
|
Builder class discovery via registry-interface synthesis. |
|
Configuration class auto-discovery for the unified step catalog system. |
|
Contract discovery for the unified step catalog system. |
|
Step Catalog Mapping Module - Config-to-Builder Resolution and Legacy Support. |
|
Simple data models for the unified step catalog system. |
|
Single source of truth for step-name <-> file-name conversion in the step catalog. |
|
Script file auto-discovery for the unified step catalog system. |
|
Lazy SAIS-SDK step-class bindings for the registry-walk builder materializer (FZ 31e1d3g3 Phase A2). |
|
Specification discovery for the unified step catalog system. |
|
Unified Step Catalog - Single class addressing all US1-US5 requirements. |