Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode Skip to content
Cursus 2.8.5
Cursus 2.8.5

Getting Started

  • Getting Started
    • Installation
    • Quickstart
    • Core Concepts
  • Installation
  • Quickstart
  • Core Concepts

Tutorials

  • Tutorials
    • Build & Compile Your First Pipeline
    • Build a Pipeline from the Catalog
    • Author a Custom Step
    • Ship Your Own Steps as a Step Pack
    • Drive Cursus with the Agent (MCP) Tools

Concepts & Guides

  • Concepts, Architecture & Design
    • System Architecture
    • Design Principles
    • DAGs & Compilation
    • Dependency Resolution
    • Step Interfaces & Builder Synthesis
    • The Configuration System
    • Registry & Interface-First Discovery
    • Step Packs: External Steps
    • Validation & Alignment
    • The Pipeline Catalog
    • Path Resolution & the Caller Hook
    • The Agent (MCP) Tool Surface
    • Execution Documents
  • How-to Guides
    • Compile, Deploy & Run a Pipeline
    • Generate & Inspect Configs
    • Define a Step Pack
    • Validate a Pipeline
    • Inspect Steps, Catalog & Registry
  • Migration
    • Migrating to 2.x

Reference

  • API Reference
    • cursus
      • cursus.api
        • cursus.api.dag
          • cursus.api.dag.base_dag
          • cursus.api.dag.pipeline_dag_resolver
          • cursus.api.dag.pipeline_dag_serializer
        • cursus.api.factory
          • cursus.api.factory.config_class_mapper
          • cursus.api.factory.configuration_generator
          • cursus.api.factory.dag_config_factory
          • cursus.api.factory.field_extractor
      • cursus.cli
        • cursus.cli.alignment_cli
        • cursus.cli.catalog_cli
        • cursus.cli.compile_cli
        • cursus.cli.config_cli
        • cursus.cli.dag_cli
        • cursus.cli.exec_doc_cli
        • cursus.cli.mcp_cli
        • cursus.cli.pipeline_catalog_cli
        • cursus.cli.project_cli
        • cursus.cli.registry_cli
        • cursus.cli.steps_cli
        • cursus.cli.strategies_cli
        • cursus.cli.validate_cli
      • cursus.core
        • cursus.core.assembler
          • cursus.core.assembler.pipeline_assembler
          • cursus.core.assembler.pipeline_template_base
        • cursus.core.base
          • cursus.core.base.builder_base
          • cursus.core.base.builder_templates
          • cursus.core.base.config_base
          • cursus.core.base.contract_base
          • cursus.core.base.enums
          • cursus.core.base.hyperparameters_base
          • cursus.core.base.step_contract
          • cursus.core.base.step_interface
        • cursus.core.compiler
          • cursus.core.compiler.dag_compiler
          • cursus.core.compiler.dynamic_template
          • cursus.core.compiler.exceptions
          • cursus.core.compiler.name_generator
          • cursus.core.compiler.single_node_compiler
          • cursus.core.compiler.validation
        • cursus.core.config_fields
          • cursus.core.config_fields.constants
          • cursus.core.config_fields.inheritance_aware_field_generator
          • cursus.core.config_fields.step_catalog_aware_categorizer
          • cursus.core.config_fields.type_aware_config_serializer
          • cursus.core.config_fields.unified_config_manager
        • cursus.core.deps
          • cursus.core.deps.dependency_resolver
          • cursus.core.deps.factory
          • cursus.core.deps.property_reference
          • cursus.core.deps.registry_manager
          • cursus.core.deps.semantic_matcher
          • cursus.core.deps.specification_registry
        • cursus.core.utils
          • cursus.core.utils.generic_path_discovery
          • cursus.core.utils.hybrid_path_resolution
          • cursus.core.utils.nvme_security
          • cursus.core.utils.project_discovery
      • cursus.mcp
        • cursus.mcp.envelope
        • cursus.mcp.registry
        • cursus.mcp.server
        • cursus.mcp.tools
          • cursus.mcp.tools.author
          • cursus.mcp.tools.catalog
          • cursus.mcp.tools.compile
          • cursus.mcp.tools.config
          • cursus.mcp.tools.dag
          • cursus.mcp.tools.execdoc
          • cursus.mcp.tools.info
          • cursus.mcp.tools.pipeline_catalog
          • cursus.mcp.tools.project
          • cursus.mcp.tools.shared
          • cursus.mcp.tools.steps
          • cursus.mcp.tools.strategies
          • cursus.mcp.tools.validate
      • cursus.mods
        • cursus.mods.exe_doc
          • cursus.mods.exe_doc.base
          • cursus.mods.exe_doc.cradle_helper
          • cursus.mods.exe_doc.data_uploading_helper
          • cursus.mods.exe_doc.generator
          • cursus.mods.exe_doc.redshift_data_loading_helper
          • cursus.mods.exe_doc.registration_helper
          • cursus.mods.exe_doc.utils
      • cursus.pipeline_catalog
        • cursus.pipeline_catalog.core
          • cursus.pipeline_catalog.core.agent_tool
          • cursus.pipeline_catalog.core.builders
          • cursus.pipeline_catalog.core.pipeline_factory
          • cursus.pipeline_catalog.core.router
        • cursus.pipeline_catalog.mods_pipelines
        • cursus.pipeline_catalog.shared_dags
          • cursus.pipeline_catalog.shared_dags.dummy
          • cursus.pipeline_catalog.shared_dags.pytorch
          • cursus.pipeline_catalog.shared_dags.singleton
          • cursus.pipeline_catalog.shared_dags.xgboost
      • cursus.processing
        • cursus.processing.categorical
          • cursus.processing.categorical.categorical_imputation_processor
          • cursus.processing.categorical.categorical_label_processor
          • cursus.processing.categorical.categorical_validation_processor
          • cursus.processing.categorical.dictionary_encoding_processor
          • cursus.processing.categorical.multiclass_label_processor
          • cursus.processing.categorical.numerical_categorical_processor
          • cursus.processing.categorical.risk_table_processor
          • cursus.processing.categorical.streaming_risk_table_processor
        • cursus.processing.dataloaders
          • cursus.processing.dataloaders.pipeline_dataloader
        • cursus.processing.datasets
          • cursus.processing.datasets.pipeline_datasets
          • cursus.processing.datasets.pipeline_iterable_datasets
        • cursus.processing.numerical
          • cursus.processing.numerical.feature_normalization_processor
          • cursus.processing.numerical.minmax_scaling_processor
          • cursus.processing.numerical.numerical_binning_processor
          • cursus.processing.numerical.numerical_imputation_processor
          • cursus.processing.numerical.streaming_numerical_imputation_processor
        • cursus.processing.processor_registry
        • cursus.processing.processors
        • cursus.processing.temporal
          • cursus.processing.temporal.sequence_ordering_processor
          • cursus.processing.temporal.sequence_padding_processor
          • cursus.processing.temporal.temporal_mask_processor
          • cursus.processing.temporal.time_delta_processor
        • cursus.processing.text
          • cursus.processing.text.bert_tokenize_processor
          • cursus.processing.text.cs_format_processor
          • cursus.processing.text.dialogue_processor
          • cursus.processing.text.gensim_tokenize_processor
        • cursus.processing.validation
      • cursus.registry
        • cursus.registry.exceptions
        • cursus.registry.hybrid
          • cursus.registry.hybrid.manager
          • cursus.registry.hybrid.models
          • cursus.registry.hybrid.setup
          • cursus.registry.hybrid.utils
        • cursus.registry.interface_registry_loader
        • cursus.registry.step_names
        • cursus.registry.step_names_base
        • cursus.registry.strategy_registry
        • cursus.registry.validation_utils
      • cursus.step_catalog
        • cursus.step_catalog.adapters
          • cursus.step_catalog.adapters.config_class_detector
          • cursus.step_catalog.adapters.config_resolver
          • cursus.step_catalog.adapters.contract_adapter
          • cursus.step_catalog.adapters.file_resolver
          • cursus.step_catalog.adapters.legacy_wrappers
          • cursus.step_catalog.adapters.workspace_discovery
        • cursus.step_catalog.builder_discovery
        • cursus.step_catalog.config_discovery
        • cursus.step_catalog.contract_discovery
        • cursus.step_catalog.mapping
        • cursus.step_catalog.models
        • cursus.step_catalog.naming
        • cursus.step_catalog.script_discovery
        • cursus.step_catalog.sdk_bindings
        • cursus.step_catalog.spec_discovery
        • cursus.step_catalog.step_catalog
      • cursus.steps
        • cursus.steps.configs
          • cursus.steps.configs.config_active_sample_selection_step
          • cursus.steps.configs.config_batch_transform_step
          • cursus.steps.configs.config_bedrock_batch_processing_step
          • cursus.steps.configs.config_bedrock_processing_step
          • cursus.steps.configs.config_bedrock_prompt_template_generation_step
          • cursus.steps.configs.config_cradle_data_loading_step
          • cursus.steps.configs.config_currency_conversion_step
          • cursus.steps.configs.config_data_uploading_step
          • cursus.steps.configs.config_dummy_data_loading_step
          • cursus.steps.configs.config_dummy_training_step
          • cursus.steps.configs.config_edx_uploading_step
          • cursus.steps.configs.config_feature_selection_step
          • cursus.steps.configs.config_label_ruleset_execution_step
          • cursus.steps.configs.config_label_ruleset_generation_step
          • cursus.steps.configs.config_lightgbm_model_eval_step
          • cursus.steps.configs.config_lightgbm_model_inference_step
          • cursus.steps.configs.config_lightgbm_training_step
          • cursus.steps.configs.config_lightgbmmt_model_eval_step
          • cursus.steps.configs.config_lightgbmmt_model_inference_step
          • cursus.steps.configs.config_lightgbmmt_training_step
          • cursus.steps.configs.config_missing_value_imputation_step
          • cursus.steps.configs.config_model_calibration_step
          • cursus.steps.configs.config_model_metrics_computation_step
          • cursus.steps.configs.config_model_wiki_generator_step
          • cursus.steps.configs.config_package_step
          • cursus.steps.configs.config_payload_step
          • cursus.steps.configs.config_percentile_model_calibration_step
          • cursus.steps.configs.config_piper_metric_generation_step
          • cursus.steps.configs.config_processing_step_base
          • cursus.steps.configs.config_pseudo_label_merge_step
          • cursus.steps.configs.config_pytorch_model_eval_step
          • cursus.steps.configs.config_pytorch_model_inference_step
          • cursus.steps.configs.config_pytorch_model_step
          • cursus.steps.configs.config_pytorch_training_step
          • cursus.steps.configs.config_redshift_data_loading_step
          • cursus.steps.configs.config_registration_step
          • cursus.steps.configs.config_risk_table_mapping_step
          • cursus.steps.configs.config_sopa_instruction_tuning_step
          • cursus.steps.configs.config_stratified_sampling_step
          • cursus.steps.configs.config_tabular_preprocessing_step
          • cursus.steps.configs.config_temporal_feature_engineering_step
          • cursus.steps.configs.config_temporal_sequence_normalization_step
          • cursus.steps.configs.config_temporal_split_preprocessing_step
          • cursus.steps.configs.config_tokenizer_training_step
          • cursus.steps.configs.config_tsa_model_calibration_step
          • cursus.steps.configs.config_tsa_model_eval_step
          • cursus.steps.configs.config_tsa_preprocessing_step
          • cursus.steps.configs.config_tsa_tabular_preprocessing_step
          • cursus.steps.configs.config_tsa_training_step
          • cursus.steps.configs.config_xgboost_model_eval_step
          • cursus.steps.configs.config_xgboost_model_inference_step
          • cursus.steps.configs.config_xgboost_model_step
          • cursus.steps.configs.config_xgboost_mt_model_eval_step
          • cursus.steps.configs.config_xgboost_mt_training_step
          • cursus.steps.configs.config_xgboost_training_step
          • cursus.steps.configs.utils
        • cursus.steps.hyperparams
          • cursus.steps.hyperparams.hyperparameters_bimodal
          • cursus.steps.hyperparams.hyperparameters_dual_sequence_tsa
          • cursus.steps.hyperparams.hyperparameters_lightgbm
          • cursus.steps.hyperparams.hyperparameters_lightgbmmt
          • cursus.steps.hyperparams.hyperparameters_lstm2risk
          • cursus.steps.hyperparams.hyperparameters_transformer2risk
          • cursus.steps.hyperparams.hyperparameters_trimodal
          • cursus.steps.hyperparams.hyperparameters_tsa
          • cursus.steps.hyperparams.hyperparameters_xgboost
          • cursus.steps.hyperparams.hyperparameters_xgboost_mt
        • cursus.steps.interfaces
          • cursus.steps.interfaces.io_view
        • cursus.steps.scripts
          • cursus.steps.scripts.active_sample_selection
          • cursus.steps.scripts.bedrock_prompt_template_generation
          • cursus.steps.scripts.bspline_calibration
          • cursus.steps.scripts.currency_conversion
          • cursus.steps.scripts.dummy_data_loading
          • cursus.steps.scripts.dummy_training
          • cursus.steps.scripts.feature_selection
          • cursus.steps.scripts.label_ruleset_execution
          • cursus.steps.scripts.label_ruleset_generation
          • cursus.steps.scripts.missing_value_imputation
          • cursus.steps.scripts.model_calibration
          • cursus.steps.scripts.model_metrics_computation
          • cursus.steps.scripts.model_wiki_generator
          • cursus.steps.scripts.package
          • cursus.steps.scripts.payload
          • cursus.steps.scripts.percentile_model_calibration
          • cursus.steps.scripts.piper_metric_generation
          • cursus.steps.scripts.pseudo_label_merge
          • cursus.steps.scripts.risk_table_mapping
          • cursus.steps.scripts.stratified_sampling
          • cursus.steps.scripts.tabular_preprocessing
          • cursus.steps.scripts.temporal_feature_engineering
          • cursus.steps.scripts.temporal_sequence_normalization
          • cursus.steps.scripts.temporal_split_preprocessing
          • cursus.steps.scripts.xgboost_model_eval
          • cursus.steps.scripts.xgboost_model_inference
        • cursus.steps.utils
          • cursus.steps.utils.s3_utils
      • cursus.validation
        • cursus.validation.alignment
          • cursus.validation.alignment.analyzer
            • cursus.validation.alignment.analyzer.script_analyzer
          • cursus.validation.alignment.config
            • cursus.validation.alignment.config.step_type_specific_rules
            • cursus.validation.alignment.config.universal_builder_rules
            • cursus.validation.alignment.config.validation_ruleset
          • cursus.validation.alignment.core
            • cursus.validation.alignment.core.level3_validation_config
            • cursus.validation.alignment.core.level_validators
            • cursus.validation.alignment.core.script_contract_alignment
            • cursus.validation.alignment.core.spec_dependency_alignment
          • cursus.validation.alignment.reporting
          • cursus.validation.alignment.unified_alignment_tester
          • cursus.validation.alignment.utils
            • cursus.validation.alignment.utils.utils
            • cursus.validation.alignment.utils.validation_models
          • cursus.validation.alignment.validators
            • cursus.validation.alignment.validators.dependency_validator
            • cursus.validation.alignment.validators.property_path_validator
            • cursus.validation.alignment.validators.registry_binding_validator
        • cursus.validation.builders
          • cursus.validation.builders.reporting
            • cursus.validation.builders.reporting.builder_reporter
            • cursus.validation.builders.reporting.scoring
          • cursus.validation.builders.universal_test
        • cursus.validation.script_testing
          • cursus.validation.script_testing.api
          • cursus.validation.script_testing.input_collector
          • cursus.validation.script_testing.result_formatter
          • cursus.validation.script_testing.script_dependency_matcher
          • cursus.validation.script_testing.script_execution_registry
          • cursus.validation.script_testing.script_input_resolver
          • cursus.validation.script_testing.utils
        • cursus.validation.utils
  • Command-Line Interface
  • Reference
    • MCP Tool Reference
    • Step Interface Catalog
    • Pipeline Catalog
Back to top
View this page
Edit this page

cursus.processing.textΒΆ

Modules

bert_tokenize_processor

cs_format_processor

dialogue_processor

gensim_tokenize_processor

Next
cursus.processing.text.bert_tokenize_processor
Previous
cursus.processing.temporal.time_delta_processor
Copyright © 2026, Tianpei Xie
Made with Sphinx and @pradyunsg's Furo