cursus.processing.processor_registry

Processor Registry for Dynamic Pipeline Construction

Maps hyperparameter step names to processor classes for flexible pipeline composition based on configuration.

build_text_pipeline_from_steps(processing_steps, tokenizer=None, max_sen_len=512, chunk_trancate=False, max_total_chunks=5, input_ids_key='input_ids', attention_mask_key='attention_mask')[source]

Build a text processing pipeline from a list of step names.

Parameters:

  • processing_steps (list[str]) – List of processor names from hyperparameters

  • tokenizer (Any | None) – HuggingFace tokenizer (optional, required for chunker/tokenizer steps)

  • max_sen_len (int) – Maximum sentence length for chunker and tokenizer

  • chunk_trancate (bool) – Whether to truncate chunks

  • max_total_chunks (int) – Maximum number of chunks

  • input_ids_key (str) – Key for input IDs (for trimodal support)

  • attention_mask_key (str) – Key for attention mask (for trimodal support)

Returns:

Chained processor pipeline

Raises:

  • ValueError – If unknown processing step is encountered

  • ImportError – If required library not available for specific steps

Return type:

Processor