cursus.processing.text.dialogue_processor¶

class TextNormalizationProcessor[source]¶

Bases: Processor

process(input_text)[source]¶

class TextUpperProcessor[source]¶

Bases: Processor

process(input_text)[source]¶

class DialogueSplitterProcessor(min_length=1)[source]¶

process(input_text)[source]¶

Splits the dialogue into individual messages based on [bom] and [eom] delimiters. :returns: List of message strings.

class DialogueChunkerProcessor(tokenizer, max_tokens=512, truncate=False, max_total_chunks=5)[source]¶

process(messages)[source]¶

Chunks a list of messages into groups such that each chunk’s token count (without special tokens) does not exceed the max_tokens limit.

Returns:: List of dialogue chunks (each chunk is a concatenated string of messages).

class EmojiRemoverProcessor[source]¶

Bases: Processor

process(input_text)[source]¶

class HTMLNormalizerProcessor[source]¶

Bases: Processor

process(input_text)[source]¶

If given a list of dialogue messages, normalize each one; otherwise normalize the single HTML string.