data_juicer.ops.mapper.text_chunk_mapper module

class data_juicer.ops.mapper.text_chunk_mapper.TextChunkMapper(max_len: Annotated[int, Gt(gt=0)] | None = None, split_pattern: str | None = '\\n\\n', overlap_len: Annotated[int, Ge(ge=0)] = 0, tokenizer: str | None = None, trust_remote_code: bool = False, *args, **kwargs)

基类：Mapper

Split input text to chunks.

__init__(max_len: Annotated[int, Gt(gt=0)] | None = None, split_pattern: str | None = '\\n\\n', overlap_len: Annotated[int, Ge(ge=0)] = 0, tokenizer: str | None = None, trust_remote_code: bool = False, *args, **kwargs)

Initialization method.

参数:

• max_len -- Split text into multi texts with this max len if not None.

• split_pattern -- Make sure split in this pattern if it is not None and force cut if the length exceeds max_len.

• overlap_len -- Overlap length of the split texts if not split in the split pattern.

• tokenizer -- The tokenizer name of Hugging Face tokenizers. The text length will be calculate as the token num if it is offered. Otherwise, the text length equals to string length. Support tiktoken tokenizer (such as gpt-4o), dashscope tokenizer ( such as qwen2.5-72b-instruct) and huggingface tokenizer.

• args -- extra args

• kwargs -- extra args

Trust_remote_code:

for loading huggingface model

recursively_chunk(text)

get_text_chunks(text, rank=None)

process_batched(samples, rank=None)