data_juicer.ops.filter.suffix_filter module¶

class data_juicer.ops.filter.suffix_filter.SuffixFilter(suffixes: str | List[str] = [], *args, **kwargs)[source]¶

Filter to keep samples with specified suffix.

__init__(suffixes: str | List[str] = [], *args, **kwargs)[source]¶

Initialization method.

Parameters:

suffixes – the suffix of text that will be keep. For example: ‘.txt’, ‘txt’ or [‘txt’, ‘.pdf’, ‘docx’]
args – extra args
kwargs – extra args

compute_stats_single(sample)[source]¶

Compute stats for the sample which is used as a metric to decide whether to filter this sample.

Parameters:

sample – input sample.
context – whether to store context information of intermediate vars in the sample temporarily.

Returns:

sample with computed stats

process_single(sample)[source]¶

For sample level, sample –> Boolean.