data_juicer.ops.filter.llm_perplexity_filter module

class data_juicer.ops.filter.llm_perplexity_filter.LLMPerplexityFilter(hf_model: str = 'Qwen/Qwen2.5-0.5B', model_params: Dict | None = None, min_score: float = 1.0, max_score: float = 100.0, query_template: str | None = None, response_template: str | None = None, *args, **kwargs)

Bases: Filter

Filter to keep samples with perplexity score, computed using a specified llm, within a specific range.

__init__(hf_model: str = 'Qwen/Qwen2.5-0.5B', model_params: Dict | None = None, min_score: float = 1.0, max_score: float = 100.0, query_template: str | None = None, response_template: str | None = None, *args, **kwargs)

Initialization method.

Parameters:

• hf_model – huggingface embedding model name.

• model_params – Parameters for initializing the API model.

• min_score – Minimum perplexity score.

• max_score – Maximum perplexity score.

• query_template – Template for building the query string.

• response_template – Template for building the response string.

• args – extra args

• kwargs – extra args

sample_with_messages(sample, system_prompt=None)

compute_stats_single(sample, rank=None)

Compute stats for the sample which is used as a metric to decide whether to filter this sample.

Parameters:

• sample – input sample.

• context – whether to store context information of intermediate vars in the sample temporarily.

Returns:

sample with computed stats

process_single(sample)

For sample level, sample –> Boolean.

Parameters:

sample – sample to decide whether to filter

Returns:

true for keeping and false for filtering