data_juicer.ops.mapper.query_intent_detection_mapper module

class data_juicer.ops.mapper.query_intent_detection_mapper.QueryIntentDetectionMapper(hf_model: str = 'bespin-global/klue-roberta-small-3i4k-intent-classification', zh_to_en_hf_model: str | None = 'Helsinki-NLP/opus-mt-zh-en', model_params: Dict = {}, zh_to_en_model_params: Dict = {}, *, label_key: str = 'query_intent_label', score_key: str = 'query_intent_label_score', **kwargs)[source]

Bases: Mapper

Mapper to predict user’s Intent label in query. Input from query_key. Output intent label and corresponding score for the query.

__init__(hf_model: str = 'bespin-global/klue-roberta-small-3i4k-intent-classification', zh_to_en_hf_model: str | None = 'Helsinki-NLP/opus-mt-zh-en', model_params: Dict = {}, zh_to_en_model_params: Dict = {}, *, label_key: str = 'query_intent_label', score_key: str = 'query_intent_label_score', **kwargs)[source]

Initialization method.

Parameters:

  • hf_model – Huggingface model ID to predict intent label.

  • zh_to_en_hf_model – Translation model from Chinese to English. If not None, translate the query from Chinese to English.

  • model_params – model param for hf_model.

  • zh_to_en_model_params – model param for zh_to_hf_model.

  • label_key – The key name in the meta field to store the output label. It is ‘query_intent_label’ in default.

  • score_key – The key name in the meta field to store the corresponding label score. It is ‘query_intent_label_score’ in default.

  • kwargs – Extra keyword arguments.

process_batched(samples, rank=None)[source]