data_juicer.ops.grouper package

Submodules

data_juicer.ops.grouper.key_value_grouper module

class data_juicer.ops.grouper.key_value_grouper.KeyValueGrouper(group_by_keys: List[str] | None = None, *args, **kwargs)

基类：Grouper

Group samples to batched samples according values in given keys.

__init__(group_by_keys: List[str] | None = None, *args, **kwargs)

Initialization method.

参数:

• group_by_keys -- group samples according values in the keys. Support for nested keys such as "__dj__stats__.text_len". It is [self.text_key] in default.

• args -- extra args

• kwargs -- extra args

process(dataset)

Dataset --> dataset.

参数:

dataset -- input dataset

返回:

dataset of batched samples.

data_juicer.ops.grouper.naive_grouper module

class data_juicer.ops.grouper.naive_grouper.NaiveGrouper(*args, **kwargs)

基类：Grouper

Group all samples to one batched sample.

__init__(*args, **kwargs)

Initialization method.

参数:

• args -- extra args

• kwargs -- extra args

process(dataset)

Dataset --> dataset.

参数:

dataset -- input dataset

返回:

dataset of batched samples.

data_juicer.ops.grouper.naive_reverse_grouper module

class data_juicer.ops.grouper.naive_reverse_grouper.NaiveReverseGrouper(batch_meta_export_path=None, *args, **kwargs)

基类：Grouper

Split batched samples to samples.

__init__(batch_meta_export_path=None, *args, **kwargs)

Initialization method.

参数:

• batch_meta_export_path -- the path to export the batch meta. Just drop the batch meta if it is None.

• args -- extra args

• kwargs -- extra args

process(dataset)

Dataset --> dataset.

参数:

dataset -- input dataset

返回:

dataset of batched samples.

Module contents

class data_juicer.ops.grouper.KeyValueGrouper(group_by_keys: List[str] | None = None, *args, **kwargs)

基类：Grouper

Group samples to batched samples according values in given keys.

__init__(group_by_keys: List[str] | None = None, *args, **kwargs)

Initialization method.

参数:

• group_by_keys -- group samples according values in the keys. Support for nested keys such as "__dj__stats__.text_len". It is [self.text_key] in default.

• args -- extra args

• kwargs -- extra args

process(dataset)

Dataset --> dataset.

参数:

dataset -- input dataset

返回:

dataset of batched samples.

class data_juicer.ops.grouper.NaiveGrouper(*args, **kwargs)

基类：Grouper

Group all samples to one batched sample.

__init__(*args, **kwargs)

Initialization method.

参数:

• args -- extra args

• kwargs -- extra args

process(dataset)

Dataset --> dataset.

参数:

dataset -- input dataset

返回:

dataset of batched samples.

class data_juicer.ops.grouper.NaiveReverseGrouper(batch_meta_export_path=None, *args, **kwargs)

基类：Grouper

Split batched samples to samples.

__init__(batch_meta_export_path=None, *args, **kwargs)

Initialization method.

参数:

• batch_meta_export_path -- the path to export the batch meta. Just drop the batch meta if it is None.

• args -- extra args

• kwargs -- extra args

process(dataset)

Dataset --> dataset.

参数:

dataset -- input dataset

返回:

dataset of batched samples.