data_juicer.core.exporter module

class data_juicer.core.exporter.Exporter(export_path, export_type=None, export_shard_size=0, export_in_parallel=True, num_proc=1, export_ds=True, keep_stats_in_res_ds=False, keep_hashes_in_res_ds=False, export_stats=True, **kwargs)

基类：object

The Exporter class is used to export a dataset to files of specific format.

__init__(export_path, export_type=None, export_shard_size=0, export_in_parallel=True, num_proc=1, export_ds=True, keep_stats_in_res_ds=False, keep_hashes_in_res_ds=False, export_stats=True, **kwargs)

Initialization method.

参数:

• export_path -- the path to export datasets.

• export_type -- the format type of the exported datasets.

• export_shard_size -- the approximate size of each shard of exported dataset. In default, it's 0, which means export the dataset to a single file.

• export_in_parallel -- whether to export the datasets in parallel.

• num_proc -- number of process to export the dataset.

• export_ds -- whether to export the dataset contents.

• keep_stats_in_res_ds -- whether to keep stats in the result dataset.

• keep_hashes_in_res_ds -- whether to keep hashes in the result dataset.

• export_stats -- whether to export the stats of dataset.

export(dataset)

Export method for a dataset.

参数:

dataset -- the dataset to export.

返回:

export_compute_stats(dataset, export_path)

Export method for saving compute status in filters

static to_jsonl(dataset, export_path, num_proc=1, **kwargs)

Export method for jsonl target files.

参数:

• dataset -- the dataset to export.

• export_path -- the path to store the exported dataset.

• num_proc -- the number of processes used to export the dataset.

• kwargs -- extra arguments.

返回:

static to_json(dataset, export_path, num_proc=1, **kwargs)

Export method for json target files.

参数:

• dataset -- the dataset to export.

• export_path -- the path to store the exported dataset.

• num_proc -- the number of processes used to export the dataset.

• kwargs -- extra arguments.

返回:

static to_parquet(dataset, export_path, **kwargs)

Export method for parquet target files.

参数:

• dataset -- the dataset to export.

• export_path -- the path to store the exported dataset.

• kwargs -- extra arguments.

返回: