data_juicer.core.exporter module

class data_juicer.core.exporter.Exporter(export_path, export_type=None, export_shard_size=0, export_in_parallel=True, num_proc=1, export_ds=True, keep_stats_in_res_ds=False, keep_hashes_in_res_ds=False, export_stats=True, **kwargs)[source]

Bases: object

The Exporter class is used to export a dataset to files of specific format.

__init__(export_path, export_type=None, export_shard_size=0, export_in_parallel=True, num_proc=1, export_ds=True, keep_stats_in_res_ds=False, keep_hashes_in_res_ds=False, export_stats=True, **kwargs)[source]

Initialization method.

Parameters:

  • export_path – the path to export datasets.

  • export_type – the format type of the exported datasets.

  • export_shard_size – the approximate size of each shard of exported dataset. In default, it’s 0, which means export the dataset to a single file.

  • export_in_parallel – whether to export the datasets in parallel.

  • num_proc – number of process to export the dataset.

  • export_ds – whether to export the dataset contents.

  • keep_stats_in_res_ds – whether to keep stats in the result dataset.

  • keep_hashes_in_res_ds – whether to keep hashes in the result dataset.

  • export_stats – whether to export the stats of dataset.

export(dataset)[source]

Export method for a dataset.

Parameters:

dataset – the dataset to export.

Returns:

export_compute_stats(dataset, export_path)[source]

Export method for saving compute status in filters

static to_jsonl(dataset, export_path, num_proc=1, **kwargs)[source]

Export method for jsonl target files.

Parameters:

  • dataset – the dataset to export.

  • export_path – the path to store the exported dataset.

  • num_proc – the number of processes used to export the dataset.

  • kwargs – extra arguments.

Returns:

static to_json(dataset, export_path, num_proc=1, **kwargs)[source]

Export method for json target files.

Parameters:

  • dataset – the dataset to export.

  • export_path – the path to store the exported dataset.

  • num_proc – the number of processes used to export the dataset.

  • kwargs – extra arguments.

Returns:

static to_parquet(dataset, export_path, **kwargs)[source]

Export method for parquet target files.

Parameters:

  • dataset – the dataset to export.

  • export_path – the path to store the exported dataset.

  • kwargs – extra arguments.

Returns: