Dataset Configuration Guide

This guide provides an overview of how to configure datasets using YAML format in the Data-Juicer framework. The configurations allow you to specify local and remote datasets, with data validation rules.

Supported Dataset Formats

Local Dataset

The local_json.yaml configuration file is used to specify datasets stored locally in JSON format. path is required to specify the local dataset path, either a single file or a directory. format is optional to specify the dataset format. For local files, DJ will automatically detect the file format and load the dataset accordingly. Formats like parquet, jsonl, json, csv, tsv, txt, and jsonl.gz are supported Refer to local_json.yaml for more details.

dataset:
  configs:
    - type: local
      path: path/to/your/local/dataset.json
      format: json

dataset:
  configs:
    - type: local
      path: path/to/your/local/dataset.parquet
      format: parquet

Remote Huggingface Dataset

The remote_huggingface.yaml configuration file is used to specify huggingface datasets. type and source are fixed to ‘remote’ and ‘huggingface’ to locate huggingface loading logic. path is required to identify the huggingface dataset. name, split and limit are optional to specify the dataset name/split and limit the number of samples to load. Refer to remote_huggingface.yaml for more details.

dataset:
  configs:
    - type: 'remote'
      source: 'huggingface'
      path: "HuggingFaceFW/fineweb"
      name: "CC-MAIN-2024-10"
      split: "train"
      limit: 1000

Remote Arxiv Dataset

The remote_arxiv.yaml configuration file is used to specify datasets stored remotely in JSON format. type and source are fixed to ‘remote’ and ‘arxiv’ to locate arxiv loading logic. lang, dump_date, force_download and url_limit are optional to specify the dataset language, dump date, force download and url limit. Refer to remote_arxiv.yaml for more details.

dataset:
  configs:
    - type: 'remote'
      source: 'arxiv'
      lang: 'en'
      dump_date: 'latest'
      force_download: false
      url_limit: 2

Other Supported Dataset Formats

Refer to load_strategy.py for more details and supported dataset formats.

Other features

Data Mixture

The mixture.yaml configuration file demonstrates how to specify data mixture rules. DJ will mix the datasets by sampling a portion of the dataset and applying proper weights. Refer to mixture.yaml for more details.

dataset:
  max_sample_num: 10000
  configs:
    - type: 'local'
      weight: 1.0
      path: 'path/to/json/file'
    - type: 'local'
      weight: 1.0
      path: 'path/to/csv/file'

Data Validation

The validator.yaml configuration file demonstrates how to specify data validation rules. DJ will validate the dataset by sampling a portion of the dataset and applying the validation rules. Refer to data_validator.py for more details and supported validators.

dataset:
  configs:
    - type: local
      path: path/to/data.json

validators:
  - type: swift_messages
    min_turns: 2
    max_turns: 20
    sample_size: 1000
  - type: required_fields
    required_fields:
      - "text"
      - "metadata"
      - "language"
    field_types:
      text: "str"
      metadata: "dict"
      language: "str"

Legacy dataset_path Configuration

The dataset_path configuration is the original way to specify the dataset path. It’s simplistic and easy to use, but lacks flexibility. It can be used in yaml or command line input. Some examples:

Command line input:

# command line input
dj-process --dataset_path path/to/your/dataset.json

# command line input with weights
dj-process --dataset_path 0.5 path/to/your/dataset1.json 0.5 path/to/your/dataset2.json

Yaml input:

dataset_path: path/to/your/dataset.json