data_juicer.format¶
- class data_juicer.format.JsonFormatter(dataset_path, suffixes=None, **kwargs)[source]¶
Bases:
LocalFormatter
The class is used to load and format json-type files.
Default suffixes is [‘.json’, ‘.jsonl’, ‘.jsonl.zst’]
- SUFFIXES = ['.json', '.jsonl', '.jsonl.zst']¶
- class data_juicer.format.LocalFormatter(dataset_path: str, type: str, suffixes: str | List[str] | None = None, text_keys: List[str] | None = None, add_suffix=False, **kwargs)[source]¶
Bases:
BaseFormatter
The class is used to load a dataset from local files or local directory.
- __init__(dataset_path: str, type: str, suffixes: str | List[str] | None = None, text_keys: List[str] | None = None, add_suffix=False, **kwargs)[source]¶
Initialization method.
- Parameters:
dataset_path – path to a dataset file or a dataset directory
type – a packaged dataset module type (json, csv, etc.)
suffixes – files with specified suffixes to be processed
text_keys – key names of field that stores sample text.
add_suffix – whether to add the file suffix to dataset meta info
kwargs – extra args
- class data_juicer.format.RemoteFormatter(dataset_path: str, text_keys: List[str] | None = None, **kwargs)[source]¶
Bases:
BaseFormatter
The class is used to load a dataset from repository of huggingface hub.
- class data_juicer.format.TextFormatter(dataset_path, suffixes=None, add_suffix=False, **kwargs)[source]¶
Bases:
LocalFormatter
The class is used to load and format text-type files.
e.g. [‘.txt’, ‘.pdf’, ‘.cpp’, ‘.docx’]
- SUFFIXES = ['.docx', '.pdf', '.txt', '.md', '.tex', '.asm', '.bat', '.cmd', '.c', '.h', '.cs', '.cpp', '.hpp', '.c++', '.h++', '.cc', '.hh', '.C', '.H', '.cmake', '.css', '.dockerfile', '.f90', '.f', '.f03', '.f08', '.f77', '.f95', '.for', '.fpp', '.go', '.hs', '.html', '.java', '.js', '.jl', '.lua', '.markdown', '.php', '.php3', '.php4', '.php5', '.phps', '.phpt', '.pl', '.pm', '.pod', '.perl', '.ps1', '.psd1', '.psm1', '.py', '.rb', '.rs', '.sql', '.scala', '.sh', '.bash', '.command', '.zsh', '.ts', '.tsx', '.vb', 'Dockerfile', 'Makefile', '.xml', '.rst', '.m', '.smali']¶
- __init__(dataset_path, suffixes=None, add_suffix=False, **kwargs)[source]¶
Initialization method.
- Parameters:
dataset_path – a dataset file or a dataset directory
suffixes – files with specified suffixes to be processed
add_suffix – Whether to add file suffix to dataset meta info
kwargs – extra args
- class data_juicer.format.ParquetFormatter(dataset_path, suffixes=None, **kwargs)[source]¶
Bases:
LocalFormatter
The class is used to load and format parquet-type files.
Default suffixes is [‘.parquet’]
- SUFFIXES = ['.parquet']¶
- class data_juicer.format.CsvFormatter(dataset_path, suffixes=None, **kwargs)[source]¶
Bases:
LocalFormatter
The class is used to load and format csv-type files.
Default suffixes is [‘.csv’]
- SUFFIXES = ['.csv']¶
- class data_juicer.format.TsvFormatter(dataset_path, suffixes=None, **kwargs)[source]¶
Bases:
LocalFormatter
The class is used to load and format tsv-type files.
Default suffixes is [‘.tsv’]
- SUFFIXES = ['.tsv']¶
- class data_juicer.format.EmptyFormatter(length, feature_keys: List[str] = [], *args, **kwargs)[source]¶
Bases:
BaseFormatter
The class is used to create empty data.
- SUFFIXES = []¶
- __init__(length, feature_keys: List[str] = [], *args, **kwargs)[source]¶
Initialization method.
- Parameters:
length – The empty dataset length.
feature_keys – feature key name list.
- property null_value¶
- class data_juicer.format.RayEmptyFormatter(length, feature_keys: List[str] = [], *args, **kwargs)[source]¶
Bases:
BaseFormatter
The class is used to create empty data for ray.
- SUFFIXES = []¶
- __init__(length, feature_keys: List[str] = [], *args, **kwargs)[source]¶
Initialization method.
- Parameters:
length – The empty dataset length.
feature_keys – feature key name list.
- property null_value¶