data_juicer.format.tsv_formatter module

class data_juicer.format.tsv_formatter.TsvFormatter(dataset_path, suffixes=None, **kwargs)

Bases: LocalFormatter

The class is used to load and format tsv-type files.

Default suffixes is [‘.tsv’]

SUFFIXES = ['.tsv']

__init__(dataset_path, suffixes=None, **kwargs)

Initialization method.

Parameters:

• dataset_path – a dataset file or a dataset directory

• suffixes – files with specified suffixes to be processed

• kwargs – extra args, e.g. delimiter = ‘,’