trinity.buffer.pipelines.task_pipeline module

trinity.buffer.pipelines.task_pipeline.check_and_run_task_pipeline(config: Config) → Dict

class trinity.buffer.pipelines.task_pipeline.TaskPipeline(config: Config)

Bases: object

A class to process task datasets through DataJuicer.

__init__(config: Config)

convert_pipeline_config(pipeline_config: TaskPipelineConfig) → Dict[str, Any]

Convert the TaskPipelineConfig to a format suitable for DataJuicer.

process() → Dict[str, Any]

Process the task datasets using DataJuicer.

Returns:

Metrics for logging.

Return type:

Dict[str, Any]

close()