trinity.buffer.pipelines.task_pipeline module

trinity.buffer.pipelines.task_pipeline module#

trinity.buffer.pipelines.task_pipeline.check_and_run_task_pipeline(config: Config) → Dict[source]#
class trinity.buffer.pipelines.task_pipeline.TaskPipeline(config: Config)[source]#

Bases: object

A class to process task datasets through DataJuicer.

__init__(config: Config)[source]#
convert_pipeline_config(pipeline_config: TaskPipelineConfig) → Dict[str, Any][source]#

Convert the TaskPipelineConfig to a format suitable for DataJuicer.

process() → Dict[str, Any][source]#

Process the task datasets using DataJuicer.

Returns:

Metrics for logging.

Return type:

Dict[str, Any]

close()[source]#
Contents