trinity.buffer.operators.data_juicer_operator module
- class trinity.buffer.operators.data_juicer_operator.DataJuicerOperator(service_config: DataJuicerServiceConfig, operators: List[Dict] | None = None, config_path: str | None = None, np: int = 4)[source]
Bases:
ExperienceOperator- __init__(service_config: DataJuicerServiceConfig, operators: List[Dict] | None = None, config_path: str | None = None, np: int = 4)[source]
Initialize the DataJuicerOperator.
- Parameters:
service_config (config) – The configuration for the DataJuicer service.
operators (List[Dict]) – A list of operators with their configurations.
config_path (str) – Path to the Data-Juicer configuration file.
np (int) – Number of processes to use for Data-Juicer. Default is 4.
Note
- Must include one of the following, and the priority is from high to low:
config_path (str)
operators (List[Dict])
- process(exps: List[Experience]) Tuple[List[Experience], Dict][source]
Process a list of experiences and return a transformed list.
- Parameters:
exps (List[Experience]) – List of experiences to process, which contains all experiences generated by the Explorer in one explore step.
- Returns:
A tuple containing the processed list of experiences and a dictionary of metrics.
- Return type:
Tuple[List[Experience], Dict]