trinity.common.workflows.eval_workflow module

trinity.common.workflows.eval_workflow module#

Evaluation Workflow Class

class trinity.common.workflows.eval_workflow.MathEvalWorkflow(*, task: Task, model: ModelWrapper, auxiliary_models: List[OpenAI] | None = None)[source]#

Bases: Workflow

A workflow for standard math evaluation.

The evaluation standard and prompting style are follow the Qwen2.5-Math model’s evaluation methodology. For more details on their approach, see: QwenLM/Qwen2.5-Math

__init__(*, task: Task, model: ModelWrapper, auxiliary_models: List[OpenAI] | None = None)[source]#
property resettable#
property repeatable#

A workflow is repeatable if it can be run multiple times within the run() or run_async() method.

format_messages()[source]#

Format message for the evaluation of qwen_boxed type.

run() → List[Experience][source]#

Run workflow and return a list of experiences.

Contents