trinity.common.workflows.eval_workflow module

Evaluation Workflow Class

class trinity.common.workflows.eval_workflow.MathEvalWorkflow(*, task: Task, model: ModelWrapper, auxiliary_models: List[OpenAI] | None = None)[source]

Bases: Workflow

A workflow for standard math evaluation.

The evaluation standard and prompting style are follow the Qwen2.5-Math model’s evaluation methodology. For more details on their approach, see: https://github.com/QwenLM/Qwen2.5-Math

__init__(*, task: Task, model: ModelWrapper, auxiliary_models: List[OpenAI] | None = None)[source]

property resettable

property repeatable: A workflow is repeatable if it can be run multiple times within the run() method.

format_messages()[source]: Format message for the evaluation of qwen_boxed type.

run() → List[Experience][source]: Run workflow and return a list of experiences.