trinity.common.workflows.rubric_judge_workflow module

Contents

trinity.common.workflows.rubric_judge_workflow module#

A workflow with LLM-as-a-judge.

class trinity.common.workflows.rubric_judge_workflow.RubricJudgeWorkflow(*, task: Task, model: ModelWrapper, auxiliary_models: List[OpenAI] | None = None)[source]#

Bases: SimpleWorkflow

A workflow using LLM-as-a-judge and rubrics to get the reward.

Adapted from https://arxiv.org/pdf/2507.17746

__init__(*, task: Task, model: ModelWrapper, auxiliary_models: List[OpenAI] | None = None)[source]#

reset(task: Task)[source]#: Modified from SimpleWorkflow.reset

run() → List[Experience][source]#: Modified from SimpleWorkflow.run

get_judge_reward(response: str, judger: OpenAI) → Tuple[bool, float][source]#: Get rewards with LLM-as-a-judge The prompts are adapted from RAR-IMPLICIT method in https://arxiv.org/pdf/2507.17746