trinity.algorithm.advantage_fn package

Submodules

• trinity.algorithm.advantage_fn.advantage_fn module
  • AdvantageFn
    • AdvantageFn.default_args()
    • AdvantageFn.compute_in_trainer()
  • GroupAdvantage
    • GroupAdvantage.group_experiences()
    • GroupAdvantage.calculate_group_advantage()
    • GroupAdvantage.process()
    • GroupAdvantage.compute_in_trainer()
• trinity.algorithm.advantage_fn.asymre_advantage module
  • ASYMREAdvantageFn
    • ASYMREAdvantageFn.__init__()
    • ASYMREAdvantageFn.default_args()
  • ASYMREGroupAdvantage
    • ASYMREGroupAdvantage.__init__()
    • ASYMREGroupAdvantage.group_experiences()
    • ASYMREGroupAdvantage.calculate_group_advantage()
    • ASYMREGroupAdvantage.default_args()
• trinity.algorithm.advantage_fn.grpo_advantage module
  • GRPOAdvantageFn
    • GRPOAdvantageFn.__init__()
    • GRPOAdvantageFn.default_args()
  • GRPOGroupedAdvantage
    • GRPOGroupedAdvantage.__init__()
    • GRPOGroupedAdvantage.group_experiences()
    • GRPOGroupedAdvantage.calculate_group_advantage()
    • GRPOGroupedAdvantage.process()
    • GRPOGroupedAdvantage.default_args()
• trinity.algorithm.advantage_fn.multi_step_grpo_advantage module
  • StepWiseGRPOAdvantageFn
    • StepWiseGRPOAdvantageFn.__init__()
    • StepWiseGRPOAdvantageFn.calculate_last_step_advantage()
    • StepWiseGRPOAdvantageFn.broadcast_advantages()
    • StepWiseGRPOAdvantageFn.process()
    • StepWiseGRPOAdvantageFn.compute_in_trainer()
    • StepWiseGRPOAdvantageFn.default_args()
• trinity.algorithm.advantage_fn.on_policy_distill_advantage module
  • OnPolicyDistillAdvantage
    • OnPolicyDistillAdvantage.__init__()
    • OnPolicyDistillAdvantage.default_args()
• trinity.algorithm.advantage_fn.opmd_advantage module
  • OPMDAdvantageFn
    • OPMDAdvantageFn.__init__()
    • OPMDAdvantageFn.default_args()
  • OPMDGroupAdvantage
    • OPMDGroupAdvantage.__init__()
    • OPMDGroupAdvantage.group_experiences()
    • OPMDGroupAdvantage.calculate_group_advantage()
    • OPMDGroupAdvantage.default_args()
• trinity.algorithm.advantage_fn.ppo_advantage module
  • PPOAdvantageFn
    • PPOAdvantageFn.__init__()
    • PPOAdvantageFn.default_args()
• trinity.algorithm.advantage_fn.rec_advantage module
  • RECGroupedAdvantage
    • RECGroupedAdvantage.__init__()
    • RECGroupedAdvantage.group_experiences()
    • RECGroupedAdvantage.calculate_group_advantage()
    • RECGroupedAdvantage.default_args()
• trinity.algorithm.advantage_fn.reinforce_advantage module
  • REINFORCEGroupAdvantage
    • REINFORCEGroupAdvantage.group_experiences()
    • REINFORCEGroupAdvantage.calculate_group_advantage()
    • REINFORCEGroupAdvantage.default_args()
• trinity.algorithm.advantage_fn.reinforce_plus_plus_advantage module
  • REINFORCEPLUSPLUSAdvantageFn
    • REINFORCEPLUSPLUSAdvantageFn.__init__()
    • REINFORCEPLUSPLUSAdvantageFn.default_args()
• trinity.algorithm.advantage_fn.remax_advantage module
  • REMAXAdvantageFn
    • REMAXAdvantageFn.__init__()
    • REMAXAdvantageFn.default_args()
• trinity.algorithm.advantage_fn.rloo_advantage module
  • RLOOAdvantageFn
    • RLOOAdvantageFn.__init__()
    • RLOOAdvantageFn.default_args()

Module contents

class trinity.algorithm.advantage_fn.AdvantageFn

Bases: ABC

abstract classmethod default_args() → Dict

Returns:

The default init arguments for the advantage function.

Return type:

Dict

classmethod compute_in_trainer() → bool

Whether the advantage should be computed in the trainer loop.

class trinity.algorithm.advantage_fn.GroupAdvantage

Bases: AdvantageFn, ExperienceOperator

For group-based advantages calculation.

abstract group_experiences(exps: List[Experience]) → Dict[str, List[Experience]]

Group experiences by a certain criterion.

Parameters:

exps (List[Experience]) – List of experiences to be grouped.

Returns:

A dictionary where keys are group identifiers and values are lists of experiences.

Return type:

Dict[str, List[Experience]]

abstract calculate_group_advantage(group_id: str, exps: List[Experience]) → Tuple[List[Experience], Dict]

Calculate advantages for a group of experiences.

Parameters:

• group_id (str) – The identifier for the group of experiences.

• exps (List[Experience]) – List of experiences in the group.

Returns:

A tuple containing the modified list of experiences and a dictionary of metrics.

Return type:

List[Experience]

process(exps: List[Experience]) → Tuple[List[Experience], Dict]

Process a list of experiences and return a transformed list.

Parameters:

exps (List[Experience]) – List of experiences to process, which contains all experiences generated by the Explorer in one explore step.

Returns:

A tuple containing the processed list of experiences and a dictionary of metrics.

Return type:

Tuple[List[Experience], Dict]

classmethod compute_in_trainer() → bool

Whether the advantage should be computed in the trainer loop.