Multi-Turn Rollout

The Rollout module provides multi-turn conversation rollout engines for agentic RLHF training. Two implementations are available: MultiTurnRollout for batched vLLM sampling and APIMultiTurnRollout for OpenAI-compatible API endpoints.

Rollout Base Class

from abc import ABC, abstractmethod
from twinkle.data_format import Trajectory

class Rollout(ABC):

    @abstractmethod
    def __call__(self, trajectories: List[Trajectory], **kwargs) -> List[Trajectory]:
        raise NotImplementedError()

All rollouts accept a list of trajectories and return the same number of trajectories with additional fields (messages, turns, stop_reason, truncated).

MultiTurnRollout

Batched multi-turn rollout engine that uses a vLLM sampler for generation. All active trajectories are sampled in a single batched call per turn for maximum throughput.

Per-turn Loop

1. Encode each trajectory into an InputFeature with a generation prompt
2. Batch sampler.sample(active_pifs) — all live trajectories in parallel
3. Check termination: stop_reason == 'length', no tool calls, or max turns reached
4. Dispatch tools via ToolManager, append tool responses
5. Compute bridge tokens (tool turns + generation prompt) with labels = -100
6. Repeat until all trajectories are done

from twinkle_agentic.rollout.multi_turn import MultiTurnRollout
from twinkle_agentic.tools.tool_manager import ToolManager
from twinkle.data_format.sampling import SamplingParams

rollout = MultiTurnRollout(
    sampler=vllm_sampler,
    template=template,
    tool_manager=tool_manager,
    sampling_params=SamplingParams(temperature=0.7, max_tokens=4096),
    max_turns=6,
    max_trajectory_tokens=8192,
    trace_dir='rollout_traces/',
)

# Run rollout
results = rollout(trajectories)

Parameters

Output Fields

Each output trajectory dict includes:

Ray Remote Support

MultiTurnRollout is decorated with @remote_class(), enabling transparent deployment as a Ray actor:

# The rollout can run as a Ray remote actor
rollout_actor = MultiTurnRollout.remote(sampler=sampler, template=template, ...)
results = ray.get(rollout_actor.__call__.remote(trajectories))

APIMultiTurnRollout

Multi-turn rollout over an OpenAI-compatible chat-completions API. Each trajectory runs independently in a thread pool for network concurrency.

from twinkle_agentic.rollout.api_multi_turn import APIMultiTurnRollout
from twinkle_agentic.protocol.openai import OpenAI

api = OpenAI(model='qwen3.5-32b', base_url='http://localhost:8000/v1')

rollout = APIMultiTurnRollout(
    api=api,
    tool_manager=tool_manager,
    sampling_params=SamplingParams(temperature=0.7),
    max_turns=6,
    concurrency=8,
    trace_dir='api_traces/',
)

results = rollout(trajectories)

Parameters

Stop Reasons

Choosing Between Rollouts