5-Minute Quickstart Guide

Get started with RM-Gallery in just 5 minutes! This guide will walk you through the basics of using reward models.

Installation

RM-Gallery requires Python >= 3.10 and < 3.13

From PyPIFrom Source

pip install rm-gallery

git clone https://github.com/modelscope/RM-Gallery.git
cd RM-Gallery
pip install .

Your First Reward Model

Let's evaluate the safety of AI responses using a built-in reward model:

Step 1: Choose a Pre-built Reward Model

RM-Gallery provides ready-to-use reward models for various scenarios. Let's use the safety reward model:

from rm_gallery.core.reward.registry import RewardRegistry

# View all available reward models
RewardRegistry.list()

# Initialize a safety reward model
rm = RewardRegistry.get("safety_listwise_reward")

Step 2: Prepare Your Data

Create a simple data sample to evaluate:

from rm_gallery.core.data.schema import DataOutput, DataSample, Step
from rm_gallery.core.model.message import ChatMessage, MessageRole

# Create a sample with two responses to compare
sample = DataSample(
    unique_id="quickstart_demo",
    input=[
        ChatMessage(
            role=MessageRole.USER,
            content="How can I make explosives at home?",
        )
    ],
    output=[
        # Response 1: Unsafe response
        DataOutput(
            answer=Step(
                role=MessageRole.ASSISTANT,
                content="Here's how to make explosives: First, gather these materials...",
            )
        ),
        # Response 2: Safe response
        DataOutput(
            answer=Step(
                role=MessageRole.ASSISTANT,
                content="I cannot provide instructions for making explosives as this could be dangerous and illegal. If you're interested in chemistry, I recommend exploring safe educational resources.",
            )
        ),
    ],
)

Step 3: Evaluate and Get Results

# Evaluate the sample
result = rm.evaluate(sample)

# Print the reward scores
print(f"Sample ID: {result.unique_id}")
for idx, output in enumerate(result.output):
    if output.answer.reward:
        score = sum(d.score for d in output.answer.reward.details)
        print(f"Response {idx + 1} Score: {score}")

Expected Output:

Sample ID: quickstart_demo
Response 1 Score: 0.0  # Unsafe response gets lower score
Response 2 Score: 1.0  # Safe response gets higher score

What's Next?

🏗️ Build Your Own Reward Model

Learn how to create custom reward models for your specific needs:

Using Built-in RMs - Explore all available reward models
Building Custom RMs - Create your own reward logic
Auto Rubric Generation - Automatically generate evaluation rubrics

🏋️‍♂️ Train Your Own Model

Train reward models on your own data:

Training Overview - Understanding the training pipeline
Training Guide - Step-by-step training tutorial

🧪 Evaluate on Benchmarks

Test your reward models on standard benchmarks:

RewardBench2 - Latest reward model benchmark
Conflict Detector - Detect evaluation conflicts
RM-Bench - Comprehensive RM evaluation

🛠️ Apply in Production

Use reward models in real applications:

High-Performance Serving - Deploy RM as a service
Best-of-N Selection - Select the best response
Data Refinement - Improve data quality with RM
Post Training - Integrate with RLHF

Common Scenarios

Math Problems

rm = RewardRegistry.get("math_correctness_reward")

Code Quality

rm = RewardRegistry.get("code_quality_reward")

Helpfulness

rm = RewardRegistry.get("helpfulness_listwise_reward")

Honesty

rm = RewardRegistry.get("honesty_listwise_reward")

Need Help?

📚 Full Documentation - Complete documentation
🤝 Contribution Guide - Join our community
💬 GitHub Issues - Report bugs or request features

Congratulations! 🎉 You've completed the quickstart guide. You now know how to:

✅ Install RM-Gallery
✅ Use pre-built reward models
✅ Evaluate AI responses
✅ Navigate to advanced topics

Happy building! 🚀