Welcome! OpenJudge is an open-source reward model platform. Your contributions help make AI alignment and evaluation more accessible to the community.

Set Up Development Environment

git clone https://github.com/YOUR_USERNAME/OpenJudge.git
cd OpenJudge

pip install -e ".[dev]"

python -c "from openjudge.graders.common import RelevanceGrader; print('✓ Installation successful')"

Follow Code Standards

Testing Guidelines

Before submitting, verify your changes work:

# Verify installation
python -c "from openjudge.graders.common import RelevanceGrader; print('✓ Works')"

# Run tests (optional)
pytest tests/ -v

To ensure the quality and reliability of OpenJudge project, all key components require proper testing. LLMGraders specifically must pass both unit and quality testing.

Unit Testing

Unit tests verify individual components of your code. Each component should include tests for initialization, successful operation, edge cases, and error handling.

import pytest
from unittest.mock import AsyncMock
from openjudge.graders.category.grader_module import YourGrader

@pytest.mark.unit
class TestYourGrader:
    def test_initialization(self):
        """Test successful initialization with required parameters"""
        ...

    @pytest.mark.asyncio
    async def test_successful_operation(self):
        """Test successful operation with valid inputs"""
        ...

    @pytest.mark.asyncio
    async def test_error_handling(self):
        """Test graceful error handling when dependencies fail"""
        ...

All external services (especially LLM APIs) must be mocked to enable offline testing.

Quality Testing

For LLMGraders, quality testing ensures the grader correctly evaluates response quality by comparing against human annotations or expected standards.

Environment Configuration

Quality tests require real API keys and detect them via environment variables:

import os
import pytest

# Check for required environment variables
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
OPENAI_BASE_URL = os.getenv("OPENAI_BASE_URL")
RUN_QUALITY_TESTS = bool(OPENAI_API_KEY and OPENAI_BASE_URL)

# Skip tests if API keys are not configured
@pytest.mark.skipif(not RUN_QUALITY_TESTS, reason="Requires API keys to run quality tests")
@pytest.mark.quality
class TestGraderQuality:
    @pytest.fixture
    def dataset(self):
        """Load dataset for quality testing"""
        ...

    @pytest.fixture
    def model(self):
        """Create a model instance for testing"""
        ...

    @pytest.mark.asyncio
    async def test_discriminative_power_with_runner(self, dataset, model):
        """Test discriminative power of the model against a dataset"""
        ...

    @pytest.mark.asyncio
    async def test_consistency_with_runner(self, dataset, model):
        """Test consistency of the model against a dataset"""
        ...

Contributing Graders

When contributing a new grader, follow these guidelines:

Grader Categories

Place your grader in the appropriate category:

Grader Specifications

Grader Method Parameter Naming Convention

To ensure consistency and usability, the aevaluate method of Graders should follow these common parameter naming conventions:

Minimal Grader Template

from openjudge.graders.base_grader import BaseGrader
from openjudge.graders.schema import GraderScore

class MyGrader(BaseGrader):
    """Evaluate [specific aspect].

    Args:
        model: LLM model for evaluation
    """

    def __init__(self, model, **kwargs):
        super().__init__(name="my_grader", **kwargs)
        self.model = model

    async def aevaluate(self, query: str, response: str, **kwargs) -> GraderScore:
        """Evaluate response quality."""
        # Your evaluation logic
        score = 0.8
        reason = "Evaluation explanation"

        return GraderScore(name=self.name, score=score, reason=reason)

Submit Your Contribution

Contribute Documentation

Get Help

Need assistance? Here's how to reach us:

Thank you for contributing to OpenJudge!