Skip to content

template

Annotation Template System - extensible framework for Label Studio annotation configurations.

Provides base classes and registry for managing annotation templates with custom labeling configurations and post-processing logic for different evaluation tasks.

AnnotationTemplateRegistry

Registry system for managing and discovering annotation templates.

Provides decorator-based registration and factory methods for template instantiation. Enables extensible template ecosystem for different annotation tasks.

Source code in rm_gallery/core/data/annotation/template.py 
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
class AnnotationTemplateRegistry:
    """
    Registry system for managing and discovering annotation templates.

    Provides decorator-based registration and factory methods for template instantiation.
    Enables extensible template ecosystem for different annotation tasks.
    """

    _templates: Dict[str, BaseAnnotationTemplate] = {}

    @classmethod
    def register(
        cls, name: str
    ) -> Callable[[Type[BaseAnnotationTemplate]], Type[BaseAnnotationTemplate]]:
        """
        Decorator for registering annotation templates with unique identifiers.

        Args:
            name: Unique template identifier for registry lookup

        Returns:
            Decorator function that registers the template class and returns it unchanged

        Example:
            @AnnotationTemplateRegistry.register("rewardbench")
            class RewardBenchTemplate(BaseAnnotationTemplate):
                ...
        """

        def decorator(
            template_class: Type[BaseAnnotationTemplate],
        ) -> Type[BaseAnnotationTemplate]:
            # Create an instance of the template with the given name
            template_instance = template_class(name)
            cls._templates[name] = template_instance
            return template_class

        return decorator

    @classmethod
    def get_template(cls, name: str) -> Optional[BaseAnnotationTemplate]:
        """
        Retrieve template instance by name.

        Args:
            name: Template identifier to look up

        Returns:
            Template instance if found, None otherwise
        """
        return cls._templates.get(name)

    @classmethod
    def get_label_config(cls, template_name: str) -> Optional[str]:
        """
        Get Label Studio XML configuration from registered template.

        Args:
            template_name: Name of registered template

        Returns:
            Label Studio XML configuration string if template exists, None otherwise
        """
        template = cls.get_template(template_name)
        return template.label_config if template else None

    @classmethod
    def list_templates(cls) -> list[str]:
        """
        List all registered template names.

        Returns:
            List of registered template identifiers
        """
        return list(cls._templates.keys())

get_label_config(template_name) classmethod

Get Label Studio XML configuration from registered template.

Parameters:

Name Type Description Default
template_name str

Name of registered template

 required

Returns:

Type Description
Optional[str]

Label Studio XML configuration string if template exists, None otherwise
Source code in rm_gallery/core/data/annotation/template.py 
132
133
134
135
136
137
138
139
140
141
142
143
144
@classmethod
def get_label_config(cls, template_name: str) -> Optional[str]:
    """
    Get Label Studio XML configuration from registered template.

    Args:
        template_name: Name of registered template

    Returns:
        Label Studio XML configuration string if template exists, None otherwise
    """
    template = cls.get_template(template_name)
    return template.label_config if template else None

get_template(name) classmethod

Retrieve template instance by name.

Parameters:

Name Type Description Default
name str

Template identifier to look up

 required

Returns:

Type Description
Optional[BaseAnnotationTemplate]

Template instance if found, None otherwise
Source code in rm_gallery/core/data/annotation/template.py 
119
120
121
122
123
124
125
126
127
128
129
130
@classmethod
def get_template(cls, name: str) -> Optional[BaseAnnotationTemplate]:
    """
    Retrieve template instance by name.

    Args:
        name: Template identifier to look up

    Returns:
        Template instance if found, None otherwise
    """
    return cls._templates.get(name)

list_templates() classmethod

List all registered template names.

Returns:

Type Description
list[str]

List of registered template identifiers
Source code in rm_gallery/core/data/annotation/template.py 
146
147
148
149
150
151
152
153
154
@classmethod
def list_templates(cls) -> list[str]:
    """
    List all registered template names.

    Returns:
        List of registered template identifiers
    """
    return list(cls._templates.keys())

register(name) classmethod

Decorator for registering annotation templates with unique identifiers.

Parameters:

Name Type Description Default
name str

Unique template identifier for registry lookup

 required

Returns:

Type Description
Callable[[Type[BaseAnnotationTemplate]], Type[BaseAnnotationTemplate]]

Decorator function that registers the template class and returns it unchanged
Example

@AnnotationTemplateRegistry.register("rewardbench") class RewardBenchTemplate(BaseAnnotationTemplate): ...

Source code in rm_gallery/core/data/annotation/template.py 
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
@classmethod
def register(
    cls, name: str
) -> Callable[[Type[BaseAnnotationTemplate]], Type[BaseAnnotationTemplate]]:
    """
    Decorator for registering annotation templates with unique identifiers.

    Args:
        name: Unique template identifier for registry lookup

    Returns:
        Decorator function that registers the template class and returns it unchanged

    Example:
        @AnnotationTemplateRegistry.register("rewardbench")
        class RewardBenchTemplate(BaseAnnotationTemplate):
            ...
    """

    def decorator(
        template_class: Type[BaseAnnotationTemplate],
    ) -> Type[BaseAnnotationTemplate]:
        # Create an instance of the template with the given name
        template_instance = template_class(name)
        cls._templates[name] = template_instance
        return template_class

    return decorator

BaseAnnotationTemplate

Bases: ABC

Abstract base class for annotation templates defining labeling interface and processing logic.

Templates encapsulate Label Studio XML configuration and annotation post-processing for specific evaluation tasks like reward benchmarking or quality assessment.

Attributes:

Name Type Description
name

Unique identifier for the template
Source code in rm_gallery/core/data/annotation/template.py 
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
class BaseAnnotationTemplate(ABC):
    """
    Abstract base class for annotation templates defining labeling interface and processing logic.

    Templates encapsulate Label Studio XML configuration and annotation post-processing
    for specific evaluation tasks like reward benchmarking or quality assessment.

    Attributes:
        name: Unique identifier for the template
    """

    def __init__(self, name: str):
        """
        Initialize annotation template with unique identifier.

        Args:
            name: Unique template name for registry identification
        """
        self.name = name

    @property
    @abstractmethod
    def label_config(self) -> str:
        """
        Return the Label Studio XML configuration for the annotation interface.

        Defines the labeling interface including rating scales, choice options,
        text areas, and other annotation components specific to the evaluation task.

        Returns:
            Label Studio XML configuration string
        """
        pass

    @abstractmethod
    def process_annotations(self, annotation_data: Dict[str, Any]) -> Dict[str, Any]:
        """
        Process raw annotation data and return structured evaluation results.

        Transforms Label Studio annotation output into domain-specific structured
        data suitable for reward model training or evaluation analysis.

        Args:
            annotation_data: Raw annotation data from Label Studio containing
                           ratings, choices, text areas, and other annotation components

        Returns:
            Processed annotation data structured for specific evaluation needs
            (e.g., reward scores, preference rankings, quality metrics)
        """
        pass

    def validate_annotation_data(self, annotation_data: Dict[str, Any]) -> bool:
        """
        Validate annotation data structure and completeness (optional override).

        Performs sanity checks on annotation data to ensure required fields
        are present and values are within expected ranges.

        Args:
            annotation_data: Annotation data to validate

        Returns:
            True if annotation data is valid and complete, False otherwise
        """
        return True

label_config abstractmethod property

Return the Label Studio XML configuration for the annotation interface.

Defines the labeling interface including rating scales, choice options, text areas, and other annotation components specific to the evaluation task.

Returns:

Type Description
str

Label Studio XML configuration string

__init__(name)

Initialize annotation template with unique identifier.

Parameters:

Name Type Description Default
name str

Unique template name for registry identification

 required
Source code in rm_gallery/core/data/annotation/template.py 
23
24
25
26
27
28
29
30
def __init__(self, name: str):
    """
    Initialize annotation template with unique identifier.

    Args:
        name: Unique template name for registry identification
    """
    self.name = name

process_annotations(annotation_data) abstractmethod

Process raw annotation data and return structured evaluation results.

Transforms Label Studio annotation output into domain-specific structured data suitable for reward model training or evaluation analysis.

Parameters:

Name Type Description Default
annotation_data Dict[str, Any]

Raw annotation data from Label Studio containing ratings, choices, text areas, and other annotation components

 required

Returns:

Type Description
Dict[str, Any]

Processed annotation data structured for specific evaluation needs
Dict[str, Any]

(e.g., reward scores, preference rankings, quality metrics)
Source code in rm_gallery/core/data/annotation/template.py 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
@abstractmethod
def process_annotations(self, annotation_data: Dict[str, Any]) -> Dict[str, Any]:
    """
    Process raw annotation data and return structured evaluation results.

    Transforms Label Studio annotation output into domain-specific structured
    data suitable for reward model training or evaluation analysis.

    Args:
        annotation_data: Raw annotation data from Label Studio containing
                       ratings, choices, text areas, and other annotation components

    Returns:
        Processed annotation data structured for specific evaluation needs
        (e.g., reward scores, preference rankings, quality metrics)
    """
    pass

validate_annotation_data(annotation_data)

Validate annotation data structure and completeness (optional override).

Performs sanity checks on annotation data to ensure required fields are present and values are within expected ranges.

Parameters:

Name Type Description Default
annotation_data Dict[str, Any]

Annotation data to validate

 required

Returns:

Type Description
bool

True if annotation data is valid and complete, False otherwise
Source code in rm_gallery/core/data/annotation/template.py 
64
65
66
67
68
69
70
71
72
73
74
75
76
77
def validate_annotation_data(self, annotation_data: Dict[str, Any]) -> bool:
    """
    Validate annotation data structure and completeness (optional override).

    Performs sanity checks on annotation data to ensure required fields
    are present and values are within expected ranges.

    Args:
        annotation_data: Annotation data to validate

    Returns:
        True if annotation data is valid and complete, False otherwise
    """
    return True