Custom Scores

Hamming AI comes with pre-built scorers that cover a wide range of use cases. If you need to score your experiments in a different way, you can do so by creating a custom scorer.

Definitions

A custom scorer is a function that computes a score of a given experiment result.

Looking at the function output, we can define a custom scorer as:

Classification Scorer: A function that returns a categorical value. (e.g. low, medium, high)
Numerical Scorer: A function that returns a real number.

Based on the execution environment, we can define a custom scorer as:

Local Scorer: A function that runs on the machine that executes the experiment.
Remote Scorer: A function that runs on Hamming AI platform. [Coming soon]

Before you begin

Follow the Evaluations Guide to get familiar with running experiments on Hamming AI. You should have a dataset ID and a secret key to continue with this guide.

Creating a Custom Scorer - Python

Create a file named scorer.py and add the following code:

scorer.py

import random

from hamming import (
    ScoringFunction,
    ClassificationScoreConfig,
    NumericScoreConfig,
    FunctionAggregateType,
    FunctionType,
    LabelColor,
    LocalScorer,
    ScoreArgs,
    Score
)


def custom_correctness_score(args: ScoreArgs) -> Score:
    output = args["output"]
    expected = args["expected"]
    
    # "output" and "expected" are Dict objects
    # matching the format of a dataset item Output Json.
    output_answer = output["answer"]
    expected_answer = expected["answer"]
    
    # Define your scoring logic here.
    if (len(output_answer) % 2) == (len(expected_answer) % 2):
        return Score(value=1, reason="The length of the answers has the same parity.")
    else:
        return Score(value=0, reason="The length of the answers has different parity.")


# 1. Classification Scorer
custom_scoring_classification = ScoringFunction(
    name="TutorialScore-Classify",
    version=1,
    score_config=ClassificationScoreConfig(
        type=FunctionType.CLASSIFICATION,
        labels={
            0: "Incorrect",
            1: "Correct",
        },
        colors={
            0: LabelColor.RED,
            1: LabelColor.GREEN,
        },
    ),
    scorer=LocalScorer(
        score_fn=custom_correctness_score
    ),
)

# 2. Numerical Scorer
custom_scoring_numeric = ScoringFunction(
    name="TutorialScore-Numeric",
    version=1,
    score_config=NumericScoreConfig(
        aggregate=FunctionAggregateType.MEAN,
    ),
    scorer=LocalScorer(
        score_fn=lambda _: Score(value=random.random(), reason="Random number")
    ),
)

Make sure to replace the placeholders with your actual keys and dataset ID created in the evaluations guide.

Create a file named evals.py and add the following code:

evals.py

from hamming import (
    Hamming,
    ClientOptions,
    RunOptions,
)
from scorer import (
    custom_scoring_classification, 
    custom_scoring_numeric
)

HAMMING_API_KEY = "<your-secret-key>"
HAMMING_DATASET_ID = "<your-dataset-id>"

hamming = Hamming(ClientOptions(api_key=HAMMING_API_KEY))

def answer_question(input):
    question = input["question"]
    #TODO; Implement your own RAG pipeline here.
    # ...
    answer = "This is a test answer"
    return {"answer": answer}

result = hamming.experiments.run(
    RunOptions(
        dataset=HAMMING_DATASET_ID,
        name="Custom Scorer Experiment - Python",
        scoring=[
            # Use your custom scorers here
            custom_scoring_classification,
            custom_scoring_numeric,
        ],
        metadata={},
    ),
    answer_question,
)

Get Started

Guides

Advanced

Definitions

Before you begin

Creating a Custom Scorer - Python

Get Started

Guides

Advanced

​Definitions

​Before you begin

​Creating a Custom Scorer - Python

Definitions

Before you begin

Creating a Custom Scorer - Python