Evaluations

Before you begin

Follow the Setting up guide to make sure that you have access to the Hamming dashboard and you have created a secret key.

Quickstart - Node.js

Learn how run an evaluation experiment with our Hamming TypeScript SDK.

Install Hamming SDK TypeScript library

Create a dataset of scenarios

Create a simple evaluation script

Make sure to replace the placeholders with your actual keys and dataset ID created in the previous step.

Create a file named evals.ts or evals.js and add the following code:

import { Hamming, ScoreType } from "@hamming/hamming-sdk";
import { OpenAI } from "openai";

const HAMMING_API_KEY = "<your-secret-key>";
const HAMMING_DATASET_ID = "<your-dataset-id>";
const OPENAI_API_KEY = "<your-openai-key>";

const hamming = new Hamming({
  apiKey: HAMMING_API_KEY,
});

const openai = new OpenAI({
  apiKey: OPENAI_API_KEY,
});

async function run() {
  await hamming.experiments.run(
    {
      name: "Example experiment from TS SDK",
      dataset: HAMMING_DATASET_ID,
      scoring: [ScoreType.AccuracyAI],
      metadata: {},
    },
    async (input) => {
      const { question } = input;
      const response = await openai.chat.completions.create({
        model: "gpt-4-turbo",
        messages: [
          {
            role: "system",
            content: "Respond with a brief sentence.",
          },
          {
            role: "user",
            content: question,
          },
        ],
      });
      const answer = response.choices[0].message.content;
      return { answer };
    }
  );
}

run().catch(console.error);

Run the first evaluation experiment

Install dependencies:

npm install openai

Run the script by executing the following command in your terminal:

npx tsx evals.ts

This will create an experiment in Hamming. Once the command runs, you’ll see a link to your experiment.

Quickstart - Python

Learn how run an evaluation experiment with our Hamming Python SDK.

Install Hamming SDK Python library

Create a dataset of scenarios

Create a simple evaluation script

Make sure to replace the placeholders with your actual keys and dataset ID created in the previous step.

Create a file named evals.py and add the following code:

evals.py

from hamming import ClientOptions, Hamming, RunOptions, ScoreType
from openai import OpenAI

HAMMING_API_KEY = "<your-secret-key>"
HAMMING_DATASET_ID = "<your-dataset-id>"
OPENAI_API_KEY = "<your-openai-key>"

hamming = Hamming(ClientOptions(api_key=HAMMING_API_KEY))
openai_client = OpenAI(api_key=OPENAI_API_KEY)


def answer_question(input):
    question = input["question"]
    response = openai_client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "Respond with a brief sentence."},
            {"role": "user", "content": question},
        ],
    )
    answer = response.choices[0].message.content
    return {"answer": answer}


def run():
    hamming.experiments.run(
        RunOptions(
            dataset=HAMMING_DATASET_ID,
            name="Example experiment from Python SDK",
            scoring=[
                ScoreType.ACCURACY_AI,
            ],
            metadata={},
        ),
        answer_question,
    )


if __name__ == "__main__":
    run()

Run the first evaluation experiment

Install dependencies:

pip install openai

Run the script by executing the following command in your terminal:

python evals.py

This will create an experiment in Hamming. Navigate to the Experiments page to see the results.

Get Started

Voice Agent Testing

Call Monitoring

Other Guides

Before you begin

Quickstart - Node.js

Quickstart - Python

Get Started

Voice Agent Testing

Call Monitoring

Other Guides

​Before you begin

​Quickstart - Node.js

​Quickstart - Python

Before you begin

Quickstart - Node.js

Quickstart - Python