Guides
Evaluations
Guides
Evaluations
Test changes of your AI application before deploying them.
Before you begin
Follow the Setting up guide to make sure that you have access to the Hamming dashboard and you have created a secret key.
Quickstart - Node.js
Learn how run an evaluation experiment with our Hamming TypeScript SDK.
npm install @hamming/hamming-sdk
- Download our Sample dataset file.
- Navigate to Create new dataset and use the drag and drop box to upload the file.
- For Input Columns, select “question”, and for Output Columns, select “answer”.
- Name it “Simple Dataset” and click Create.
- Copy the dataset ID from by clicking on the Copy ID button.
Make sure to replace the placeholders with your actual keys and dataset ID created in the previous step.
evals.js
const { Hamming, ScoreType } = require("@hamming/hamming-sdk");
const { OpenAI } = require("openai");
const HAMMING_API_KEY = "<your-secret-key>";
const HAMMING_DATASET_ID = "<your-dataset-id>";
const OPENAI_API_KEY = "<your-openai-key>";
const hamming = new Hamming({
apiKey: HAMMING_API_KEY,
});
const openai = new OpenAI({
apiKey: OPENAI_API_KEY,
});
async function run() {
await hamming.experiments.run(
{
name: "Example experiment from TS SDK",
dataset: HAMMING_DATASET_ID,
scoring: [ScoreType.AccuracyAI],
metadata: {},
},
async (input) => {
const { question } = input;
const response = await openai.chat.completions.create({
model: "gpt-4-turbo",
messages: [
{
role: "system",
content: "Respond with a brief sentence.",
},
{
role: "user",
content: question,
},
],
});
const answer = response.choices[0].message.content;
return { answer };
}
);
}
run().catch(console.error);
Install dependencies:
npm install openai
Run the script by executing the following command in your terminal:
node evals.js
This will create an experiment in Hamming. Once the command runs, you’ll see a link to your experiment.
Quickstart - Python
Learn how run an evaluation experiment with our Hamming Python SDK.
pip install hamming-sdk
- Download our Sample dataset file.
- Navigate to Create new dataset and use the drag and drop box to upload the file.
- For Input Columns, select “question”, and for Output Columns, select “answer”.
- Name it “Simple Dataset” and click Create.
- Copy the dataset ID from by clicking on the Copy ID button.
Make sure to replace the placeholders with your actual keys and dataset ID created in the previous step.
evals.py
from hamming import ClientOptions, Hamming, RunOptions, ScoreType
from openai import OpenAI
HAMMING_API_KEY = "<your-secret-key>"
HAMMING_DATASET_ID = "<your-dataset-id>"
OPENAI_API_KEY = "<your-openai-key>"
hamming = Hamming(ClientOptions(api_key=HAMMING_API_KEY))
openai_client = OpenAI(api_key=OPENAI_API_KEY)
def answer_question(input):
question = input["question"]
response = openai_client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": "Respond with a brief sentence."},
{"role": "user", "content": question},
],
)
answer = response.choices[0].message.content
return {"answer": answer}
def run():
hamming.experiments.run(
RunOptions(
dataset=HAMMING_DATASET_ID,
name="Example experiment from Python SDK",
scoring=[
ScoreType.ACCURACY_AI,
],
metadata={},
),
answer_question,
)
if __name__ == "__main__":
run()
Install dependencies:
pip install openai
Run the script by executing the following command in your terminal:
python evals.py
This will create an experiment in Hamming. Navigate to the Experiments page to see the results.