​
Example: Sam’s Car Dealership 🚗

Sam’s Car Dealership offers a range of vehicle services, including maintenance and repairs for customers, while also providing leasing options and selling both new and used cars. Their agent can schedule appointments and collect customer information. In this example, we will build a Hamming test agent taking on the role of the customer to evaluate how well Sam’s AI voice agent schedules appointments for Sam’s customers.

​
Components of the Voice Test Agent

Dataset of Scenarios

Hamming team can create a dataset of scenarios for you to help you cover more scenarios
A dataset of scenarios is a collection of customer phone call scenarios that you want Hamming’s test agent to simulate. You can upload an existing dataset of scenarios in CSV or JSON format or create a new one from the UI. Each dataset of scenarios you create has a dataset_id.

Scoring Prompts

Hamming includes common prompt templates for scoring voice agents like task completion, repetition checker, etc.
To evaluate your AI voice agent you need to create prompts (i.e. LLM-as-a-judge) that define the success criteria for each call. This is critical to ensure your AI voice agent is evaluated only on the things that matter to your business.

Scorer

The scorer wraps a scoring prompt and adds configuration knobs to control how your voice agent is evaluated.

Hamming Test Agent

To test your AI voice agent, the Hamming test agent will simulate multiple interactions using the scenarios provided in your dataset of scenarios. Remember, Hamming’s test agent takes on the role of your customers!

​
Step-by-Step Tutorial

1

Create A Dataset of Scenarios

Pro Tip: Hamming accepts CSV & JSON files up to 10MB.
  1. Visit app.hamming.ai/datasets to create a dataset of scenarios.
  2. Click on +Add New Dataset of Scenarios. You can either upload an existing dataset of scenarios or create an empty one and then use the JSON editor to build it. The dataset of scenarios should contain all phone calls scenarios you’d like Hamming’s test agent to simulate.
  3. Enter a dataset of scenarios name and description.
  4. Select the input columns. The input columns are the data points you’d like the Hamming test agent to reference.
2

Create A Scoring Prompt

Pro Tip: We recommend using gpt-4o with temperature 0 to get the best results.
  1. Visit app.hamming.ai/prompts to create a scoring prompt. The prompt should contain concise instructions for the scorer.
  2. Click on +Add Prompt.
  3. Enter a prompt name and description.
  4. Select the model that you want to use to evaluate your AI voice agent.
  5. Add a system prompt.
  1. Add a user prompt.
Important: The result of the prompt should be structured as an XML format, as outlined below.
  1. Save the prompt and deploy to production.
3

Create A Scorer

Important: Ensure that the values in the scorer align with the labels from your scoring prompt. For example, if the value “0” represents “No” in your prompt, make sure the scorer also labels “0” as “No.”
  1. Visit app.hamming.ai/evals to create a scorer, which simply acts as a wrapper for the scoring prompt.
  2. Click on +Add Scorer.
  3. Enter a scorer name and description.
  4. Add values and labels which will be used to display the results of the test.
  5. Select the scoring prompt you created in Step 2 and set the prompt label to production.
  6. Change the Variable Mappings field to Output and enter transcript in the field on the right.
4

Create A Hamming Test Agent

Pro Tip: If you want to speak to Hamming’s test agent, you can enter your own phone number and Hamming’s test agent will call you.
  1. Visit app.hamming.ai/voice-agents to set up a Hamming test agent (customer).
  2. Click on +Add Voice Agent.
  3. Enter a name.
  4. Select Function from the Scoring dropdown and select the scorer you created in the previous step.
  5. Configure a prompt for the Hamming test agent with input variables from your dataset of scenarios, allowing it to simulate a variety of customer situations.
It is critical that you specify the input variables from your dataset of scenarios created in the Test Agent (Customer) prompt as shown in the template below. This ensures that the Hamming Test agent references the input variables when speaking to your voice agent.
5

Configure post-call webhooks

If your agent doesn’t have any function calling, then you can skip to the next step.
Most voice agents are dynamic and perform actions during the call. To evaluate the performance of your voice agent, you need to configure a post-call webhook so we can capture the output of the call. (audio file, transcript, traces, etc.)
  1. For the Hamming test agent you created in the previous step, click on Edit.
  2. Find the Retell and Vapi webhooks and copy the webhook URL.
Webhook URL
  1. Log-in to your Retell and put our webhook URL in the Agent-level webhook URL field.
Webhook URL
  1. Log-in to your Vapi account and put our webhook URL in the Server URL field.
Webhook URL
  1. This will allow Hamming to capture the complete output of the call (what was said and what was done) and evaluate the performance of your voice agent.
6

Run Hamming Test Agent

  1. Under dataset of scenarios, select the dataset of scenarios you created in Step 1.
  2. Click on Run.
  3. Enter the phone number of your ai voice agent.
  4. Click on Run Calls.
  5. Once your calls have finished, you will be able to see the results in the evaluation column.