Voice Agent Testing Tutorial

1. Visit app.hamming.ai/datasets to create a dataset.
2. Click on +Add New Dataset. You can either upload an existing dataset or create an empty dataset and then use the JSON editor to build one. The dataset should contain all phone calls scenarios you’d like Hamming’s test agent to simulate.
3. Enter a dataset name and description.
4. Select the input columns. The input columns are the data points you’d like the Hamming test agent to reference.

1. Visit app.hamming.ai/prompts to create a scoring prompt. The prompt should contain concise instructions for the scorer.
2. Click on +Add Prompt.
3. Enter a prompt name and description.
4. Select the model that you want to use to evaluate your AI voice agent.
5. Add a system prompt.

You are an expert data analyst, with 30 years of experience analyzing
phone call transcripts. You have a great attention to details. 

6. Add a user prompt.

[TASK]
  You are given the <transcript> of a phone call between a receptionist (HUMAN) and a customer (BOT).
  Your task is to classify the transcript of the phone call, given the [CRITERIA].

[CRITERIA]
Was the receptionist (HUMAN) able to schedule an appointment for the customer? 1 - Yes; 0 - No;
If there is not enough information in the transcript to determine the correct classification, label as 2 - N/A;

Assign one of the labels above and provide reasoning for why the call should be assigned that label.

[INPUT]

{" "}

<transcript>{{ transcript }}</transcript>

[OUTPUT]

{" "}

<score>
  <reason>
    Find evidence in the transcript to prove whether the interviewer succeeded
    or failed to meet the criteria. Provide a detailed, step-by-step explanation
    for assigning a label, based on the available evidence or the absence of it.
  </reason>
  <value>0 | 1 | 2</value>
</score>

7. Save the prompt and deploy to production.

1. Visit app.hamming.ai/evals to create a scorer, which simply acts as a wrapper for the scoring prompt.
2. Click on +Add Scorer.
3. Enter a scorer name and description.
4. Add values and labels which will be used to display the results of the test.
5. Select the scoring prompt you created in Step 2 and set the prompt label to production.
6. Change the Variable Mappings field to Output and enter transcript in the field on the right.

1. Visit app.hamming.ai/voice-agents to set up a Hamming test agent (customer).
2. Click on +Add Voice Agent.
3. Enter a name.
4. Select Function from the Scoring dropdown and select the scorer you created in the previous step.
5. Configure a prompt for the Hamming test agent with input variables from your dataset, allowing it to simulate a variety of customer situations.

You are calling a car dealership and talking to the front-desk assistant. Follow the scenario below:

Your name: {name}
Department you want to talk to: {department_name}
Buying / Leasing interest: {buy_or_lease}
Your Customer Status: {existing_customer}
Your budget for buying car: {budget}
Your phone number: {phone_number}
When to schedule appointment: {appointment_schedule}
Service Requested: {type_of_service}

[RULES]
Only answer questions when you are asked. Don't disclose any information without being asked first.
Say goodbye at the end.

Most voice agents are dynamic and perform actions during the call. To evaluate the performance of your voice agent, you need to configure a post-call webhook so we can capture the output of the call. (audio file, transcript, traces, etc.)

1. For the Hamming test agent you created in the previous step, click on Edit.
2. Find the Retell and Vapi webhooks and copy the webhook URL.

3. Log-in to your Retell or Vapi account and put our webhook URL in the Post-call webhook field.
4. This will allow Hamming to capture the complete output of the call (what was said and what was done) and evaluate the performance of your voice agent.

1. Under dataset, select the dataset you created in Step 1.
2. Click on Run.
3. Enter the phone number of your ai voice agent.
4. Click on Run Calls.
5. Once your calls have finished, you will be able to see the results in the evaluation column.