Prompt Optimizer

Getting started

You can access our hosted Web Application, or request access to the Prompt Optimizer API.

How it works

Prompt Optimizer uses LLMs to generate a well-structured prompt, given a task description and a few examples. It generates a set of candidate prompts and ranks them based on their performance. The prompt with the highest score is then returned as the best prompt.

Describe task

Write a short description of the task you want to solve, in natural language.

Define task inputs and outputs

Input and output definitions can be automatically extracted from the task description.

Define the input and output format for the task. This includes the data type, structure, and any constraints.

Provide examples

We can generate synthetic examples based on the task description and input/output format.

Providing 2-3 hand-labeled examples will help us generate more realistic synthetic data.

Provide 10-20 examples that include inputs and corresponding outputs for the task. The examples will be used as a training set for the LLMs.

Generate candidate prompts [Automated]

We use LLMs to generate a set of candidate prompts. This takes care of selecting the best candidates for few-shot examples.

🎯 Rank and select the best prompt [Automated]

Rank the candidate prompts based on their performance. Behind the scenes, we run a thorough evaluations process. The prompt with the highest score is then returned as the best prompt.

Programmatic access

We’re slowly rolling out programmatic access to the Prompt Optimizer. You can use the following end-to-end example to get started.

Make sure to replace the placeholders with your actual keys and dataset ID created in the previous step.

Create a file named main.py and add the following code:

main.py
import requests
import logging

hamming_url = "https://app.hamming.ai"
hamming_api_key = "<hamming-api-key>" # Replace with your Hamming API key

headers = {
    "Authorization": f"Bearer {hamming_api_key}",
    "Content-Type": "application/json"
}
def start_job():
    url = f"{hamming_url}/api/rest/prompt-optimizer/start-job"
    payload = {}
    response = requests.post(url, headers=headers, json=payload)
    response.raise_for_status()
    return response.json()
def extract_fields(job_id, task):
    url = f"{hamming_url}/api/rest/prompt-optimizer/extract-fields"
    payload = {"jobId": job_id, "task": task}
    response = requests.post(url, headers=headers, json=payload)
    response.raise_for_status()
    return response.json()
def generate_examples(job_id, task, fields, examples):
    url = f"{hamming_url}/api/rest/prompt-optimizer/generate-examples"
    payload = {
        "jobId": job_id,
        "task": task,
        "fields": fields,
        "examples": examples
    }
    response = requests.post(url, headers=headers, json=payload)
    response.raise_for_status()
    return response.json()
def optimize(job_id, task, fields, examples):
    url = f"{hamming_url}/api/rest/prompt-optimizer/optimize"
    payload = {
        "jobId": job_id,
        "task": task,
        "fields": fields,
        "examples": examples
    }
    response = requests.post(url, headers=headers, json=payload)
    response.raise_for_status()
def get_job_status(job_id):
    url = f"{hamming_url}/api/rest/prompt-optimizer/job-status"
    payload = {
        "jobId": job_id,
    }
    response = requests.get(url, headers=headers, params=payload)
    response.raise_for_status()
    return response.json()
if __name__ == "__main__":
    job_response = start_job()
    job_id = job_response.get("jobId")

    task_description = "I will give you a description of the data I'm trying to visualize. Return back to me the type of chart that I should use, so my data is easy to understand. Choose from pie chart, bar chart, waterfall chart, etc."
    fields_response = extract_fields(job_id, task_description)
    fields = fields_response.get("fields")

    sample_examples = []
    examples_response = generate_examples(job_id, task_description, fields, sample_examples)
    examples = examples_response.get("examples")

    print("Starting job optimization, this will take 30 seconds...")
    optimized_response = optimize(job_id, task_description, fields, examples)

    # poll ever 10 seconds for job-status
    import time
    done = False
    job_status_response = None
    while not done:
        job_status_response = get_job_status(job_id)
        status = job_status_response.get("status")
        print("Watching job status: ", status)
        if status == "complete":
            done = True
        else:
            time.sleep(10)
    print("Optimization Complete:", job_status_response.get("prompt"))

Make sure to replace the placeholders with your actual keys and dataset ID created in the previous step.

Create a file named main.ts and add the following code:

main.ts
const hammingUrl = "https://app.hamming.ai";
const hammingApiKey = "<hamming-api-key>"; // Replace with your Hamming API key

type Field = {
    name: string;
    desc: string;
    isInput: boolean;
};

type Example = {
    input: {
        [key: string]: string;
    };
    output: {
        [key: string]: string;
    };
};

type StartJobResponse = {
    jobId: string;
};

type JobResponse = {
    jobId: string;
    status: string;
    prompt: string;
};

const headers = {
    Authorization: `Bearer ${hammingApiKey}`,
    'Content-Type': 'application/json',
};

async function startJob(): Promise<StartJobResponse> {
    const url = `${hammingUrl}/api/rest/prompt-optimizer/start-job`;
    const response = await fetch(url, {
        method: 'POST',
        headers: headers,
        body: JSON.stringify({}),
    });
    if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
    }
    return response.json() as Promise<StartJobResponse>;
}

async function extractFields(
    jobId: string,
    task: string
): Promise<{ fields: Field[] }> {
    const url = `${hammingUrl}/api/rest/prompt-optimizer/extract-fields`;
    const response = await fetch(url, {
        method: 'POST',
        headers: headers,
        body: JSON.stringify({ jobId, task }),
    });
    if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
    }
    return response.json() as Promise<{ fields: Field[] }>;
}

async function generateExamples(
    jobId: string,
    task: string,
    fields: Field[],
    examples: Example[]
): Promise<{ examples: Example[] }> {
    const url = `${hammingUrl}/api/rest/prompt-optimizer/generate-examples`;
    const response = await fetch(url, {
        method: 'POST',
        headers: headers,
        body: JSON.stringify({ jobId, task, fields, examples }),
    });
    if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
    }
    return response.json() as Promise<{ examples: Example[] }>;
}

async function optimize(
    jobId: string,
    task: string,
    fields: Field[],
    examples: Example[]
) {
    const url = `${hammingUrl}/api/rest/prompt-optimizer/optimize`;
    const response = await fetch(url, {
        method: 'POST',
        headers: headers,
        body: JSON.stringify({ jobId, task, fields, examples }),
    });
    if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
    }
}

async function getJobStatus(jobId: string): Promise<JobResponse> {
    const url = `${hammingUrl}/api/rest/prompt-optimizer/job-status?jobId=${jobId}`;
    const response = await fetch(url, {
        method: 'GET',
        headers: headers,
    });
    if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
    }
    return response.json() as Promise<JobResponse>;
}

async function main() {
    try {
        const jobResponse = await startJob();
        const jobId = jobResponse.jobId;

        const taskDescription =
            "I will give you a description of the data I'm trying to visualize. Return back to me the type of chart that I should use, so my data is easy to understand. Choose from pie chart, bar chart, waterfall chart, etc.";
        const fieldsResponse = await extractFields(jobId, taskDescription);
        const fields = fieldsResponse.fields;

        const sampleExamples: Example[] = [];
        const examplesResponse = await generateExamples(
            jobId,
            taskDescription,
            fields,
            sampleExamples
        );
        const examples = examplesResponse.examples;

        console.log('Starting job optimization, this will take 30 seconds...');
        await optimize(jobId, taskDescription, fields, examples);

        // poll every 10 seconds for job-status
        let done = false;
        let jobStatusResponse: JobResponse | undefined;
        while (!done) {
            jobStatusResponse = await getJobStatus(jobId);
            const status = jobStatusResponse.status;
            console.log('Watching job status: ', status);
            if (status === 'complete') {
                done = true;
            } else {
                await new Promise((resolve) => setTimeout(resolve, 10000));
            }
        }
        if (!jobStatusResponse) {
            throw new Error('Job status response is undefined');
        }
        console.log('Optimization Complete:', jobStatusResponse.prompt);
    } catch (error) {
        console.error('Error:', error);
    }
}

main();

API Reference

For early access to the API, please contact us.

Coming soon

Get Started

Voice Guides

Guides

Advanced

Prompt Optimizer

Getting started

How it works

Programmatic access

API Reference

Get Started

Voice Guides

Guides

Advanced

​Getting started

​How it works

​Programmatic access

​API Reference

Getting started

How it works

Programmatic access

API Reference