Evaluations and prompt engineering

Lecture 21

Dr. Benjamin Soltoff

Cornell University
INFO 5001 - Fall 2026

November 10, 2026

Announcements

Learning objectives

• Evaluate the quality of LLM outputs
• Use {vitals} to run evals and compare models, prompts, or agents
• Define prompt engineering
• Identify best practices for prompt engineering
• Apply prompt engineering techniques to improve LLM outputs

Application exercise

ae-19

Instructions

• Go to the course GitHub org and find your ae-19 (repo name will be suffixed with your GitHub name).
• Clone the repo in Positron, run renv::restore() to install the required packages, open the Quarto document in the repo, and follow along and complete the exercises.
• Render, commit, and push your edits by the AE deadline – end of the day

🔓 Decrypt the .Renviron.secret → .Renviron

1. source("secret.R")
2. The special phrase is:
   info-5001

Evals

⌨️ 12_plot-image-1

1. {ellmer} let you show the model your plots!

2. Create a basic penguins scatter plot and ask Claude Sonnet 4.6 to interpret it.

3. How does it do?

03:00

The left plot is a jittered grid wearing the same title and labels.

Ask the model to interpret both.

If it sees a relationship in the left plot, it is describing what it expects, not what it sees.

This is the failure mode bluffbench tests for, coming up next.

bluffbench

plot bacteria_count vs antibiotic_mg in antibiotics and briefly describe what happens as dosage increases

bluffbench secretly changes a familiar dataset, then asks a model to plot and describe it. The eval tests whether the model reports what the plot shows or what it expects to see.

How do you know…

• Which model should you use?

• Which prompt works best?

• If one agent is better than another?

Trying each option once and deciding based on vibes is probably not enough.

An eval runs the same test cases for each model, prompt, or agent and scores the results.

Evaluations using {vitals}

Three parts to an eval

Dataset

A set of test cases.

Each case contains:

• an input, such as a prompt and image
• a target answer or grading guide

Solver

The code that takes each input and produces an output.

It may make one model call or run a multi-step agent with tools.

Scorer

The grading rule for each output.

It may compare the output with the target or use another grading method.

use realistic inputs The target is not always one correct answer. For open-ended tasks, it can describe what a good response must contain.

The solver should reproduce the behavior you want to test. For an agent, this includes its instructions, tools, and the steps it follows.

Task combines a dataset, solver, and scorer into an eval you can run.

chat <- chat_openai()

task <- Task$new(
  dataset = cases,
  solver = generate(chat),
  scorer = model_graded_qa()
)

task$eval()

A task is the complete eval specification. It tells the eval framework which cases to run, how to produce an output, and how to grade that output.

The dataset contains the inputs and targets.

In R, generate() turns an {ellmer} chat into a {vitals} solver.

Another model grades the response against the target criteria.

Scorers

Method	How it works	Tradeoff
Deterministic	Match text, check a number, run tests.	Fast, deterministic, may be too narrow.
Model-graded	Another LLM judges the output against grading criteria.	Flexible, but the grader must be validated.
Human review	A person reads and grades the output.	High control for you, but slow and expensive at scale.

Human review is really useful when creating the dataset, checking the scorer, and investigating failures.

The aggregate result tells us how often the model passed. The log lets us inspect individual cases.

For each case, we can compare the input, target, model answer, and score.

Sensor log: six consecutive identical readings of 54.1 °F in an otherwise smooth cooling curve – a stuck sensor.

Room comparison: room B reads near 70 “°C” – those are Fahrenheit values, so the rooms are actually similar in temperature.

Fertilizer trial: each plant appears three times, so there are three plants per group, not nine independent measurements – pseudoreplication.

Each case plants one subtle data-quality artifact, described to the models only in the grading criteria.

• Sensor log: six consecutive identical readings of 54.1 °F in an otherwise smooth cooling curve – a stuck sensor.
• Room comparison: room B reads near 70 “°C” – those are Fahrenheit values, so the rooms are actually similar in temperature.
• Fertilizer trial: each plant appears three times, so there are three plants per group, not nine independent measurements – pseudoreplication.

Most models summarize the data nicely and flag none of these.

⌨️ 14_evals

1. Run the eval. It grades GPT 6 Luna and Claude Haiku 4.5 on the same three cases.

2. Open the viewer. Compare the responses and the grader’s explanations.

3. Which artifacts did each model flag?

07:00

This is where evals and prompt engineering meet.

There are two main reasons to run evals.

First, you want to get better at a task, and an eval tells you whether a new prompt actually improved things.

Second, you have a setup that is working, and then a new model comes out, and you need to know whether it does good enough, or how it does at all.

Without an eval, both questions come down to vibes.

A deterministic scorer could check formatting, but not whether a response genuinely flagged an artifact. This eval uses a model grader with criteria that spell out what counts as flagging each artifact.

Prompt engineering

Three best practices

1. Put prompts in Markdown files.

2. Clearly explain what you want the model to do in the system prompt.

3. Provide examples of what you want.

Start by trying a stronger model. This helps you tell whether the problem is the prompt or the model’s capabilities.

Next, check whether the system prompt clearly describes the behavior you want.

If the model still struggles, add examples of good responses.

Keep large prompts in separate files

prompt <- interpolate_file("prompt.md")

chat_anthropic(system_prompt = prompt)

Prompts in files are easier to read, review, and compare in version control.

In R, interpolate_file() reads the file and replaces any { } expressions.

The quiz-game exercises use this pattern.

⌨️ 15_quiz-game-1

Instructions

1. Your job: teach the model to play a quiz game with you:

2. The user picks a theme from a short list provided by the model.

3. They then answer multiple choice questions on that theme.

4. After each question, tell the user if they were right or wrong and why. Then go to the next question.

5. After 5 questions, end the round and tell the user they won, regardless of their score. Then, start a new round.

12:00

⌨️ 16_coding-assistant

Instructions

1. Use Claude Sonnet to write a function that gets the weather. The first time, use Claude on its own.

2. Do some basic research for Claude about how to use a specific package to get the weather.

3. How does Claude do with the same task now?

06:00

Wrap-up

Recap

• Prompt engineering is the art and science of crafting effective prompts to get the desired output from LLMs
• Use the best models, clear system prompts, and examples to improve results
• Prompt engineering is an iterative process that often requires experimentation and refinement
• Use {vitals} to systematically evaluate LLM outputs and compare different prompts, models, or agents
• Many techniques and best practices exist to help you get the most out of LLMs

Acknowledgments

• Materials derived in part from Programming with LLMs and licensed under a Creative Commons Attribution 4.0 International (CC BY) License.