Programming with LLMs

Lecture 22

Dr. Benjamin Soltoff

Cornell University
INFO 5001 - Fall 2025

November 13, 2025

Announcements

Announcements

TODO

Learning objectives

• Understand the difference between LLM providers and models
• Choose appropriate models for different tasks
• Handle multi-modal inputs (images, PDFs)
• Extract structured data from LLM responses
• Perform parallel and batch LLM calls for efficiency

Application exercise

ae-20

Instructions

• Go to the course GitHub org and find your ae-20 (repo name will be suffixed with your GitHub name).
• Clone the repo in Positron, run renv::restore() to install the required packages, open the Quarto document in the repo, and follow along and complete the exercises.
• Render, commit, and push your edits by the AE deadline – end of the day

Providers and Models

Provider

company that hosts and serves models

Model

a specific LLM with particular capabilities

How are models different?

1. Content: How many tokens can you give the model?
2. Speed: How many tokens per second?
3. Cost: How much does it cost to use the model?
4. Intelligence: How “smart” is the model?
5. Capabilities: Vision, reasoning, tools, etc.

1. Content: 200k tokens is normal, 1M tokens in some newer models
2. Speed: Gemini Flash hits ~350 tokens/sec, 100-200 is very fast, 50-100 normal. Speed is often a trade-off with intelligence.
3. Cost: $1-5 per million tokens is normal for big frontier models, < $1 for smaller, faster models
4. Intelligence is a stand-in for lots of concepts

We’ll start with Anthropic, a company that builds LLMs, in particular a family of models they call Claude.

Claude comes in different sizes, with different levels of intelligence.

They have a small model that’s fast and cheap, a large model that’s intelligent but expensive, and a just-right sized model that balances speed, cost, and intelligence.

In line with Anthropic’s branding, these models are named after different types of poetry.

• Haiku is small, fast, and cheap
• Sonnet is larger, more expensive, and more intelligent
• Opus is the largest, most expensive, and most intelligent

Anthropic’s poetry naming offsets the fact that Anthropic has one of the most consistent naming schemes in the industry. (Yes, that’s says a lot.)

But there isn’t just one Claude model at different sizes. Anthropic keeps training new models which they generally release with a new version number.

• Claude 3.5 was released a year ago (in June 2024)
• Claude 3.7 was released earlier this year (Feb. 2025)
• Claude 4 was just released in May 2025

All four of these models are still active and available.

When you’re choosing a model, you can get higher quality responses by moving up a size tier.

And when new releases come out, you can often get similar quality improvements by moving up a version number. (This move tends to be cheaper than moving up a size tier.)

Oh, wait, sorry, to be very technically accurate, it turns out that the Opus models are only available for Claude 3 and Claude 4, but not for the intermediate versions 3.5 and 3.7.

It remains to be seen if this will be an enduring pattern, or just they way things happened for 3.5 and 3.7.

Choose a model

Task	OpenAI	Anthropic	Gemini
Coding	GPT-5	Claude 4 Sonnet	Gemini 2.5 Pro
Fast/General	GPT-5 mini	Claude 3.5 Sonnet	Gemini 2.0 Flash
Complex Tasks	o3	Claude 4 Opus	Gemini 2.5 Pro
Cost-Effective	Mini	Haiku	Flash

Learn more

• Ranking of models: Artificial Analysis
• OpenAI models
• Anthropic models
• Google Gemini models

Providers

• chat_openai()

• chat_anthropic()

• chat_google_gemini()

Local models

• chat_ollama()

Enterprise

• chat_aws_bedrock()

Chat in Easy Mode

library(ellmer)

chat <- chat("anthropic")

Chat in Easy Mode

library(ellmer)

chat <- chat("anthropic")
#> Using model = "claude-sonnet-4-20250514".

Chat in Easy Mode

library(ellmer)

chat <- chat("openai")
#> Using model = "gpt-4.1".

Chat in Easy Mode

library(ellmer)

chat <- chat("openai/gpt-4.1-nano")

⌨️ 07_models

Instructions

1. Use {ellmer} to list available models from Anthropic and OpenAI.

2. Send the same prompt to different models and compare the responses.

3. Feel free to change the prompt!

04:00

Favorite models for exercises

• OpenAI
  • gpt-4.1-nano
  • gpt-5
• Anthropic
  • claude-sonnet-4-20250514
  • claude-3-5-haiku-20241022

Multi-modal input

A picture is worth a thousand words

Or for an LLM, a picture is roughly 227 words, or 170 tokens.

🖼️ 🔍 Open Images Dataset

🌆 content_image_file

library(ellmer)

chat <- chat("openai/gpt-4.1-nano")
chat$chat(
  content_image_file("cute-cats.jpg"),
  "What do you see in this image?"
)

🐈 content_image_url

library(ellmer)

chat <- chat("openai/gpt-4.1-nano")
chat$chat(
  content_image_url("https://placecats.com/bella/400/400"),
  "What do you see in this image?"
)

⌨️ 08_vision

Instructions

1. I’ve put some images of food in the data/recipes/images folder.

2. Your job: show the food to the LLM and see if it gets hungry.

05:00

📑 content_pdf_file

library(ellmer)

chat <- chat("openai/gpt-4.1-nano")
chat$chat(
  content_pdf_file("financial-report.pdf"),
  "What's my tax liability for 2024?"
)

📑 content_pdf_url

library(ellmer)

chat <- chat("openai/gpt-4.1-nano")
chat$chat(
  content_pdf_url("http://pdf.secdatabase.com/1757/0001104659-25-042659.pdf"),
  "Describe Tesla’s executive compensation and stock award programs."
)

⌨️ 09_pdf

Instructions

1. We have the actual recipes as PDFs in the data/recipes/pdf folder.

2. Your job: ask the LLM to convert the recipes to markdown.

05:00

Structured output

How would you extract name and age?

age_free_text <- list(
    "I go by Alex. 42 years on this planet and counting.",
    "Pleased to meet you! I'm Jamal, age 27.",
    "They call me Li Wei. Nineteen years young.",
    "Fatima here. Just celebrated my 35th birthday last week.",
    "The name's Robert - 51 years old and proud of it.",
    "Kwame here - just hit the big 5-0 this year."
)

If you wrote R code, it might look like this…

word_to_num <- function(x) {
    # normalize
    x <- tolower(x)
    # direct numbers
    if (grepl("\\b\\d+\\b", x)) return(as.integer(regmatches(x, regexpr("\\b\\d+\\b", x))))
    # hyphenated like "5-0"
    if (grepl("\\b\\d+\\s*-\\s*\\d+\\b", x)) {
        parts <- as.integer(unlist(strsplit(regmatches(x, regexpr("\\b\\d+\\s*-\\s*\\d+\\b", x)), "\\s*-\\s*")))
        return(10 * parts[1] + parts[2])
    }
    # simple word numbers
    ones <- c(
        zero=0, one=1, two=2, three=3, four=4, five=5, six=6, seven=7, eight=8, nine=9,
        ten=10, eleven=11, twelve=12, thirteen=13, fourteen=14, fifteen=15, sixteen=16,
        seventeen=17, eighteen=18, nineteen=19
    )
    tens <- c(twenty=20, thirty=30, forty=40, fifty=50, sixty=60, seventy=70, eighty=80, ninety=90)
    # e.g., "nineteen"
    if (x %in% names(ones)) return(ones[[x]])
    # e.g., "thirty five" or "thirty-five"
    x2 <- gsub("-", " ", x)
    parts <- strsplit(x2, "\\s+")[[1]]
    if (length(parts) == 2 && parts[1] %in% names(tens) && parts[2] %in% names(ones)) {
        return(tens[[parts[1]]] + ones[[parts[2]]])
    }
    if (length(parts) == 1 && parts[1] %in% names(tens)) return(tens[[parts[1]]])
    return(NA_integer_)
}

# Extract name candidates
extract_name <- function(s) {
    # patterns that introduce a name
    pats <- c(
        "I go by\\s+([A-Z][a-z]+)",
        "I'm\\s+([A-Z][a-z]+(?:\\s+[A-Z][a-z]+)?)",
        "They call me\\s+([A-Z][a-z]+(?:\\s+[A-Z][a-z]+)?)",
        "^([A-Z][a-z]+) here",
        "The name's\\s+([A-Z][a-z]+)",
        "^([A-Z][a-z]+)\\s" # fallback: leading capital word
    )
    for (p in pats) {
        m <- regexpr(p, s, perl = TRUE)
        if (m[1] != -1) {
            return(sub(p, "\\1", regmatches(s, m)))
        }
    }
    NA_character_
}

# Extract age phrases and convert to number
extract_age <- function(s) {
    # capture common age phrases around a number
    m <- regexpr("(\\b\\d+\\b|\\b\\d+\\s*-\\s*\\d+\\b|\\b[Nn][a-z-]+\\b)\\s*(years|year|birthday|young|this)", s, perl = TRUE)
    if (m[1] != -1) {
        token <- sub("(years|year|birthday|young|this)$", "", trimws(substring(s, m, m + attr(m, "match.length") - 1)))
        return(word_to_num(token))
    }
    # handle pure word-number without trailing keyword (e.g., "Nineteen years young." handled above)
    m2 <- regexpr("\\b([A-Z][a-z]+)\\b\\s+years", s, perl = TRUE)
    if (m2[1] != -1) {
        token <- tolower(sub("\\s+years.*", "", regmatches(s, m2)))
        return(word_to_num(token))
    }
    # handle hyphenated "big 5-0"
    m3 <- regexpr("big\\s+(\\d+\\s*-\\s*\\d+)", s, perl = TRUE)
    if (m3[1] != -1) {
        token <- sub("big\\s+", "", regmatches(s, m3))
        return(word_to_num(token))
    }
    NA_integer_
}

If you wrote R code, it might look like this…

dplyr::tibble(
  name = purrr::map_chr(age_free_text, extract_name),
  age = purrr::map_int(age_free_text, extract_age)
)

# A tibble: 6 × 2
  name     age
  <chr>  <int>
1 Alex      42
2 Jamal     NA
3 Li Wei    NA
4 Fatima    NA
5 Robert    51
6 Kwame      5

age_free_text

[[1]]
[1] "I go by Alex. 42 years on this planet and counting."

[[2]]
[1] "Pleased to meet you! I'm Jamal, age 27."

[[3]]
[1] "They call me Li Wei. Nineteen years young."

[[4]]
[1] "Fatima here. Just celebrated my 35th birthday last week."

[[5]]
[1] "The name's Robert - 51 years old and proud of it."

[[6]]
[1] "Kwame here - just hit the big 5-0 this year."

But if you ask an LLM…

library(ellmer)

chat <- chat(
  "openai/gpt-5-nano",
  system_prompt = "Extract the name and age."
)

chat$chat(age_free_text[[1]])
#>

chat$chat(age_free_text[[2]])
#>

But if you ask an LLM…

library(ellmer)

chat <- chat(
  "openai/gpt-5-nano",
  system_prompt = "Extract the name and age."
)

chat$chat(age_free_text[[1]])
#> Name: Alex; Age: 42

chat$chat(age_free_text[[2]])
#> Name: Jamal; Age: 27

Wouldn’t this be nice?

chat$chat(age_free_text[[1]])
#> list(
#>   name = "Alex",
#>   age = 42
#> )

Structured chat output

chat$chat_structured(age_free_text[[1]])
#> list(
#>   name = "Alex",
#>   age = 42
#> )

Structured chat output

type_person <- type_object(
  name = type_string(),
  age = type_integer()
)

chat$chat_structured(age_free_text[[1]], type = type_person)
#> list(
#>   name = "Alex",
#>   age = 42
#> )

Structured chat output

type_person <- type_object(
  name = type_string(),
  age = type_integer()
)

chat$chat_structured(age_free_text[[1]], type = type_person)
#> $name
#> [1] "Alex"
#>
#> $age
#> [1] 42

ellmer’s type functions

type_person <- type_object(
  name = type_string("The person's name"),
  age = type_integer("The person's age in years")
)

⌨️ 10_structured-output

Instructions

1. We also have text versions of the recipes in data/recipes/txt.

2. Use ellmer::type_*() to extract structured data from the recipe you used in activity 09.

3. I’ve given you the expected structure, you just need to implement it.

07:00

Parallel and batch calls

Structured chat output

type_person <- type_object(
  name = type_string(),
  age = type_integer()
)

chat$chat_structured(age_free_text[[1]], type = type_person)
#> $name
#> [1] "Alex"
#>
#> $age
#> [1] 42

Structured chat output

type_person <- type_object(
  name = type_string(),
  age = type_integer()
)

chat$chat_structured(age_free_text, type = type_person)
#> ???

Structured chat output

type_person <- type_object(
  name = type_string(),
  age = type_integer()
)

chat$chat_structured(age_free_text, type = type_person)
#> Error in `FUN()`:
#> ! `...` must be made up strings or <content> objects, not a list.

Parallel chat calls

parallel_chat_structured(
  chat,
  age_free_text,
  type = type_person
)

Parallel chat calls

parallel_chat_structured(
  chat,
  age_free_text,
  type = type_person
)
#> [working] (0 + 0) -> 2 -> 4 | ■■■■■■■■■■■■■■■■■■■■■             67%

Parallel chat calls

parallel_chat_structured(
  chat,
  age_free_text,
  type = type_person
)
#>     name age
#> 1   Alex  42
#> 2  Jamal  27
#> 3 Li Wei  19
#> 4 Fatima  35
#> 5 Robert  51
#> 6  Kwame  50

Batch chat calls

batch_chat_structured(
  chat,
  age_free_text,
  type = type_person
)

Batch chat calls

batch_chat_structured(
  chat,
  age_free_text,
  type = type_person,
  path = "people.json"
)

Batch chat calls

chat <- chat("anthropic/claude-3-5-haiku-20241022")
batch_chat_structured(
  chat,
  age_free_text,
  type = type_person,
  path = "people.json"
)

Batch vs. parallel

Parallel

• 🌲 Works for any provider/model
   

• ⚡ Much faster for small jobs

• 💸 More expensive

• 😓 Not easy to stop/resume

Batch

• 🌱 Only works for some providers (OpenAI, Anthropic)

• ⏲️ Finishes… eventually

• 🏦 Much cheaper per prompt

• 😎 Easy to stop/resume

⌨️ 11_batch

Instructions

1. Take your type_recipe or Recipe model from activity 10…

2. And apply it to all the text recipes in data/recipes/txt.

3. Use parallel processing with gpt-4.1-nano from OpenAI.

4. Use the Batch API with Claude 3.5 Haiku from Anthropic.

5. Save the results to data/recipes/recipes.json and try out the Shiny recipe cookbook app!

07:00

Wrap-up

Recap

• LLM providers and models have different capabilities, costs, and speeds
• Choose models based on task requirements
• Multi-modal inputs allow LLMs to process images and PDFs
• Structured output can be achieved using type definitions
• Parallel and batch calls improve efficiency for large-scale LLM tasks

Acknowledgments

• Materials derived in part from Programming with LLMs and licensed under a Creative Commons Attribution 4.0 International (CC BY) License.