Tool calling + agents

Lecture 22

Dr. Benjamin Soltoff

Cornell University
INFO 5001 - Fall 2026

November 12, 2026

Announcements

Announcements

• Homework 06

Learning objectives

• Identify the value of calling external tools from LLMs
• Define and call tools using {ellmer}
• Define an AI agent
• Identify the basic tools for AI agents
• Implement a simple AI agent in R
• Provide an AI agent with skills to complete a task

Application exercise

ae-20

Instructions

• Go to the course GitHub org and find your ae-20 (repo name will be suffixed with your GitHub name).
• Clone the repo in Positron, run renv::restore() to install the required packages, open the Quarto document in the repo, and follow along and complete the exercises.
• Render, commit, and push your edits by the AE deadline – end of the day

🔓 Decrypt the .Renviron.secret → .Renviron

1. source("secret.R")
2. The special phrase is:
   info-5001

Tool calling

A robot wearing a tool belt, selects a tool from a wall of futuristic space tools. A sign above the wall reads “Tools”.

Recall: How do LLMs work?

1. You write some words

2. The LLM writes some more words

3. You use those words

On their own, can LLMs… access the internet? send an email? interact with the world?

Let’s try it

library(ellmer)

chat <- chat("openai/gpt-6-luna")

chat$chat("What's the weather like in Ithaca, NY?")
chat$chat("What day is it?")

Run these in the terminal.

Ask: how could we get the model to know these answers? Let’s start with the easiest one: what day is it?

Answer: we could just tell it in the system prompt.

What about the weather? The model can’t know that because it’s not connected to the internet.

Tools

a.k.a. functions, tool calling or function calling

• Bring real-time or up-to-date information to the model
• Let the model interact with the world
• Foundation for agentic AI

Tools are extra capabilities that you give to the LLM, like the ability to search the posit::conf(2025) schedule. These capabilities are things that the model can’t do on its own. The goal is let the model pull in real-time, or up-to-date, or high-quality information as needed, depending on the conversation.

They can also give the model new abilities to take action on behalf of the user.

Chatbot systems

How do tool calls work?

What should I wear to campus tomorrow?

So how do we connect the LLM to the code that this data scientist has written (or could write) to give the model new abilities?

Human in the loop

👨‍💻 _demos/18_manual-tools/app.R

Run locally

Wait… I can write code!

library(ellmer)

ellmer::create_tool_def(weathR::point_forecast, verbose = TRUE)

tool(
  weathR::point_forecast,
  name = "point_forecast",
  description = "Retrieve point forecast meteorological data.",
  arguments = list(
    lat = type_number("Latitude."),
    lon = type_number("Longitude."),
    timezone = type_string(
      "An Olson timezone name. Defaults to -1 for local time.",
      required = FALSE
    ),
    dir_numeric = type_boolean(
      "Whether to return directions as numeric values rather than character values. Defaults to 
FALSE.",
      required = FALSE
    )
  )
)

Wait… I can write code!

get_weather <- tool(
  \(lat, lon) weathR::point_forecast(lat, lon),
  name = "point_forecast",
  description = "Get forecast data for a specific latitude and longitude.",
  arguments = list(
    lat = type_number("Latitude of the location."),
    lon = type_number("Longitude of the location.")
  )
)

Wait… I can write code!

get_weather(lat = 42.4397, lon = -76.4953)

Simple feature collection with 156 features and 8 fields
Geometry type: POINT
Dimension:     XY
Bounding box:  xmin: -76.4953 ymin: 42.4397 xmax: -76.4953 ymax: 42.4397
Geodetic CRS:  WGS 84
First 10 features:
                      time temp dewpoint humidity p_rain wind_speed wind_dir        skies
1  2026-11-12 09:00:00 EST   46 7.222222       96     79          3       NE Rain Showers
2  2026-11-12 10:00:00 EST   47 8.333333      100     89          5       NE Rain Showers
3  2026-11-12 11:00:00 EST   48 8.888889      100     96          6        N Rain Showers
4  2026-11-12 12:00:00 EST   48 8.888889      100     98          7        N Rain Showers
5  2026-11-12 13:00:00 EST   49 8.333333       93     90          8        N Rain Showers
6  2026-11-12 14:00:00 EST   48 7.777778       93     90          9       NE Rain Showers
7  2026-11-12 15:00:00 EST   47 7.222222       93    100          9        N Rain Showers
8  2026-11-12 16:00:00 EST   46 6.111111       89     91          9        N Rain Showers
9  2026-11-12 17:00:00 EST   44 5.555556       93     86          9        N Rain Showers
10 2026-11-12 18:00:00 EST   42 5.000000       96     85          9        N Rain Showers
                   geometry
1  POINT (-76.4953 42.4397)
2  POINT (-76.4953 42.4397)
3  POINT (-76.4953 42.4397)
4  POINT (-76.4953 42.4397)
5  POINT (-76.4953 42.4397)
6  POINT (-76.4953 42.4397)
7  POINT (-76.4953 42.4397)
8  POINT (-76.4953 42.4397)
9  POINT (-76.4953 42.4397)
10 POINT (-76.4953 42.4397)

Wait… I can write code!

chat <- chat_openai(model = "gpt-6-luna", echo = "output")

# Register the tool with the chatbot
chat$register_tool(get_weather)

chat$chat("What should I wear to class tomorrow in Ithaca, NY?")
● #> [{"time":"2026-11-12 09:00:00 EST","temp":46,"dewpoint":7.2222,"humidity":96,"p_rain":79…
#> Tomorrow in Ithaca looks **cold and wet**, around **32–39°F**, with rain/snow
#> possible—especially in the morning.

#> Wear a **warm base layer and sweater under a water-resistant coat**, plus **long pants
#> and waterproof shoes or boots**. Bring an **umbrella** or hood, and consider gloves
#> if you’ll be walking early. The sidewalks may be slick.

Zooming out again, the conversation starts with a user sending a message to the chatbot.

In this case, the user asks the model whether they can go to the pool today. (I live in Atlanta, GA.)

How much does the model know about the weather in Atlanta, GA?

What’s the weather like today in Atlanta, GA?

I’m unable to provide real-time weather updates. For the latest weather forecast in Atlanta, GA, please check a trusted weather service such as the National Weather Service, Weather.com, or a weather app on your device.

Go ahead and guess

Sure! Since it’s July in Atlanta, GA, you can generally expect hot and humid weather. Temperatures might range from the mid-80s to mid-90s Fahrenheit (around 29-35°C). There could also be some afternoon thunderstorms or showers, which are common during summer months in Georgia. Keep a fan or air conditioning handy if you’re indoors, and stay hydrated!

Now imagine we’ve told the LLM that it has access to a tool called get_weather(). Knowing that it 1) needs to know the weather in Atlanta and 2) it doesn’t know the weather in Atlanta, GA; the model decides to use the weather tool.

It sends back a message that includes a tool call, basically instructions to call the get_weather() function with the argument 30303. Notice that the model does know that Atlanta, GA is in the 30303 ZIP code.

The programmer can then take that message from the LLM with the instructions to call the tool and they go off and run the code that that satisfies the tool request, in this case querying the weather API for the weather in 30303 because that has been a stable fact for a long time.

The weather API sends back some data:

• The forecast is mostly sunny
• High of 98ºF
• Low of 78ºF

The programmer sends that data back to the LLM. Note that it’s often in a pretty raw format, like JSON. Not the kind of thing you’d want to read directly unless you’re the kind of person who likes reading computer data formats.

The LLM doesn’t mind JSON though, so it reads the data and generates a response that incorporates the data from the tool call.

Recap: Tool definitions in R

tool_get_weather <- tool(
  tool_fn,
  description = "How and when to use the tool",
  arguments = list(
    .... = type_string(),
    .... = type_integer(),
    .... = type_enum(
      c("choice1", "choice2"),
      required = FALSE
    )
  )
)

Agents

Agents

What’s an agent?

• An LLM, in a harness, that calls tools repeatedly in a loop

• Deciding each next step from the last result

How many tools does it take to make an agent?

An agent keeps calling tools in a loop, deciding each next step from the result of the last one.

What kind of tools?

• Read tools to observe the world

• Write tools to change it

This is Hadley Wickham’s definition, from his “What is an agent?” post on Substack.

Simon Willison’s shorter version: an LLM with a read tool and a write tool.

The “you know it when you see it” version: an autonomous system that needs minimal intervention to finish a task.

All three describe the same thing. The loop is what makes it an agent.

The agent loop

• Explore: the agent reads to understand the world

• Act: the agent writes to change it

• Observe: the agent reads again to see what changed

• …and repeat until the task is done

TODO: replace with an excalidraw diagram, in the style of the tools-how-it-works sequence in slides-07.

The loop is why agents feel different from a chat: the model, not the user, chooses the next step.

The harness (ellmer or chatlas) does the busy work: it runs each tool, sends the result back, and lets the model continue.

Brainstorm: tools for agents

Read tools

• Read a file

Write tools

• Write a file

Read and write is the smallest split that still makes an agent useful.

A coding agent is just an agent whose tools mirror how a human works on a codebase.

Hadley: “A coding agent is six functions in a trenchcoat” — read file, write file, edit file, list files, search, run command, plus a big system prompt of advice.

Oh, and prompting!

Blockbuster Agent

Welcome to the Last Blockbuster

• You work at the Last Blockbuster in the universe, in Bend, Oregon.

• The store runs on its annual membership renewal drive.

• Your job is to find members who have gone quiet and win them back.

This is the story world for the rest of the workshop: every exercise is a shift at the Last Blockbuster, with a real-ish records folder — members.csv, rentals.csv, dues.csv, and the manager’s notes.md.

The manager is a recurring character: a new task arrives mid-shift, or an old paper register turns up and changes yesterday’s task. When that happens in an exercise, roll with it — real work looks like that too.

A nice side effect: as the human, you’ll often know the ground truth before you hand the task to the agent. That’s what makes it possible to judge the agent’s work instead of just accepting it.

⌨️ 19_agent-1

1. Your agent works in the Last Blockbuster’s records folder.

2. Wrap up a read tool and a write tool for it to use, then ask it to build the win-back list.

3. Run the script it writes, then inspect the trace. Did it read the store rules before it wrote code?

05:00

The coding-agent system prompt is given. Attendees wrap the file operations into functions (inputs, work, outputs), document the tools, register them, and ask the agent to write find_lapsed.R.

The attendee runs that script from inside blockbuster/.

Watch for the key choice: the agent writes a script instead of trying to read 1,500 rental rows through the tool.

Dry run in R: the trace reads README.md, then notes.md, then writes find_lapsed.*. Both generated scripts produced 11 rows.

Discussion: Did your agent…?

• Read README.md and notes.md before it decided anything?

• Write a script, instead of reading all 1,500 rentals through its tools?

• Say things about the store’s rules that it never actually read?

• Would you hand this list to your manager?

• If you ran the code: how many lapsed members did it find?

Most agents find all 11 lapsed members but skip dues.csv, the unreturned-tape rule, or the sort order — so the lists differ by 2–3 names.

The hallucination question is the important one: agents confidently cite “store rules” they never read. Compare what your agent claimed against notes.md.

Ask the room: the manager calls whoever is on this list — what would make you trust it?

⌨️ 20_agent-2

1. Give your agent a list files tool and an edit tool.

2. Build the win-back list script again. Do these tools make your agent more effective?

3. Your manager found an old paper register!

4. Without naming the file, ask the agent to bring the list up to date.

05:00

list_files surfaces rentals-old.csv without the prompt naming it.

edit_file patches find_lapsed.R instead of rewriting it.

A failed exact-match edit is a loud, safe error instead of a silent change.

The count changing from 11 to 8 makes the effect of the two new tools visible.

Dry run in R: the follow-up trace calls list_files, then patches find_lapsed.* with edit_file. Both patched scripts produced eight rows without Greta Moss, Peter Ibarra, or Nora Ellis.

Discussion: Did your agent…?

• Write a better find_lapsed script than in the last exercise, even before the export arrived?

• Use list_files to find the export, instead of guessing the filename?

• Patch the find_lapsed script with edit_file, or rewrite the whole script?

• Hit an edit_file error — and was the error loud and safe?

• Still get the right answer even when it skipped a tool?

Yes to the first question: every test run’s first script encoded the full lapsed rules — the dues window, the unreturned-tape exception, the tier ordering — and named the right 11 members, while last exercise’s scripts always missed two or three. The difference tracked whether the agent read README.md and notes.md first: it did every time here, and never in the last exercise.

In test runs the agent always found the export via list_files, but patched the script only about half the time — sometimes it rewrote the file with write_file, and sometimes its prompt-1 script already globbed rentals*.csv, so no patch was needed.

The edit_file error question matters: wrong argument names produced a loud error the agent recovered from, which is exactly the failure mode you want compared to a silent wrong edit.

Close on the last bullet: tools guide the agent, but they don’t force it — the same answer can arrive by different routes, which is why traces matter.

Agents, recap

• An agent is an LLM, in a harness, calling tools in a loop

• Read tools observe; write tools change

• The harness does the busy work — you write the tools and the system prompt

Next: how to give an agent knowledge without making the system prompt enormous.

Agent Skills

Agent Skills

The same format works across agents: Posit Assistant, Claude Code, and many others follow the Agent Skills spec.

Skills are plain markdown, checked into your project — no packaging, no build step.

SKILL.md

---
name: renewal-letters
description: Draft a renewal letter for a lapsed Last Blockbuster member.
---

# Renewal letters

Offer Basic members a free-rental code.
Offer Family members a family-night bundle.
Offer Founders members a commemorative card.

Two required frontmatter fields:

• name: lowercase, hyphens, max 64 characters, must match the folder name.
• description: max 1024 characters, and it must say what the skill does AND when to use it.

The description is the only thing the agent sees at startup, so it decides whether your skill ever gets used.

The body is free markdown: steps, rules, gotchas, a template.

Anatomy of a skill

.agents/skills/
└── renewal-letters/
    ├── SKILL.md      # Required: metadata + instructions
    ├── references/   # Optional: documentation
    ├── scripts/      # Optional: executable code
    └── assets/       # Optional: templates, data files

Progressive disclosure

1. Discovery
   the agent starts with name and description

2. Activation
   when it decides to, the agent reads the full SKILL.md

3. Execution
   after that, it’s in the agent’s robot hands

This is why skills scale: you can keep 100 skills on hand for the cost of their descriptions.

The full instructions load only when a task calls for them.

Same idea you’ll implement by hand in the exercise: the system prompt lists the skill, a tool loads the body.

TODO: excalidraw diagram in the style of the tools sequence in slides-07.

Why not just use the system prompt?

System prompt

instructions for every session

Skills

on-demand instructions

• The system prompt is always in context — every token, every request

• Skills load on demand — only when a task matches

• So skills let you give an agent deep expertise without paying for it up front

You wrote a system prompt for the Blockbuster agent. That works for one small set of rules.

Now imagine 50 jobs: renewal letters, store opening procedures, staff notes, rental policies, and social media voice.

Stuffed into the system prompt, they crowd out the conversation and cost tokens on every request.

As skills, they cost about 100 tokens each until the moment one is needed.

System prompt vs. user prompt

Short answer: Put instructions and background knowledge in the system prompt.

Put the current task in the user prompt.

Anthropic’s Prompt Engineering Overview and OpenAI’s OpenAI Cookbook are excellent, and contain lots of tips and examples.

Google’s Prompt Design Strategies may also be useful.

The scene: win-back letters

Your win-back work paid off and found three Founders members we really want to keep. Your job is to write those letters.

Member	Tier	Last rented
Elaine Wu	Founders	The Princess Bride
Theo Bennett	Founders	The Matrix
Marisol Vega	Founders	The Goonies

This is the ground truth for the exercise: the win-back list from the last round narrowed to three members, and all three are Founders tier — the ones we most want to keep — so each letter should offer the commemorative card, not a free-rental code or family bundle.

A good letter also greets the member by name, references their last rented title, invites them back, and never mentions overdue fees or unlisted discounts. Letters land in letters/drafts/, named like elaine-wu.md.

You don’t need to read the skill or the CSVs in detail to judge the agent: right three people, right perk, warm voice, no fee talk, three files.

If your agent drafts for a Basic member like Malik Johnson, or invents a tier fact, you’ll now spot it immediately.

⌨️ 21_skills-1

1. Your Blockbuster agent gets a skill: renewal-letters holds the letter-writing instructions.

2. List the skill in the system prompt, and give the agent a tool to read it.

3. Draft letters for the top three lapsed members. Watch the agent read the skill before it starts drafting — and check them against the scene we just saw.

05:00

The read_skill tool is like the read_file tool, pointed at the skills folder.

The system prompt listing is the manual version of what real harnesses generate automatically at startup.

The trace should show read_skill firing when the agent starts the letter task.

Discussion: Did your agent…?

• Read the skill before it drafted a single letter?

• Pick the right top three members from lapsed.csv?

• Invent tier facts — a Founders perk for someone who isn’t one?

• Mention overdue fees, even though the skill forbids it?

• Leave the data files untouched?

Stress-test evidence, five runs: the mechanism you built works — read_skill fired first in every single run, before any drafting.

But the agent picked the wrong “top three” in one of five runs, drafting for Malik Johnson instead of Elaine Wu — reading the file isn’t the same as reading it carefully.

That same run compounded the error: it called a Basic-tier member “one of our Founders members” and gave him a commemorative card. Wrong pick, wrong perk.

Another run wrote “late fees we won’t dwell on” in a letter — a direct brush with the skill’s no-fees rule.

The good news: in all five runs the data files were left untouched, tool calls were clean, and four of five letter sets were correct.

Close on the failure modes: skills load reliably, but grounding — who are the top three, what tier are they — still depends on the agent checking its sources.

Writing a skill that works

• Add what the agent lacks; omit what it knows

• The description is the trigger

• Concrete beats generic

• Keep it short

The test for every line: would the agent get this wrong without it?

If the agent already writes clean markdown, don’t tell it to write clean markdown.

Give a default, not a menu: one way to do it, with an escape hatch.

Gotchas are the highest-value content: the environment-specific facts that defy reasonable assumptions.

⌨️ 22_skills-2

1. The skill has been swapped for a first draft with common problems.

2. Use the diagnostic questions to fix SKILL.md, then re-run the letter task.

3. Compare: same agent, same tools, different skill, different letters and token spend.

05:00

The planted problems: the description is background about the manager’s notes, while the actual trigger sits in the body. The body also repeats policies, generic advice, and conflicting process instructions.

Four questions lead attendees from the misplaced trigger to the repeated facts and conflicting process. They move and delete text; they do not need to invent new instructions.

The good skill says when to draft a renewal letter and keeps only the voice, letter structure, tier incentives, filename rule, and promises caveat.

Discussion: Did your agent…?

• Pick renewal-letters first, out of the four skills on offer?

• Read a second skill before it started writing?

• Follow the bad skill’s rules straight past the letter checklist?

• Treat a vague description as a reason to hedge?

• Write the same letters before and after your fix?

Stress-test evidence, three runs per skill version, four skills competing: the fixed skill was read first and only, in every run. Rival skills never won the first slot.

But under the bad skill, the agent read a second skill (lapsed-audit) in two of three runs — a vague description with no use-when boundary made it hedge, and hedging costs tokens and attention.

The bad skill’s planted rules fought the checklist: “mention the title only if it was a recent release” deleted all three title references in one run, and “start every letter with the tier incentive” was ignored in two of three. A bad skill doesn’t just misinform — it makes behavior unpredictable.

What stayed right the whole time: correct members, correct Founders perk, correct filenames, no fee talk, in all six runs.

Close on the before/after: same agent, same tools, same task — the only change was the skill, and the letters changed with it.

Skills in the ecosystem

skills.sh

posit-dev/skills

npx skills add

Can you install the skill from…

AGENTS.md and the system prompt shape everything the agent does.

Skills carry the long tail of knowledge: too specific to always pay for, too valuable to leave out.

MCP servers are about tools — connections to databases, APIs, other software. Bonus material covers MCP.

Skills, recap

• A skill is 📂 with SKILL.md: metadata plus instructions

• Progressive disclosure: descriptions init, body on demand

• Same format in Claude Code, Codex, Posit Assistant

You built the mechanism yourself today: listing, loading, and writing skills.

Next: put skills to work inside the assistant you use every day.

Wrap-up

Recap

• Tool calling lets LLMs access real-time information and interact with the world
• We can define tools in R using {ellmer}
• An AI agent is an LLM in a harness that calls tools repeatedly in a loop
• Agents need read tools to observe and write tools to change
• Skills are a way to give agents deep expertise without bloating the system prompt

Acknowledgments

• Materials derived in part from Programming with LLMs and licensed under a Creative Commons Attribution 4.0 International (CC BY) License.