HW 06 - Labeling legislation with LLMs

Important

This homework is due November 18 at 11:59pm ET.

Learning objectives

• Use LLMs to perform a text classification task
• Write prompts in Markdown files and compare prompting strategies
• Extract structured output from LLMs at scale using batch processing
• Evaluate LLM performance against human-coded labels
• Weigh performance against cost when choosing a model

Getting started

• Go to the info5001-fa26 organization on GitHub. Click on the repo with the prefix hw-06. It contains the starter documents you need to complete the homework.

• Clone the repo and start a new workspace in Positron. See the Homework 0 instructions for details on cloning a repo and starting a new R project.

Note🔓 Decrypt the .Renviron.secret → .Renviron

1. source("secret.R")
2. The special phrase is:
   info-5001

General guidance

TipGuidelines + tips

As we’ve discussed in lecture, your plots should include an informative title, axes should be labeled, and careful consideration should be given to aesthetic choices.

Remember that continuing to develop a sound workflow for reproducible data analysis is important as you complete the homework and other assignments in this course. There will be periodic reminders in this assignment to remind you to render, commit, and push your changes to GitHub. You should have at least 3 commits with meaningful commit messages by the end of the assignment.

TipWorkflow + formatting

Make sure to

• Update author name on your document.
• Label all code chunks informatively and concisely.
• Follow the Tidyverse code style guidelines.
• Make at least 3 commits.
• Resize figures where needed, avoid tiny or huge plots.
• Turn in an organized, well formatted document.

WarningPresenting results of multiple models

You will compare nine combinations of models and prompts. Do not print raw R output and leave it to us to make sense of it.

• Condense the results into one or a handful of well-designed graphs or tables.
• Your plots should include an informative title, axes should be labeled, and careful consideration should be given to aesthetic choices.
• You can create simple formatted tables using {gt}.

ImportantStart the LLM work early

Batch jobs can take up to 24 hours to finish. Plan to submit your batch jobs for Exercise 5 so you have plenty of time to analyze the results.

Labeling legislative documents

The dataset data/leg_lite.feather contains descriptions of 500 bills introduced in the U.S. Congress since 2010. Each bill was coded by trained human researchers into one of the Comparative Agendas Project’s major policy categories. We will treat those human-coded labels as the truth and see how well an LLM can reproduce them.

Sample of legislative descriptions with policy categories

Description	Category number	Policy category label
A bill to amend title 38, United States Code, to improve the disability compensation evaluation procedure of the Secretary of Veterans Affairs for veterans with mental health conditions related to military sexual trauma, and for other purposes.	3	Health
A bill to amend the Energy Policy and Conservation Act to improve energy efficiency of certain appliances and equipment, and for other purposes.	8	Energy
To establish the Maumee Valley National Heritage Area in Ohio and Indiana, and for other purposes.	8	Energy
To direct the Secretary of the Army to undertake a comprehensive review of the Corps of Engineers policy guidelines on vegetation management for levees, and for other purposes.	20	Public lands and water management
To authorize the Secretary of Veterans Affairs to make grants to eligible educational institutions to provide child care services on campus.	6	Education

The dataset contains the following variables:

Variable	Description
id	Unique identifier for the bill
year	Year the bill was introduced
cong	Session of Congress
bill_type	HR (House) or S (Senate)
bill_no	Bill number
description	Short description of the bill
policy	Policy category number (1-20)
policy_label	Policy category label

Comparative Agendas Project major policy categories

Category number	Policy category label
1	Macroeconomics
2	Civil rights, minority issues, civil liberties
3	Health
4	Agriculture
5	Labor and employment
6	Education
7	Environment
8	Energy
9	Immigration
10	Transportation
11	Law, crime, family issues
12	Social welfare
13	Community development and housing issues
14	Banking, finance, and domestic commerce
15	Defense
16	Space, technology, and communications
17	Foreign trade
18	International affairs and foreign aid
19	Government operations
20	Public lands and water management

NoteDifferences from the codebook

The data uses the 2014 version of the codebook, which differs from the most recent codebook published online. You can find the major policy topics summarized on page 65. The original codebook does not include anything for number 11. This dataset fills in the gap by shifting all the larger numbers down by 1 (i.e., 12 becomes 11, 13 becomes 12, etc.). The topic “Other/miscellaneous” has been removed.

Use the numbers in the table above, not the numbers in the PDF.

Metrics used in this assignment

You already know accuracy: the proportion of bills where the predicted category matches the true category. We will also use the following metrics, defined here. Compute each of them yourself with {dplyr}.

NoteMetric definitions

Invalid rate. The proportion of bills where the LLM returned a category number that is not one of the 20 valid codes (e.g. 0, 21, or missing). Invalid predictions count as incorrect when computing every other metric.

Category hit rate. For a single category \(k\), the proportion of bills truly in category \(k\) that the LLM labeled as \(k\).

\[ \text{hit rate}_k = \frac{\text{number of bills in category } k \text{ labeled } k}{\text{number of bills in category } k} \]

Balanced accuracy. The average of the 20 category hit rates.

\[ \text{balanced accuracy} = \frac{1}{20} \sum_{k = 1}^{20} \text{hit rate}_k \]

Regular accuracy is dominated by the largest categories. Balanced accuracy gives each category equal weight, so a model cannot score well by doing great on Health bills and badly on Immigration bills.

Agreement rate. When the same model and prompt label the same bill several times, the proportion of bills that received the same category on every run.

WarningDon’t use yardstick::bal_accuracy()

{yardstick} has a function named bal_accuracy(), but its multiclass definition is different from the one above. Compute balanced accuracy yourself using the definition above.

Separating LLM calls from analysis

The repo separates code that calls an LLM from code that analyzes the results. We don’t want to send requests to the API every time we render the Quarto document. The following table shows where to put your code and prompts.

File	Purpose
scripts/leg-label.R	All code that sends requests to an LLM. To be run interactively. It saves the results to data/.
prompts/cap-naive.md, prompts/cap-codebook.md, prompts/cap-engineered.md	Your three prompts.
hw-06-leg-label.qmd	Your write-up to be submitted via Gradescope. It reads the saved results from data/ and never calls an LLM.

Part 1: Get to know the task

Exercise 1

Explore the data. Make a plot showing the number of bills in each policy category.

Suppose you built a “classifier” that ignores the description and always predicts the most common category. What would its accuracy be? What would its balanced accuracy be? These are the baselines any LLM needs to beat.

Exercise 2

Label some bills yourself. The template draws a random sample of 10 bills (with a fixed seed so everyone gets the same 10) and prints only their descriptions.

1. Before looking at the true labels, read each description and assign it a category number using the table above. Record your answers in the template.
2. Compute your accuracy on these 10 bills.
3. Which bills did you get wrong? For two of them (or, if you missed fewer than two, the ones you found hardest), explain why you chose the category you did and whether you think the human coder’s label is clearly better, or whether the bill could reasonably fit more than one category.

Part 2: Generate labels with LLMs

Exercise 3

Write three prompts. Write each prompt in its own Markdown file in prompts/. Each prompt must include the placeholder {{LEGISLATIVE_DESCRIPTION}} where the bill description will be inserted by interpolate_file().

• Naive (cap-naive.md). Ask the LLM to assign the description a Comparative Agendas Project major policy category number. Do not list the categories. The prompt must be fewer than 100 words (not counting the legislative description itself).
• Codebook (cap-codebook.md). The naive prompt plus the full list of 20 category numbers and labels.
• Engineered (cap-engineered.md). Apply the prompt engineering practices from lecture to write the best prompt you can. At minimum, it should:
  • Briefly define what each category covers. You can draw on the codebook (remember the numbering differences).
  • Tell the model how to choose when a bill touches more than one category.
  • Include at least three example descriptions with their correct categories. Write your own examples rather than copying descriptions from leg_lite.feather, otherwise the model would be graded on bills it was given the answers to.

You may use an LLM to help draft the engineered prompt. If you do, say so and describe what you changed from its draft.

Your prompts are embedded in your rendered document automatically.

Exercise 4

Define the output and test your code. In scripts/leg-label.R:

1. Define a type specification with type_object() containing at least these three fields, in this order:
   • reasoning: a short explanation of why the bill belongs in the chosen category
   • policy_code: the category number (use type_integer())
   • confidence: the model’s confidence in its label, from 0 to 100 (use type_integer())
2. Write a function that takes a model name, a prompt file, and a data frame of bills, and returns a data frame with one row per bill containing the bill id, the true policy, the model’s three fields, and the token usage and cost. Give it an option to use either parallel_chat_structured() or batch_chat_structured().
3. Test your function on 5 bills with the naive prompt and an inexpensive model using parallel processing. Check to make sure the output is in the correct structure.

TipWriting the labeling function

Look back at 11_batch from AE 18 for an example of batch_chat_structured(). Some arguments you’ll need:

• interpolate_file() accepts a vector of values for a placeholder, and returns one prompt per value.
• include_tokens = TRUE and include_cost = TRUE add token and cost columns to the results.
• batch_chat_structured() needs a path to save its progress. Use a different file for every model/prompt combination. If you run the same code again with the same path, it picks up the saved job instead of submitting (and paying for) a new one.
• wait = FALSE returns immediately instead of blocking your R session. It returns NULL until the job is finished, so make sure your function handles that. Re-run the same call later to collect the results.

Exercise 5

Label all 500 bills. Run every combination of three models and your three prompts (nine combinations in total) on all 500 bills. Use batch processing for every model hosted by OpenAI or Anthropic.

Choose three models that meet these requirements:

• At least one from OpenAI and at least one from Anthropic.
• At least one small, inexpensive model and at least one larger, more capable model.

Good options include openai/gpt-6-luna, openai/gpt-5.6-terra, anthropic/claude-haiku-4-5, and anthropic/claude-sonnet-4-6. Use models_openai() and models_anthropic() to see pricing for each.

Combine the results from all nine combinations into one data frame with columns identifying the model and prompt, and save it as data/leg-labels.feather. In your write-up, state which three models you chose and why.

NoteOptional: try a local model

If you have a computer with enough memory, you may replace one of the three models with an open-weight model you run locally, as discussed in lecture 20. This won’t work on Posit Workbench – you have to run Positron locally on your own machine.

• Install Ollama or LM Studio and download a model that supports structured output. Smaller models (roughly 3-8 billion parameters) will run on most laptops.
• Connect with chat_ollama(model = "...") or chat_lmstudio(model = "...").
• Local models can’t use batch processing. Use parallel_chat_structured(), and expect it to run much slower than a hosted model. Try 5 bills first and time it to estimate how long 500 will take.
• Local models cost $0 per token. Record how long the run took, and the model’s size, so you can discuss it in Exercise 7.

You must still use at least one OpenAI or Anthropic model. If your local model can’t produce usable structured output, report what happened and fall back to a hosted model.

Now is a good time to render, commit (with a descriptive and concise commit message), and push again. Make sure that you commit and push all changed documents and your Git pane is completely empty before proceeding.

Part 3: Evaluate the labels

All the remaining exercises use the saved data/leg-labels.feather file, read into your Quarto document.

Exercise 6

Compare performance. For each of the nine model/prompt combinations, calculate:

• Accuracy
• Balanced accuracy
• Invalid rate

Present the results in a single well-designed table or figure. Then answer:

• Which combination performed best? Does the answer depend on whether you use accuracy or balanced accuracy?
• How much did the prompt matter compared to the model?
• How do the LLMs compare to the baselines from Exercise 1?

Exercise 7

Consider cost. For each combination, calculate the total cost. Make a scatterplot of accuracy versus total cost, with one point per combination, that makes it easy to tell which model and which prompt each point represents.

Which combination is the best value for money? Imagine you had to label all 383,987 bills in the full dataset. Using your results, estimate what your recommended combination would cost.

If you used a local model, evaluate it in terms of computational efficiency. How long would it take to label all 383,987 bills?

Note{ellmer} reports on estimated costs

{ellmer} estimates the costs associated with each conversation based on the number of tokens used and the model provider’s published pricing. It does not take into account the batch processing discount – the reported costs from {ellmer} are twice the actual costs for OpenAI and Anthropic models.

Exercise 8

Look at the mistakes. Choose the best-performing combination from Exercise 6 and make a confusion matrix for it, drawn as a heatmap.

With 20 categories, a confusion matrix of raw counts is hard to read, and it hides mistakes in the small categories. Instead:

• Recode any invalid predictions to a single “Invalid” category.
• Count the bills for each (true category, predicted category) pair.
• Within each true category, convert the counts to proportions so each row sums to 1. The diagonal cells are now the category hit rates, and the other cells in a row show where that category’s bills ended up.
• Plot the proportions with geom_tile(), with the true category on one axis and the predicted category on the other.

Then answer:

1. Which three categories does the model struggle with most?
2. What are the most common confusions (true category labeled as some other category)? Do they make sense given what the categories cover?
3. Are these mistakes a human coder could plausibly make too? Think back to Exercise 2.

Exercise 9

Is the model’s confidence meaningful? Using the same best-performing combination, group bills into confidence bins (e.g. 0-49, 50-69, 70-79, 80-89, 90-100, or bins of your choice that make sense for your data). For each bin, calculate the number of bills and the accuracy, and plot the results.

If the model’s confidence were meaningful, what pattern would you expect to see? Do you see it? Would you use the confidence score to decide which bills a human should double-check?

Exercise 10

Is the model consistent? LLMs don’t always give the same answer to the same question. Using your best-performing combination, label a random sample of 100 bills two more times (generate the labels in your R script). Together with your original run, you now have three labels for each of those 100 bills.

1. Calculate the agreement rate.
2. Compare the accuracy (using the original run’s labels) for bills where all three runs agreed versus bills where they did not.
3. In 2-3 sentences, explain what this means for someone who wants to use an LLM to label data for a research project.

Render, commit, and push one last time. Make sure that you commit and push all changed documents and your Git pane is completely empty before proceeding.

Wrap up

Submission

• Go to http://www.gradescope.com and click Log in in the top right corner.
• Click School Credentials \(\rightarrow\) Cornell University NetID and log in using your NetID credentials.
• Click on your INFO 5001 course.
• Click on the assignment, and you’ll be prompted to submit it.
• Mark all the pages associated with exercise. All the pages of your homework should be associated with at least one question (i.e., should be “checked”).
• Select all pages of your .pdf submission to be associated with the “Workflow & formatting” question.

Grading

• Exercise 1: 2 points
• Exercise 2: 4 points
• Exercise 3: 7 points
• Exercise 4: 4 points
• Exercise 5: 6 points
• Exercise 6: 7 points
• Exercise 7: 4 points
• Exercise 8: 4 points
• Exercise 9: 4 points
• Exercise 10: 4 points
• Workflow + formatting: 4 points
• Total: 50 points

NoteWorkflow & formatting criteria

The “Workflow & formatting” component assesses the reproducible workflow. This includes:

• Following {tidyverse} code style
• All code being visible in rendered PDF without automatic wrapping (no more than 80 characters)
• Appropriate figure sizing, and figures with informative labels and legends
• Keeping all LLM calls in scripts/leg-label.R so the Quarto document renders without calling an API