library(tidyverse)
library(httr2)
library(jsonlite)AE 12: Querying the OMDB API with httr2
Application exercise
Packages
We will use the following packages in this application exercise.
- tidyverse: For data import, wrangling, and visualization.
- httr2: For querying APIs.
- jsonlite: For some formatting
Writing an API function
If an R package has not already been written for an application programming interface (API), you can write your own function to query the API. In this application exercise, we will write a function to query the Open Movie Database.
httr vs. httr2
The httr package is the most popular package for querying APIs in R. However, the httr2 package is a newer package that is more consistent with the tidyverse and has a more intuitive syntax. The developers of httr have superceded it with httr2 and recommend all new software development moving forward use httr2. Therefore we will use httr2 in this application exercise.
Create a request
The first step in querying an API is to create a request. A request is an object that contains the information needed to query the API. The request() function creates a request object. The base_url argument specifies the base URL of the API. The req <- syntax assigns the request object to the variable req.
omdb_req <- request(base_url = "http://www.omdbapi.com/")
omdb_req<httr2_request>GET http://www.omdbapi.com/Body: emptyPerform a dry run
The req_dry_run() function performs a dry run of the request. A dry run is a test run of the request that does not actually query the API. It is useful for testing the request before actually querying the API.
omdb_req |>
req_dry_run()GET / HTTP/1.1
Host: www.omdbapi.com
User-Agent: httr2/0.2.3 r-curl/5.0.1 libcurl/8.1.2
Accept: */*
Accept-Encoding: deflate, gzipDetermine the shape of an API request
In order to submit a request, we need to define the shape of the request, or the exact URL used to submit the request. The URL of the request is the base URL of the API plus the path of the request.
APIs typically have three major components to the request:
- The base URL for the web service (here it is
http://www.omdbapi.com/). - The resource path which is the complete destination of the web service endpoint (OMDB API does not have a resource path).
- The query parameters which are the parameters passed to the web service endpoint.
In order to create your request you need to read the documentation to see exactly how these components need to be specified.
Your turn: Use the OMDB documentation to determine the shape of the request for information on Sharknado.
# add an example request hereGenerate the query
Store your API key
In order to access the OMDB API you need an API key. If you do not have one, use the example key provided on Canvas.
Your turn: Store your API key in .Renviron so you can access it in your code. Once you have saved the file, restart your R session to ensure the new environment variable is loaded.
# from the console run:
usethis::edit_r_environ()
# in .Renviron add:
omdb_key="your-key"# add code hereFetch the response
The req_perform() function fetches the response from the API. The response is stored as a response object.
# add code hereWhat did we get?
The HTTP response contains a number of useful pieces of information.
Status code
The status code is a number that indicates whether the request was successful. A status code of 200 indicates success. A status code of 400 or 500 indicates an error. resp_status() retrieves the numeric HTTP status code, whereas resp_status_desc() retrieves a brief textual description of the status code.
# add code hereHTTP status codes
Hopefully all you receive is a 200 code indicating the query was successful. If you get something different, the error code is useful in debugging your code and determining what (if anything) you can do to fix it
| Code | Status |
|---|---|
| 1xx | Informational |
| 2xx | Success |
| 3xx | Redirection |
| 4xx | Client error (you did something wrong) |
| 5xx | Server error (server did something wrong) |
Body
The body of the response contains the actual data returned by the API. The body is a string of characters.
You can extract the body in various forms using the resp_body_*() family of functions. The resp_body_string() function retrieves the body as a string.
# add code hereJSON
Here the result is actually formatted as using JavaScript Object Notation (JSON), so we can use resp_body_json() to extract the data and store it as a list object in R.
# add code hereConvert to data frame
For data analysis purposes, we prefer that the data be stored as a data frame. The as_tibble() function converts the list object to a tibble.
# add code hereWrite a function to query the OMDB API
Sharknado proved so popular that four sequels were made. Let’s write a function to query the OMDB API for information on any of the Sharknado films.
Your turn: Your function should:
- Take a single argument (the title of the film)
- Print a message using the
message()function to track progress - Use throttling to ensure we do not overload the server and exceed any rate limits. Add
req_throttle()to the request pipeline to limit the rate to 15 requests per minute. - Return a tibble with the information from the API
omdb_api <- function(title){
# add code here
}Once you have written your function, test it out by querying the API for information on “Sharknado 2”. Then apply an iterative operation to query the API for information on all five Sharknado films and store it in a single data frame.
# test function
omdb_api(title = "Sharknado")NULL# titles of films
sharknados <- c(
"Sharknado", "Sharknado 2", "Sharknado 3",
"Sharknado 4", "Sharknado 5"
)
# iterate over titles and query API
sharknados_df <- map(.x = sharknados, .f = omdb_api) |>
list_rbind()
sharknados_dfdata frame with 0 columns and 0 rowsAcknowledgments
- These exercises draw substantially on the httr2 vignettes and reference documentation.
Session information
sessioninfo::session_info()─ Session info ───────────────────────────────────────────────────────────────
setting value
version R version 4.3.1 (2023-06-16)
os macOS Ventura 13.5.2
system aarch64, darwin20
ui X11
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz America/New_York
date 2023-10-16
pandoc 3.1.1 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown)
─ Packages ───────────────────────────────────────────────────────────────────
package * version date (UTC) lib source
cli 3.6.1 2023-03-23 [1] CRAN (R 4.3.0)
colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.3.0)
curl 5.0.1 2023-06-07 [1] CRAN (R 4.3.0)
digest 0.6.31 2022-12-11 [1] CRAN (R 4.3.0)
dplyr * 1.1.2 2023-04-20 [1] CRAN (R 4.3.0)
evaluate 0.21 2023-05-05 [1] CRAN (R 4.3.0)
fansi 1.0.4 2023-01-22 [1] CRAN (R 4.3.0)
fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.3.0)
forcats * 1.0.0 2023-01-29 [1] CRAN (R 4.3.0)
generics 0.1.3 2022-07-05 [1] CRAN (R 4.3.0)
ggplot2 * 3.4.2 2023-04-03 [1] CRAN (R 4.3.0)
glue 1.6.2 2022-02-24 [1] CRAN (R 4.3.0)
gtable 0.3.3 2023-03-21 [1] CRAN (R 4.3.0)
here 1.0.1 2020-12-13 [1] CRAN (R 4.3.0)
hms 1.1.3 2023-03-21 [1] CRAN (R 4.3.0)
htmltools 0.5.5 2023-03-23 [1] CRAN (R 4.3.0)
htmlwidgets 1.6.2 2023-03-17 [1] CRAN (R 4.3.0)
httpuv 1.6.11 2023-05-11 [1] CRAN (R 4.3.0)
httr2 * 0.2.3 2023-05-08 [1] CRAN (R 4.3.0)
jsonlite * 1.8.5 2023-06-05 [1] CRAN (R 4.3.0)
knitr 1.43 2023-05-25 [1] CRAN (R 4.3.0)
later 1.3.1 2023-05-02 [1] CRAN (R 4.3.0)
lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.3.0)
lubridate * 1.9.2 2023-02-10 [1] CRAN (R 4.3.0)
magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.3.0)
munsell 0.5.0 2018-06-12 [1] CRAN (R 4.3.0)
pillar 1.9.0 2023-03-22 [1] CRAN (R 4.3.0)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.3.0)
promises 1.2.0.1 2021-02-11 [1] CRAN (R 4.3.0)
purrr * 1.0.1 2023-01-10 [1] CRAN (R 4.3.0)
R6 2.5.1 2021-08-19 [1] CRAN (R 4.3.0)
rappdirs 0.3.3 2021-01-31 [1] CRAN (R 4.3.0)
Rcpp 1.0.10 2023-01-22 [1] CRAN (R 4.3.0)
readr * 2.1.4 2023-02-10 [1] CRAN (R 4.3.0)
rlang 1.1.1 2023-04-28 [1] CRAN (R 4.3.0)
rmarkdown 2.22 2023-06-01 [1] CRAN (R 4.3.0)
rprojroot 2.0.3 2022-04-02 [1] CRAN (R 4.3.0)
rstudioapi 0.14 2022-08-22 [1] CRAN (R 4.3.0)
scales 1.2.1 2022-08-20 [1] CRAN (R 4.3.0)
sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.0)
stringi 1.7.12 2023-01-11 [1] CRAN (R 4.3.0)
stringr * 1.5.0 2022-12-02 [1] CRAN (R 4.3.0)
tibble * 3.2.1 2023-03-20 [1] CRAN (R 4.3.0)
tidyr * 1.3.0 2023-01-24 [1] CRAN (R 4.3.0)
tidyselect 1.2.0 2022-10-10 [1] CRAN (R 4.3.0)
tidyverse * 2.0.0 2023-02-22 [1] CRAN (R 4.3.0)
timechange 0.2.0 2023-01-11 [1] CRAN (R 4.3.0)
tzdb 0.4.0 2023-05-12 [1] CRAN (R 4.3.0)
utf8 1.2.3 2023-01-31 [1] CRAN (R 4.3.0)
vctrs 0.6.3 2023-06-14 [1] CRAN (R 4.3.0)
withr 2.5.0 2022-03-03 [1] CRAN (R 4.3.0)
xfun 0.39 2023-04-20 [1] CRAN (R 4.3.0)
yaml 2.3.7 2023-01-23 [1] CRAN (R 4.3.0)
[1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
──────────────────────────────────────────────────────────────────────────────