tree_mod <- decision_tree(engine = "rpart") |>
set_mode("classification")
tree_wf <- workflow() |>
add_formula(children ~ .) |>
add_model(tree_mod)AE 21: Tune better models to predict children in hotel bookings
Suggested answers
Application exercise
Answers
Your Turn 1
Fill in the blanks to return the accuracy and ROC AUC for this model using 10-fold cross-validation.
Fill in the blanks to return the accuracy and ROC AUC for this model using 10-fold cross-validation.
set.seed(100)
______ |>
______(resamples = hotels_folds) |>
______Answer:
set.seed(100)
tree_wf |>
fit_resamples(resamples = hotels_folds) |>
collect_metrics()# A tibble: 3 × 6
.metric .estimator mean n std_err .config
<chr> <chr> <dbl> <int> <dbl> <chr>
1 accuracy binary 0.773 10 0.00567 pre0_mod0_post0
2 brier_class binary 0.158 10 0.00322 pre0_mod0_post0
3 roc_auc binary 0.832 10 0.00672 pre0_mod0_post0Your Turn 2
Create a new parsnip model called rf_mod, which will learn an ensemble of classification trees from our training data using the {ranger} package. Update your tree_wf with this new model.
Fit your workflow with 10-fold cross-validation and compare the ROC AUC of the random forest to your single decision tree model — which predicts the test set better?
Hint: you’ll need https://www.tidymodels.org/find/parsnip/
# model
rf_mod <- _____ |>
_____("ranger") |>
_____("classification")
# workflow
rf_wf <- tree_wf |>
update_model(_____)
# fit with cross-validation
set.seed(100)
_____ |>
fit_resamples(resamples = hotels_folds) |>
collect_metrics()Answer:
# model
rf_mod <- rand_forest(engine = "ranger") |>
set_mode("classification")
# workflow
rf_wf <- tree_wf |>
update_model(rf_mod)
# fit with cross-validation
set.seed(100)
rf_wf |>
fit_resamples(resamples = hotels_folds) |>
collect_metrics()# A tibble: 3 × 6
.metric .estimator mean n std_err .config
<chr> <chr> <dbl> <int> <dbl> <chr>
1 accuracy binary 0.829 10 0.00382 pre0_mod0_post0
2 brier_class binary 0.123 10 0.00176 pre0_mod0_post0
3 roc_auc binary 0.912 10 0.00319 pre0_mod0_post0Your Turn 3
Edit the random forest model to tune the mtry and min_n hyper-parameters; call the new model spec rf_tuner.
Update your workflow to use the tuned model.
Then use tune_grid() to find the best combination of hyper-parameters to maximize roc_auc; let tune set up the grid for you.
How does it compare to the average ROC AUC across folds from fit_resamples()?
rf_mod <- rand_forest(engine = "ranger") |>
set_mode("classification")
rf_wf <- workflow() |>
add_formula(children ~ .) |>
add_model(rf_mod)
set.seed(100) # Important!
rf_results <- rf_wf |>
fit_resamples(
resamples = hotels_folds,
metrics = metric_set(roc_auc),
# change me to control_grid() with tune_grid
control = control_resamples(save_workflow = TRUE)
)
rf_results |>
collect_metrics()# A tibble: 1 × 6
.metric .estimator mean n std_err .config
<chr> <chr> <dbl> <int> <dbl> <chr>
1 roc_auc binary 0.912 10 0.00319 pre0_mod0_post0Answer:
rf_tuner <- rand_forest(
engine = "ranger",
mtry = tune(),
min_n = tune()
) |>
set_mode("classification")
rf_wf <- rf_wf |>
update_model(rf_tuner)
set.seed(100) # Important!
rf_results <- rf_wf |>
tune_grid(
resamples = hotels_folds,
control = control_grid(save_workflow = TRUE)
)i Creating pre-processing data to finalize 1 unknown parameter: "mtry"Your Turn 4
Use fit_best() to take the best combination of hyper-parameters from rf_results and use them to predict the test set.
How does our actual test ROC AUC compare to our cross-validated estimate?
hotels_best <- fit_best(rf_results)
# cross validated ROC AUC
rf_results |>
show_best(metric = "roc_auc", n = 5)# A tibble: 5 × 8
mtry min_n .metric .estimator mean n std_err .config
<int> <int> <chr> <chr> <dbl> <int> <dbl> <chr>
1 5 2 roc_auc binary 0.912 10 0.00331 pre0_mod03_post0
2 7 18 roc_auc binary 0.911 10 0.00358 pre0_mod04_post0
3 3 31 roc_auc binary 0.908 10 0.00304 pre0_mod02_post0
4 12 6 roc_auc binary 0.908 10 0.00425 pre0_mod06_post0
5 9 35 roc_auc binary 0.907 10 0.00386 pre0_mod05_post0# test set ROC AUC
augment(hotels_best, new_data = hotels_test) |>
roc_auc(truth = children, .pred_children)# A tibble: 1 × 3
.metric .estimator .estimate
<chr> <chr> <dbl>
1 roc_auc binary 0.912# test set ROC curve
augment(hotels_best, new_data = hotels_test) |>
roc_curve(truth = children, .pred_children) |>
autoplot()
Acknowledgments
- Materials derived from Tidymodels, Virtually: An Introduction to Machine Learning with Tidymodels by Allison Hill.
- Dataset and some modeling steps derived from A predictive modeling case study and licensed under a Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA) License.
NoteSession information
sessioninfo::session_info()─ Session info ───────────────────────────────────────────────────────────────
setting value
version R version 4.5.2 (2025-10-31)
os macOS Tahoe 26.6
system aarch64, darwin20
ui X11
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz America/New_York
date 2026-08-04
pandoc 3.8.3 @ /Applications/Positron.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)
quarto 1.10.18 @ /Applications/quarto/bin/quarto
─ Packages ───────────────────────────────────────────────────────────────────
! package * version date (UTC) lib source
P cli 3.6.5 2025-04-23 [?] RSPM (R 4.5.0)
P digest 0.6.37 2024-08-19 [?] RSPM (R 4.5.0)
P evaluate 1.0.4 2025-06-18 [?] RSPM (R 4.5.1)
P fastmap 1.2.0 2024-05-15 [?] RSPM (R 4.5.0)
P here 1.0.1 2020-12-13 [?] RSPM (R 4.5.0)
P htmltools 0.5.8.1 2024-04-04 [?] RSPM (R 4.5.0)
P htmlwidgets 1.6.4 2023-12-06 [?] RSPM (R 4.5.0)
P jsonlite 2.0.0 2025-03-27 [?] RSPM (R 4.5.0)
P knitr 1.50 2025-03-16 [?] RSPM (R 4.5.0)
P renv 1.2.4 2026-08-03 [?] RSPM
P rlang 1.3.0 2026-07-05 [?] RSPM
P rmarkdown 2.29 2024-11-04 [?] RSPM
P rprojroot 2.1.0 2025-07-12 [?] RSPM (R 4.5.0)
P sessioninfo 1.2.3 2025-02-05 [?] RSPM (R 4.5.0)
P xfun 0.52 2025-04-02 [?] RSPM (R 4.5.1)
P yaml 2.3.10 2024-07-26 [?] RSPM (R 4.5.0)
[1] /Users/bcs88/Projects/info-5001/course-site/renv/library/macos/R-4.5/aarch64-apple-darwin20
[2] /Users/bcs88/Library/Caches/org.R-project.R/R/renv/sandbox/macos/R-4.5/aarch64-apple-darwin20/4cd76b74
P ── Loaded and on-disk path mismatch.
──────────────────────────────────────────────────────────────────────────────