Warning: package 'rvest' was built under R version 4.5.3
library(dplyr)
Warning: package 'dplyr' was built under R version 4.5.3
Attaching package: 'dplyr'
The following objects are masked from 'package:stats':
filter, lag
The following objects are masked from 'package:base':
intersect, setdiff, setequal, union
library(stringr)
Warning: package 'stringr' was built under R version 4.5.3
library(readr)
Warning: package 'readr' was built under R version 4.5.3
Attaching package: 'readr'
The following object is masked from 'package:rvest':
guess_encoding
## ---- 0. Ask permission first ----------------------------------------------## Always before the first request. robotstxt::paths_allowed() returns TRUE or## FALSE for the exact path you intend to fetch.robotstxt::paths_allowed("https://en.wikipedia.org/wiki/List_of_countries_by_foreign-exchange_reserves")
en.wikipedia.org
[1] TRUE
## ---- 1. The sandbox: books.toscrape.com ------------------------------------## Built and published for scraping practice. 1,000 books, 50 pages, no## JavaScript. Learn the verbs here before you point them at anyone's server.books_pg <-read_html("https://books.toscrape.com/catalogue/page-1.html")books <-tibble(title = books_pg |>html_elements("article.product_pod h3 a") |>html_attr("title"),price = books_pg |>html_elements("article.product_pod p.price_color") |>html_text2(),rating = books_pg |>html_elements("article.product_pod p.star-rating") |>html_attr("class") |>str_remove("star-rating ")) |>mutate(price_gbp =parse_number(price))head(books)
# A tibble: 6 × 4
title price rating price_gbp
<chr> <chr> <chr> <dbl>
1 A Light in the Attic £51.77 Three 51.8
2 Tipping the Velvet £53.74 One 53.7
3 Soumission £50.10 One 50.1
4 Sharp Objects £47.82 Four 47.8
5 Sapiens: A Brief History of Humankind £54.23 Five 54.2
6 The Requiem Red £22.65 One 22.7
library(knitr)library(kableExtra)
Warning: package 'kableExtra' was built under R version 4.5.2
Attaching package: 'kableExtra'
The following object is masked from 'package:dplyr':
group_rows
## ---- 2. The Wikipedia table -----------------------------------------------## html_table() does the parsing. The work is everything after it.wiki_url <-paste0("https://en.wikipedia.org/wiki/","List_of_countries_by_foreign-exchange_reserves")wiki_pg <-read_html(wiki_url)## Never assume the table you want is [[1]]. Look first.tabs <- wiki_pg |>html_elements("table.wikitable")length(tabs) # how many candidates?
## Clean-up. Wikipedia tables arrive with footnote markers, thin spaces,## multi-row headers and a stray total row. Expect to redo this when the## article is edited — which is the point of the exercise.new_names <-c('a_1', 'a_2', 'a_3', 'a_4', 'a_5', 'a_6', 'a_7', 'a_8')names(reserves_raw) <-make.unique(dput(names(reserves_raw)))
Warning: There were 3 warnings in `mutate()`.
The first warning was:
ℹ In argument: `across(...)`.
Caused by warning:
! 154 parsing failures.
row col expected actual
18 -- a number —.mw-parser-output .sr-only{border:0;clip:rect(0,0,0,0);clip-path:polygon(0px 0px,0px 0px,0px 0px);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px;white-space:nowrap}N/a
23 -- a number —N/a
25 -- a number —N/a
26 -- a number —N/a
30 -- a number —N/a
... ... ........ ......................................................................................................................................................................................................
See problems(...) for more details.
ℹ Run `dplyr::last_dplyr_warnings()` to see the 2 remaining warnings.
## ---- 3. The same quantity, from an API -------------------------------------## Foreign reserves are published by the World Bank as FI.RES.TOTL.CD,## "Total reserves (includes gold, current US$)". One call, versioned,## documented, and identical for whoever runs it.library(WDI)
Warning: package 'WDI' was built under R version 4.5.3
reserves_api <-WDI(indicator ="FI.RES.TOTL.CD",start =2015, end =2024, extra =TRUE)head(reserves_api)
country iso2c iso3c year FI.RES.TOTL.CD status lastupdated
1 Afghanistan AF AFG 2024 NA 2026-07-13
2 Afghanistan AF AFG 2023 NA 2026-07-13
3 Afghanistan AF AFG 2022 NA 2026-07-13
4 Afghanistan AF AFG 2021 NA 2026-07-13
5 Afghanistan AF AFG 2020 9748946327 2026-07-13
6 Afghanistan AF AFG 2019 8497655795 2026-07-13
region capital longitude latitude
1 Middle East, North Africa, Afghanistan & Pakistan Kabul 69.1761 34.5228
2 Middle East, North Africa, Afghanistan & Pakistan Kabul 69.1761 34.5228
3 Middle East, North Africa, Afghanistan & Pakistan Kabul 69.1761 34.5228
4 Middle East, North Africa, Afghanistan & Pakistan Kabul 69.1761 34.5228
5 Middle East, North Africa, Afghanistan & Pakistan Kabul 69.1761 34.5228
6 Middle East, North Africa, Afghanistan & Pakistan Kabul 69.1761 34.5228
income lending
1 Low income IDA
2 Low income IDA
3 Low income IDA
4 Low income IDA
5 Low income IDA
6 Low income IDA
## ---- 4. Cache what you pulled ----------------------------------------------write_csv(reserves, paste0("reserves_wiki_", Sys.Date(), ".csv"))
—- 5. The question this script exists to raise ————————— Compare 2 and 3. Same concept, two data-generating processes: - Wikipedia: edited by volunteers, sourced from many places, no schema, no versioning, no uncertainty, changes without notice - World Bank: one compiler, documented methodology, stable indicator code Scraping was the wrong tool here. Knowing when it is the only tool is the skill this course is actually teaching.
The time stamps on the World Bank data is probably the most imporant part of understanding the scraping, this gives a point of reference to check the data against. While Wikipedia is a decent source, to get those same timestamp if would likely require scraping (or using the api) of each bank being referenced in the table. As is stated, scraping is best when it is the only tool available. APIs or other data access points get around all of those messy data issues which come from even the most successful scraping attempts.
AI note Gemini was used for dealing with the tibble and visualizations.