```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
if (!require('tidyverse')) install.packages("tidyverse") # wrangling
if (!require('RSelenium')) install.packages("RSelenium") # we know it, we love it
if (!require('wdman')) install.packages("wdman") # selenium server config
if (!require('rvest')) install.packages("rvest") # retrieve HTML objects 
if (!require('httr2')) install.packages("httr2") # start a browser session
if (!require('httpcache')) install.packages("httpcache") # clears cache
if (!require('openxlsx')) install.packages("openxlsx") # excel creation and manipulation 
```

## Exercises
There are two options for you to work on during the lab session. 
  1. gather data from a website of your choice
	2. gather data on theses from https://cadmus.eui.eu/handle/1814/7088/recent-submissions

For the first exercise, just choose your own approach. Access information you need and try to create a loop for several links. 

For the second approach, we will go through the exercise step-by-step.

First, visit the webpage and parse the html, store it into an object called "html"
```{r}

```

Now, use the SelectorGadget or manually select the CSS selector to retrieve title, author, date, abstract, and department of all theses on the webpage.
```{r}

```


*Challenge*
If you look at the content of dept, you will see that there is some noise in the data. We just want the abbreviation of the Departments stored in the vector. Try to extract only this information by using regular expressions. 
```{r}

```


This gave you the first five results. Let's create a loop that goes through the first twenty results. 

What's the structure of the pagination? 
https://cadmus.eui.eu/handle/1814/7088/recent-submissions?offset=0, https://cadmus.eui.eu/handle/1814/7088/recent-submissions?offset=5, https://cadmus.eui.eu/handle/1814/7088/recent-submissions?offset=10, etc...

Create a loop to scrape the links to each thesis
```{r}

```


*Challenge*
Do the same in a function
```{r}

```


Now, navigate to each page, scrape the title, author name, the full abstract, the Department, type (thesis, defense, presentation etc.), defense date and the full-text link. Return a data frame. Remove content that is not a thesis.
```{r}

```


Great, you made it! 