# Descriptive statistics and data manipulation

Now that we are familiar with some R objects and know how to import data, it is time to write some
code. In this chapter, we are going to compute descriptive statistics for a single dataset, but
also for a list of datasets. However, I will not give a list of functions to compute descriptive
statistics; if you need a specific function you can find easily in the *Help* pane in Rstudio or
using any modern internet search engine. What I will do is show you a workflow that allows you to
compute the descripitive statisics you need fast.
R has a lot of built-in functions for descriptive statistics; however, if you want to compute
statistics by, say, gender, some more complex manipulations are needed. At least this was true in
the past. Nowadays, thanks to the packages from the `tidyverse`, it is very easy and fast to
compute descriptive statistics by any stratifying variable(s). The package we are going to use for
this is called `dplyr`. `dplyr` contains a lot of functions that make manipulating
data and computing descriptive statistics very easy. To make things easier for now, we are going to
use example data included with `dplyr`. So no need to import an external dataset; this does not
change anything to the example that we are going to study here; the source of the data does not
matter for this. Using `dplyr` is possible only if the data you are working with is already in
a useful shape. When data is more messy, you will need to first manipulate it to bring it a *tidy*
format. For this, we will use `tidyr`, which is very useful package to reshape data and to do
advanced cleaning of your data.
All these tidyverse functions are also called *verbs*. However, before getting to know these verbs,
let's do an analysis using standard, or *base* R functions. This will be the benchmark against
which we are going to measure a `{tidyverse}` workflow.

## A data exploration exercice using *base* R

Let's first load the `starwars` data set, included in the `{dplyr}` package:

```{r}
library(dplyr)
data(starwars)
```

Let's first take a look at the data:

```{r}
head(starwars)
```

This data contains information on Star Wars characters. The first question you have to answer is
to find the average height of the characters:

```{r}
mean(starwars$height)
```

Let's also take a look at the standard deviation:

```{r}
sd(starwars$height)
```

It might be more informative to compute these two statistics by species, so for this, we are going
to use `aggregate()`:

```{r}
aggregate(starwars$height,
          by = list(Species = starwars$species),
          mean)
```

Even if you are not familiar with `aggregate()`, I believe the above lines are quite self-explanatory.
You need to provide `aggregate()` with 3 things; the variable you want to summarize (or only the
data frame, if you want to summarize all variables), a list of grouping variables and then the
function that will be applied to each subset. You can easily add another grouping variable:

```{r}
aggregate(starwars$height,
          by = list(Species = starwars$species,
                    Homeworld = starwars$homeworld),
          mean)
```

or use another function:

```{r}
aggregate(starwars$height,
          by = list(Species = starwars$species),
          sd)
```

`aggregate()` returns a `data.frame` object:

```{r}
class(aggregate(starwars$height, by = list(Species = starwars$species), mean))
```

`tapply()` is another *base* R alternative:

```{r}
tapply(starwars$height, list(starwars$species), mean)
```

which returns an `array` object, which is similar to a vector.

However, `tapply()` does not work if you want the mean by species for all the variables in the
data frame:

```{r, eval=FALSE}
tapply(starwars, list(starwars$species), mean)
```

```
Error in tapply(starwars, list(starwars$species), mean) :
  arguments must have same length
```

In both cases, you can only specify one function. So if you need the average and the standard
deviation you have to do it in two steps.

Let's continue now, by only computing the average height by species, but for males:

```{r}
starwars_males <- subset(starwars, gender == "male")

aggregate(starwars_males$height,
          by = list(Species = starwars_males$species),
          mean)
```

I first use `subset()` to create a subset of the data in which I only kept males. Then, I rerun
the analysis from before again. `subset()` can also be used to select columns. So if you want
the average of the height and mass for males, you could do something like this:

```{r}
starwars_males_height_mass <- subset(starwars, gender == "male", select = c(height, mass, species))

aggregate(starwars_males_height_mass,
          by = list(Species = starwars_males_height_mass$species),
          mean)
```

This is starting to get a bit verbose, but it is quite easy to follow and very powerful. It certainly
beats having to write loops to achieve the same thing.

Let's now consider this new dataset:

```{r, include=FALSE}
survey_data_base <- as.data.frame(
    tibble::tribble(
        ~id, ~var1, ~var2, ~var3,
        1, 1, 0.2, 0.3,
        2, 1.4, 1.9, 4.1,
        3, 0.1, 2.8, 8.9,
        4, 1.7, 1.9, 7.6
        )
)
```

```{r}
survey_data_base
```

I will explain later where this comes from. Depending on what you want to do with this data, it is
not in the right shape. So let's reshape it, using the aptly-called `reshape()` command:

```{r}
survey_data_long <- reshape(survey_data_base,
        varying = list(2:4), v.names = "variable", direction = "long")
```

We can now easily compute the average of `variable` for each `id`:

```{r}
aggregate(survey_data_long$variable,
          by = list(Id = survey_data_long$id),
          mean)
```

There is also the possiblity to merge two datasets with `merge()`. I won't go into that however.
As you can see, R comes with very powerful functions right out of the box, ready to use. When I was
studying, unfortunately, my professors had been brought up on FORTRAN loops, so we had to do to all
this using loops (not reshaping, thankfully), which was not so easy.
Now that we have seen how *base* R works, let's redo the analysis using `{tidyverse}` verbs.
But before deep diving into the `{tidyverse}`, let's take a moment to discuss about our lord and
saviour, `%>%`.

## Smoking is bad for you, but pipes are your friend

The title of this section might sound weird at first, but by the end of it, you'll get this
(terrible) pun.

You probably know the following painting by René Magritte, *La trahison des images*:

```{r, echo=FALSE}
knitr::include_graphics("assets/pas_une_pipe.png")
```

It turns out there's an R package from the `tidyverse` that is called `magrittr`. What does this
package do? It brings *pipes* to R. Pipes are a concept from the Unix operating system; if you're
using a GNU+Linux distribution or macOS, you're basically using a *modern* unix (that's an
oversimplification, but I'm an economist by training, and outrageously oversimplifying things is
what we do, deal with it).

The idea of pipes is to take the output of a command, and *feed* it as the input of another
command. The `magrittr` package brings pipes to R, by using the weird looking `%>%`. Try the
following:

```{r, include = FALSE}
library(magrittr)
```

```{r, eval = FALSE}
library(magrittr)
```

```{r}
16 %>% sqrt
```

This looks quite weird, but you probably understand what happened; `16` got *fed* as the first
argument of the function `sqrt()`. You can chain multiple functions:

```{r}
16 %>% sqrt %>% `+`(18)
```

The output of `16` (`16`) got fed to `sqrt()`, and the output of `sqrt(16)` (4) got fed to `+(18)`
(22). Without `%>%` you'd write the line just above like this:

```{r}
sqrt(16) + 18
```

It might not be very clear right now why this is useful, but the `%>%` is probably one of the
most useful infix operators, because when using packages from the `tidyverse`, you will
naturally want to chain a lot of functions together. Without the `%>%` it would become messy very fast.

`%>%` is not the only pipe operator in `magrittr`. There's `%T%`, `%<>%` and `%$%`. All have their
uses, but are basically shortcuts to some common tasks with `%>%` plus another function. Which
means that you can live without them, and because of this, I will not discuss them.

## The `{tidyverse}`'s *enfant prodige*: `{dplyr}`

The best way to get started with the tidyverse packages is to get to know `{dplyr}`. `{dplyr}` provides
a lot of very useful functions that makes it very easy to get discriptive statistics or add new columns
to your data.

### A first taste of data manipulation with `{dplyr}`

This section will walk you through a typical analysis using `{dplyr}` funcitons. Just go with it; I
will give more details in the next sections.

First, let's load `dplyr` and the included `starwars` dataset. Let's also take a look at the first 5
lines of the dataset:

```{r}
library(dplyr)

data(starwars)

head(starwars)
```

`data(starwars)` loads the example dataset called `starwars` that is included in the package `dplyr`.
As I said earlier, this is just an example; you could have loaded an external dataset, from a
`.csv` file for instance. This does not matter for what comes next.

R includes a lot of functions for descriptive statistics, such as `mean()`, `sd()`, `cov()`, and many
more. What `dplyr` brings to the table (among other niceties) is the possibility to apply these
functions to the dataset easily. For example, imagine you want the average height of everyone in
the dataset. Using the basic R functions, you could write this:

```{r}
mean(starwars$height)
```

`starwars$height` means that the user wants to access the column called `height` from the dataset
`starwars`. Remember that the `$` symbol is how you access elements of a named list. This is the
same for columns of datasets as you can see. This is then given as an argument to the function
`mean()`. But what if the user wants the average height by species? Before `dplyr`, a solution to
this simple problem would have required more than a single command. Now this is as easy as:

```{r}
starwars %>%
  group_by(species) %>%
  summarise(mean(height))
```

The usefulness of the `%>%` (pipe operator) becomes apparent now. Without it, one would write
instead:

```{r}
summarise(group_by(starwars, species), mean(height))
```

as you can clearly see, it is much more difficult to read. Imagine now that I want the average height
by species, but only for males. Again, this is very easy using `%>%`:

```{r}
starwars %>%
  filter(gender == "male") %>%
  group_by(species) %>%
  summarise(mean(height))
```

Again, the `%>%` makes the above lines of code very easy to read. Without it, one would need to write:

```{r}
summarise(group_by(filter(starwars, gender == "male"), species), mean(height))
```

I think you agree with me that this is not very readable. One way to make it more readable would
be to save intermediary variables:

```{r}
filtered_data <- filter(starwars, gender == "male")

grouped_data <- group_by(filter(starwars, gender == "male"), species)

summarise(grouped_data, mean(height))
```

But this can get very tedious. Once you're used to `%>%`, you won't go back to not use it.

Before continuing and to make things clearer; `filter()`, `group_by()` and `summarise()` are
functions that are included in `dplyr`. `%>%` is actually a function from `magrittr`, but this
package gets loaded on the fly when you load `dplyr`, so you do not need to worry about it.
`mean()` is a function *native* to R.

The result of all these operations that use `dplyr` functions are actually other datasets, or
`tibbles`. This means that you can save them in variable, and then work with these as any other
datasets.

```{r}
mean_height <- starwars %>%
  group_by(species) %>%
  summarise(mean(height))

class(mean_height)

head(mean_height)
```

You could then write this data to disk using `rio::export()` for instance. If you need more than the
mean of the height, you can keep adding as many functions as needed:

```{r}
summary_table <- starwars %>%
  group_by(species) %>%
  summarise(ave_height = mean(height), var_height = var(height), n_obs = n())

print(summary_table)
```

I've added more functions, namely `var()`, to get the variance of height, and `n()`, which
is a function from `dplyr`, not base R, to get the number of observations. This is quite useful,
because we see that for a lot of species we only have one single individual! Let's focus on the
species for which we have more than 1 individual. Since we save all the previous operations (which
produce a `tibble`) in a variable, we can keep going from there:

```{r}
summary_table2 <- summary_table %>%
  filter(n_obs > 1)

print(summary_table2)
```

There's a lot of `NA`s; this is because by default, `mean()` and `var()` return `NA` if even one
single observation is `NA`. This is good, because it forces you to look at the data
to see what is going on. If you would get a number, even if there were `NA`s you could very easily
miss these missing values. It is better for functions to fail early and often than the opposite.
`mean()` and `var()` have a `na.rm` option that the user can set to `TRUE` to get the result by
ignoring the `NA`s:

```{r}
starwars %>%
  group_by(species) %>%
  summarise(ave_height = mean(height, na.rm = TRUE), var_height = var(height, na.rm = TRUE), n_obs = n()) %>%
  filter(n_obs > 1)
```

In the code above, I have combined the two previous steps to get the result I'm interested in. There's
a line in the final output that says `NA` for the species. Let's go back to the raw data and find
these lines:

```{r}
starwars %>%
  filter(is.na(species))
```

To test for `NA`, one uses the function `is.na()` not something like `species == "NA"` or anything
like that. `!is.na()` does the opposite:

```{r}
starwars %>%
  filter(!is.na(species))
```

The `!` function negates a predicate function (a predicate function is a function that returns
`TRUE` or `FALSE`). We can then rerun our analysis from before:

```{r}
starwars %>%
  filter(!is.na(species)) %>%
  group_by(species) %>%
  summarise(ave_height = mean(height, na.rm = TRUE), var_height = var(height, na.rm = TRUE), n_obs = n()) %>%
  filter(n_obs > 1)
```

And why not compute the same table, but first add another stratifying variable?

```{r}
starwars %>%
  filter(!is.na(species)) %>%
  group_by(species, gender) %>%
  summarise(ave_height = mean(height, na.rm = TRUE), var_height = var(height, na.rm = TRUE), n_obs = n()) %>%
  filter(n_obs > 1)
```

Ok, that's it for a first taste. We have already discovered some very useful `{dplyr}` functions,
`filter()`, `group_by()` and summarise `summarise()`.

Now, we are going to learn more about these functions in more detail.

### Filter the rows of a dataset with `filter()`

We're going to use the `Gasoline` dataset from the `plm` package, so install that first:

```{r, eval = FALSE}
install.packages("plm")
```

Then load the required data:

```{r}
data(Gasoline, package = "plm")
```

and load dplyr:

```{r}
library(dplyr)
```

This dataset gives the consumption of gasoline for 18 countries from 1960 to 1978. When you load
the data like this, it is a standard `data.frame`. `dplyr` functions can be used on standard
`data.frame` objects, but also on `tibble`s. `tibble`s are just like data frame, but with a better
print method (and other niceties). I'll discuss the `{tibble}` package later, but for now, let's
convert the data to a `tibble` and change its name:

```{r}
gasoline <- as_tibble(Gasoline)
```

`filter()` is pretty straightforward. What if you would like to subset the data to focus on the
year 1969? Simple:

```{r}
filter(gasoline, year == 1969)
```

Let's use `%>%`, since we're familiar with it now:

```{r}
gasoline %>% filter(year == 1969)
```

You can also filter more than just one year, by using the `%in%` operator:

```{r}
gasoline %>% filter(year %in% seq(1969, 1973))
```

It is also possible use `between()`, a helper function:

```{r}
gasoline %>% filter(between(year, 1969, 1973))
```

To select non-consecutive years:

```{r}
gasoline %>% filter(year %in% c(1969, 1973, 1977))
```

`%in%` tests if an object is part of a set.


### Select columns with `select()`

While `filter()` allows you to keep or discard rows of data, `select()`
allows you to keep or discard entire columns. To keep columns:

```{r}
gasoline %>% select(country, year, lrpmg)
```

To discard them:

```{r}
gasoline %>% select(-country, -year, -lrpmg)
```

To rename them:

```{r}
gasoline %>% select(country, date = year, lrpmg)
```

There's also `rename()`:

```{r}
gasoline %>% rename(date = year)
```

`rename()` does not do any kind of selection, but just renames.

You can also use `select()` to re-order columns:

```{r}
gasoline %>% select(year, country, lrpmg, everything())
```

`everything()` is a helper function, and there's also `starts_with()`,
and `ends_with()`. For example, what if we are only interested
in columns whose name start with "l"?

```{r}
gasoline %>% select(starts_with("l"))
```

`ends_with()` works in a similar fashion. There is also `contains()`:

```{r}
gasoline %>% select(country, year, contains("car"))
```

Another verb, similar to `select()`, is `pull()`. Let's compare the two:

```{r}
gasoline %>% select(lrpmg)
```

```{r}
gasoline %>% pull(lrpmg)
```

`pull()`, unlike `select()`, does not return a `tibble`, but only the column you want.

### Group the observations of your dataset with `group_by()`

`group_by()` is a very useful verb; as the name implies, it allows you to create groups and then,
for example, compute descriptive statistics by groups. For example, let's group our data by
country:

```{r}
gasoline %>% group_by(country)
```

It looks like nothing much happened, but if you look at the second line of the output you can read
the following:

```{r}
## # Groups:   country [18]
```

this means that the data is grouped, and every computation you will do now will take these groups
into account. It is also possible to group by more than one variable:

```{r}
gasoline %>% group_by(country, year)
```

and so on. You can then also ungroup:

```{r}
gasoline %>% group_by(country, year) %>% ungroup()
```

Once your data is grouped, the operations that will follow will be executed inside each group.

### Get summary statistics with `summarise()`

Ok, now that we have learned the basic verbs, we can start to do more interesting stuff. For
example, one might want to compute the average gasoline consumption in each country, for
the whole period:

```{r}
gasoline %>%
  group_by(country) %>%
  summarise(mean(lgaspcar))
```

`mean()` was given as an argument to `summarise()`, which is a `dplyr` verb. What we get is another
tibble, that contains the variable we used to group, as well as the average per country. We can
also rename this column:

```{r}
gasoline %>%
  group_by(country) %>%
  summarise(mean_gaspcar = mean(lgaspcar))
```

and because the output is a `tibble`, we can continue to use `dplyr` verbs on it:

```{r}
gasoline %>%
  group_by(country) %>%
  summarise(mean_gaspcar = mean(lgaspcar)) %>%
  filter(country == "france")
```

`summarise()` is a very useful verb. For example, we can compute several descriptive statistics at once:

```{r}
gasoline %>%
  group_by(country) %>%
  summarise(mean_gaspcar = mean(lgaspcar),
            sd_gaspcar = sd(lgaspcar),
            max_gaspcar = max(lgaspcar),
            min_gaspcar = min(lgaspcar))
```

Because the output is a `tibble`, you can save it in a variable of course:

```{r}
desc_gasoline <- gasoline %>%
  group_by(country) %>%
  summarise(mean_gaspcar = mean(lgaspcar),
            sd_gaspcar = sd(lgaspcar),
            max_gaspcar = max(lgaspcar),
            min_gaspcar = min(lgaspcar))
```

And then you can answer questions such as, *which country has the maximum average gasoline
consumption?*:

```{r}
desc_gasoline %>%
  filter(max(mean_gaspcar) == mean_gaspcar)
```

Turns out it's Turkey. What about the minimum consumption?

```{r}
desc_gasoline %>%
  filter(min(mean_gaspcar) == mean_gaspcar)
```

Because the output of `dplyr` verbs is a tibble, it is possible to continue working with it. This
is one shortcoming of using the base `summary()` function. The object returned by that function
is not very easy to manipulate.


### Adding columns with `mutate()` and `transmute()`

`mutate()` adds a column to the `tibble`, which can contain any transformation of any other
variable:

```{r}
gasoline %>%
  group_by(country) %>%
  mutate(n())
```

Using `mutate()` I've added a column that counts how many times the country appears in the `tibble`,
using `n()`, another `dplyr` function. There's also `count()` and `tally()`, which we are going to
see further down. It is also possible to rename the column on the fly:

```{r}
gasoline %>%
  group_by(country) %>%
  mutate(count = n())
```

It is possible to do any arbitrary operation:

```{r}
gasoline %>%
  group_by(country) %>%
  mutate(spam = exp(lgaspcar + lincomep))
```

`transmute()` is the same as `mutate()`, but only returns the created variable:

````{r}
gasoline %>%
  group_by(country) %>%
  transmute(spam = exp(lgaspcar + lincomep))
```

### Joining `tibble`s with `full_join()`, `left_join()`, `right_join()` and all the others

I will end this section on `dplyr` with the very useful verbs: the `*_join()` verbs. Let's first
start by loading another dataset from the `plm` package. `SumHes` and let's convert it to `tibble`
and rename it:

```{r}
data(SumHes, package = "plm")

pwt <- SumHes %>%
  as_tibble() %>%
  mutate(country = tolower(country))
```

Let's take a quick look at the data:

```{r}
glimpse(pwt)
```

We can merge both `gasoline` and `pwt` by country and year, as these two variables are common to
both datasets. There are more countries and years in the `pwt` dataset, so when merging both, and
depending on which function you use, you will either have `NA`'s for the variables where there is
no match, or rows that will be dropped. Let's start with `full_join`:

```{r}
gas_pwt_full <- gasoline %>%
  full_join(pwt, by = c("country", "year"))
```

Let's see which countries and years are included:

```{r}
gas_pwt_full %>%
  count(country, year)
```

As you see, every country and year was included, but what happened for, say, the U.S.S.R? This country
is in `pwt` but not in `gasoline` at all:

```{r}
gas_pwt_full %>%
  filter(country == "u.s.s.r.")
```

As you probably guessed, the variables from `gasoline` that are not included in `pwt` are filled with
`NA`s. One could remove all these lines and only keep countries for which these variables are not
`NA` everywhere with `filter()`, but there is a simpler solution:

```{r}
gas_pwt_inner <- gasoline %>%
  inner_join(pwt, by = c("country", "year"))
```
Let's use the `tabyl()` from the `janitor` packages which is a very nice alternative to the `table()`
function from base R:

```{r}
library(janitor)

gas_pwt_inner %>%
  tabyl(country)
```

Only countries with values in both datasets were returned. It's almost every country from `gasoline`,
apart from Germany (called "germany west" in `pwt` and "germany" in `gasoline`. I left it as is to
provide an example of a country not in `pwt`). Let's also look at the variables:

```{r}
glimpse(gas_pwt_inner)
```

The variables from both datasets are in the joined data.

Contrast this to `semi_join()`:


```{r}
gas_pwt_semi <- gasoline %>%
  semi_join(pwt, by = c("country", "year"))

glimpse(gas_pwt_semi)

gas_pwt_semi %>%
  tabyl(country)
```

Only columns of `gasoline` are returned, and only rows of `gasoline` that were matched with rows
from `pwt`. `semi_join()` is not a commutative operation:


```{r}
pwt_gas_semi <- pwt %>%
  semi_join(gasoline, by = c("country", "year"))

glimpse(pwt_gas_semi)

gas_pwt_semi %>%
  tabyl(country)
```

The rows are the same, but not the columns.

`left_join()` and `right_join()` return all the rows from either the dataset that is on the
"left" (the first argument of the fonction) or on the "right" (the second argument of the
function) but all columns from both datasets. So depending on which countries you're interested in,
you're going to use either one of these functions:

```{r}
gas_pwt_left <- gasoline %>%
  left_join(pwt, by = c("country", "year"))

gas_pwt_left %>%
  tabyl(country)
```

```{r}
gas_pwt_right <- gasoline %>%
  right_join(pwt, by = c("country", "year"))

gas_pwt_right %>%
  tabyl(country)
```

The last merge function is `anti_join()`:

```{r}
gas_pwt_anti <- gasoline %>%
  anti_join(pwt, by = c("country", "year"))

glimpse(gas_pwt_anti)

gas_pwt_anti %>%
  tabyl(country)
```

`gas_pwt_anti` has the columns the `gasoline` dataset as well as the only country from `gasoline`
that is not in `pwt`: "germany".

That was it for the basic `{dplyr}` verbs. Next, we're going to learn about `{tidyr}`.

## Reshaping data with `tidyr`

Another important package from the `tidyverse` that goes hand in hand with `dplyr` is `tidyr`. `tidyr`
is the package you need when it's time to reshape data. As of March 2019, the development version 
of `tidyr` introduced two new functions that make reshaping data easier, `pivot_long()` and `pivot_wide()`.
To install the development version of `tidyr`, use the following line:

```{r, eval=FALSE}
devtools::install_github("tidyverse/tidyr")
```

The legacy functions, `spread()` and `gather()` will remain in the package but their use will be 
discouraged.

I will start by presenting `pivot_wide()` and `pivot_long()`, and then, for reference, will show
how to solve the similar problems using `gather()` and `spread()`.

### `pivot_wide()` and `pivot_long()`

Let's first create a fake dataset:

```{r}
library(tidyr)
```

```{r}
survey_data <- tribble(
  ~id, ~variable, ~value,
  1, "var1", 1,
  1, "var2", 0.2,
  NA, "var3", 0.3,
  2, "var1", 1.4,
  2, "var2", 1.9,
  2, "var3", 4.1,
  3, "var1", 0.1,
  3, "var2", 2.8,
  3, "var3", 8.9,
  4, "var1", 1.7,
  NA, "var2", 1.9,
  4, "var3", 7.6
)

head(survey_data)
```

I used the `tribble()` function from the `{tibble}` package to create this fake dataset.
I'll discuss this package later, for now, let's focus on `{tidyr}.`

Let's suppose that we need the data to be in the wide format which means `var1`, `var2` and `var3`
need to be their own columns. To do this, we need to use the `pivot_wide()` function. Why *wide*?
Because the data set will be wide, meaning, having more columns than rows.

```{r}
survey_data %>% 
  pivot_wider(keys = id, names_from = variable, values_from = value)
```

Let's go through `pivot_wide()`'s arguments: the first is `keys = ` which requires the variable
that uniquely identifies the rows to be supplied. `names_from = ` is where you input the variable that will 
generate the names of the new columns. In our case, the `variable` colmuns has three values; `var1`,
`var2` and `var3`, and these are now the names of the new columns. Finally, `values_from = ` is where
you can specify the column containing the values that will fill the data frame.
I find the argument names `names_from = ` and `values_from = ` quite explicit. 

As you can see, there are some missing values. Let's suppose that we know that these missing values
are true 0's. `pivot_wide()` has an argument called `values_fill = ` that makes it easy to replace
the missing values:

```{r}
survey_data %>% 
  pivot_wider(keys = id, names_from = variable, values_from = value, values_fill = list(value = 0))
```

A list of variables and their respective values to replace NA's with must be supplied to `values_fill`. 

Let's now use another dataset, which you can get from
[here](https://github.com/b-rodrigues/modern_R/tree/master/datasets/unemployment/all) 
(downloaded from: http://www.statistiques.public.lu/stat/TableViewer/tableView.aspx?ReportId=12950&IF_Language=eng&MainTheme=2&FldrName=3&RFPath=91). This data set gives the unemployment rate for each Luxembourguish
canton from 2001 to 2015. We will come back to this data later on to learn how to plot it. For now, 
let's use it to learn more about `{tidyr}`.

```{r}
unemp_lux_data <- rio::import("datasets/unemployment/all/unemployment_lux_all.csv")

head(unemp_lux_data)
```

Now, let's suppose that for our purposes, it would make more sense to have the data in a wide format,
where columns are "divison times year" and the value is the unemployment rate. This can be easily done
with providing more columns to `names_from = `. 

```{r}
unemp_lux_data2 <- unemp_lux_data %>% 
  filter(year %in% seq(2013, 2017), str_detect(division, ".*ange$"), !str_detect(division, ".*Canton.*")) %>% 
  select(division, year, unemployment_rate_in_percent) %>% 
  rowid_to_column()

unemp_lux_data2 %>% 
  pivot_wide(names_from = c(division, year), values_from = unemployment_rate_in_percent)
```

In the `filter()` statement, I only kept data from 2013 to 2017, "division"s ending with the string "ange"
("division" can be a canton or a commune, for example "Canton Redange", a canton, or "Hesperange" a commune),
and removed the cantons as I'm only interested in communes. 
I then only kept the columns I'm interested in and pivoted the data to a wide format.
Also, I needed to add a unique identifier to the data frame. For this, I used `rowid_to_column()` function, 
from the `{tibble}` package, which adds a new column to the data frame with an id, going from 1 to
the number of rows in the data frame. If I did not add this identifier, the statement would work still:

```{r}
unemp_lux_data3 <- unemp_lux_data %>% 
  filter(year %in% seq(2013, 2017), str_detect(division, ".*ange$"), !str_detect(division, ".*Canton.*")) %>% 
  select(division, year, unemployment_rate_in_percent)

unemp_lux_data3 %>% 
  pivot_wider(names_from = c(division, year), values_from = unemployment_rate_in_percent)
```

and actually look even better, but only because there are no repeated values; there is only one
unemployment rate for each "commune times year". I will come back to this later on, with another
example that might be clearer.


You might have noticed that because there is no data for 
the years 2016 and 2017, these columns do not appear in the data. But suppose that we need to have 
these columns, so that a colleague from another department can fill in the values. This is possible
by providing a data frame with the detailed specifications of the result data frame. This optional 
data frame must have at least two columns, `.name`, which are the column names you want, and 
`.value` which contains the values.

```{r, include=FALSE}
unemp_spec <- unemp_lux_data %>% 
  tidyr::expand(division, year = c(year, 2016, 2017), .value = "unemployment_rate_in_percent") %>%
  unite(".name", division, year, remove = FALSE)

unemp_spec
```

```{r, eval=FALSE}
unemp_spec <- unemp_lux_data %>% 
  expand(division, year = c(year, 2016, 2017), .value = "unemployment_rate_in_percent") %>%
  unite(".name", division, year, remove = FALSE)

unemp_spec
```

The spec has more rows than the data frame (80 vs 48), so this will cause issues if you try to use
the specification directly:

```{r, eval=FALSE}
unemp_lux_data  %>% 
  pivot_wider(spec = unemp_spec)
```

```
Error in data.frame(row = row_id, col = col_id) : arguments imply differing number of rows: 80, 48
```

So to make it work, we first need to create a column that uniquely identifies each row:

```{r}
unemp_lux_data4 <- unemp_lux_data %>% 
  select(division, year, unemployment_rate_in_percent) %>% 
  rowid_to_column() %>% 
  pivot_wider(spec = unemp_spec) 

unemp_lux_data4
```

You can notice that now we have columns for 2016 and 2017 too. Let's clean the data a little bit more:

```{r}
unemp_lux_data4 %>% 
  select(-rowid) %>% 
  fill(matches(".*"), .direction = "down") %>% 
  slice(n())
```

We will learn about `fill()`, anoher `{tidyr}` function a bit later in this chapter, but its basic
purpose is to fill rows with whatever value comes before or after the missing values. `slice(n())`
then only keeps the last row of the data frame, which is the row that contains all the values (expect
for 2016 and 2017, which has missing values, as we wanted).

Here is another example of the importance of having an identifier column when using a spec:

```{r, include=FALSE}
data(mtcars)
mtcars_spec <- mtcars %>% 
    tidyr::expand(am, cyl, .value = "mpg") %>%
    unite(".name", am, cyl, remove = FALSE)

mtcars_spec
```

```{r, eval=FALSE}
data(mtcars)
mtcars_spec <- mtcars %>% 
    expand(am, cyl, .value = "mpg") %>%
    unite(".name", am, cyl, remove = FALSE)

mtcars_spec
```

We can now transform the data:

```{r}
mtcars %>% 
    pivot_wider(spec = mtcars_spec)
```

As you can see, there are several values of "mpg" for some combinations of "am" times "cyl". If 
we remove the other columns, each row will not be uniquely identified anymore, and thus an error will be 
returned:

```{r, eval=FALSE}
mtcars %>% 
  select(am, cyl, mpg) %>% 
  pivot_wide(spec = mtcars_spec)
```

```
Error in data.frame(row = row_id, col = col_id) : arguments imply differing number of rows: 6, 32
```

So you have to be careful with this.

`pivot_long()` is used when you need to go from a wide to a long dataset, meaning, a dataset where 
there are some columns that should not be columns, but rather, the levels of a factor variable. 
Let's suppose that the "am" column is split into two columns, `1` for 
automatic and `0` for manual transmissions, and that the values filling these colums are miles per
gallon, "mpg":

```{r}
mtcars_wide_am <- mtcars %>% 
  pivot_wider(names_from = am, values_from = mpg)

mtcars_wide_am %>% 
  select(`0`, `1`, everything())
```

As you can see, the "0" and "1" columns should not be their own columns, unless there is a very
specific and good reason they should... but rather, they should be the levels of another column (in
our case, "am").

We can go back to a long dataset like so:

```{r}
mtcars_wide_am %>% 
  pivot_longer(cols = c(`1`, `0`), names_to = "am", values_to = "mpg") %>% 
  select(am, mpg, everything())
```

In the cols argument, you need to list all the variables that need to be transformed. Only `1` and 
`0` must be pivoted, so I list them. Just for illustration purposes, imagine that we would need
to pivot 50 columns. It would be faster to list the columns that do not need to be pivoted. This 
can be achieved by listing the columns that must be excluded with `-` in front, and maybe using 
`match()` with a regular expression:

```{r}
mtcars_wide_am %>% 
  pivot_longer(cols = -matches("^[[:alpha:]]"), names_to = "am", values_to = "mpg") %>% 
  select(am, mpg, everything())
```

Every column that starts with a letter is ok, so there is no need to pivot them. I use the `match()` 
function with a regular expression so that I don't have to type the names of all the columns. `select()`
is used to re-order the columns, only for viewing purposes

`names_to = ` takes a string as argument, which will be the name of the name column containing the 
levels `0` and `1`, and `values_to = ` also takes a string as argument, which will be the name of 
the column containing the values. Finally, you can see that there are a lot of `NA`s in the 
output. These can be removed easily:

```{r}
mtcars_wide_am %>% 
  pivot_longer(cols = c(`1`, `0`), names_to = "am", values_to = "mpg", na.rm = TRUE) %>% 
  select(am, mpg, everything())
```

Now for a more advanced example, let's suppose that we are dealing with the following wide dataset:

```{r}
mtcars_wide <- mtcars %>% 
    pivot_wider(spec = mtcars_spec)

mtcars_wide
```

The difficulty here is that we have columns with two levels of information. For instance, the 
column "0_4" contains the miles per gallon values for manual cars (`0`) with `4` cylinders.
The first step is to first pivot the columns:

```{r}
mtcars_wide %>% 
  pivot_longer(cols = matches("0|1"), names_to = "am_cyl", values_to = "mpg", na.rm = TRUE) %>% 
  select(am_cyl, mpg, everything())
```

Now we only need to separate the "am_cyl" column into two new columns, "am" and "cyl":

```{r}
mtcars_wide %>% 
  pivot_longer(cols = matches("0|1"), names_to = "am_cyl", values_to = "mpg", na.rm = TRUE) %>% 
  separate(am_cyl, into = c("am", "cyl"), sep = "_") %>% 
  select(am, cyl, mpg, everything())
```

It is also possible to construct a specification data frame, just like for `pivot_wide()`:

```{r}
mtcars_spec_long <- mtcars_wide %>% 
  pivot_longer_spec(matches("0|1"), values_to = "mpg") %>% 
  separate(name, c("am", "cyl"), sep = "_")

mtcars_spec_long
```

This spec can now be specified to `pivot_long()`:

```{r}
mtcars_wide %>% 
  pivot_longer(spec = mtcars_spec_long, na.rm = TRUE) %>% 
  select(am, cyl, mpg, everything())
```

Defining specifications give a lot of flexibility and in some complicated cases are the way to go.

### `spread()` and `gather()`

`survey_data` is a long dataset. We can reshape it to be wide using the `spread()` function:

```{r}
wide_data <- survey_data %>%
  spread(variable, value)

head(wide_data)
```

This means that we spread the column called "variable", which will produce one column per category
of "variable". Then we fill in the rows with the data contained in the column "value".

To go from a wide dataset to a long one, we use `gather()`:

```{r}
long_data <- wide_data %>%
  gather(variable, value, var1, var2)

print(long_data)
```

`long_data` and `survey_data` are the same datasets, but in a different order.

In the `wide_data` `tibble`, we had 3 columns: `id`, `var1` and `var2`. We want to stack 'var1' and
'var2' in a new column, that we choose to call "variable". This is the "key". For the value, we are
using the values contained in `var1` and `var2`. Sometimes using `spread()` or `gather()` requires
some trial and error. I advise you play around with the examples above to really grasp how these
powerful functions work.


### `fill()` and `full_seq()`

`fill()` is pretty useful to... fill in missing values. For instance, in `survey_data`, some "id"s
are missing:

```{r}
survey_data
```

It seems pretty obvious that the first `NA` is supposed to be `1` and the second missing is supposed
to be `4`. With `fill()`, this is pretty easy to achieve:


```{r, include=FALSE}
survey_data %>%
    fill(.direction = "down", variable = "id")
```

```{r, eval=FALSE}
survey_data %>%
    fill(.direction = "down", variable = "id")
```

`full_seq()` is similar:

```{r}
full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1)
```

We can add this as the date column to our survey data:

```{r}
survey_data %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4))
```

I use the base `rep()` function to repeat the date 4 times and then using `mutate()` I have added
it the data frame.

Putting all these operations together:

```{r, include=FALSE}
survey_data %>%
    tidyr::fill(.direction = "down", variable = "id") %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4)) %>%
    spread(variable, value)
```


```{r, eval=FALSE}
survey_data %>%
    fill(.direction = "down", variable = "id") %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4)) %>%
    spread(variable, value)
```

As you can see, this creates a lot of explicit `NA` values. The best would be to fill in the missing
values and add the date column, and then work with that format. For example, to get the average
of the values by date:

```{r, include=FALSE}
survey_data %>%
    tidyr::fill(.direction = "down", variable = "id") %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4)) %>%
    group_by(date) %>%
    summarise(mean(value))
```


```{r, eval=FALSE}
survey_data %>%
    fill(.direction = "down", variable = "id") %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4)) %>%
    group_by(date) %>%
    summarise(mean(value))
```

Or by `date` and `variable`:


```{r, include=FALSE}
survey_data %>%
    tidyr::fill(.direction = "down", variable = "id") %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4)) %>%
    group_by(date, variable) %>%
    summarise(mean(value))
```

```{r, eval=FALSE}
survey_data %>%
    fill(.direction = "down", variable = "id") %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4)) %>%
    group_by(date, variable) %>%
    summarise(mean(value))
```

As you can see, you can chain any `{tidyverse}` verbs, wether they come from `{dplyr}` or `{tidyr}`.

### Put order in your columns with `separate()`, `unite()`, and in your rows with `separate_rows()`

```{r, include=FALSE}
survey_data_not_tidy <- survey_data %>%
    tidyr::fill(.direction = "down", variable = "id") %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4)) %>%
    mutate(variable_date = paste(variable, date, sep = "/")) %>% select(id, variable_date, value)
```

Sometimes, data can be in a format that makes working with it needlessly painful. For example, you
get this:

```{r}
survey_data_not_tidy
```

Dealing with this is simple, thanks to `separate()`:

```{r}
survey_data_not_tidy %>%
    separate(variable_date, into = c("variable", "date"), sep = "/")
```

The `variable_date` column gets separated into two columns, `variable` and `date`. One also needs
to specify the separator, in this case "/".

`unite()` is the reverse operation, which can be useful when you are confronted to this situation:

```{r, include=FALSE}
survey_data2 <- survey_data_not_tidy %>%
    separate(variable_date, into = c("variable", "date"), sep = "/") %>%
    separate(date, into = c("year", "month", "day"), sep = "-")
```

```{r}
survey_data2
```

In some situation, it is better to have the date as a single column:

```{r}
survey_data2 %>%
    unite(date, year, month, day, sep = "-")
```

Another awful situation is the following:

```{r, include=FALSE}
survey_data_from_hell <- data.frame(
  id = c(1, 1, NA, 2, 3, 3, 4, NA, 4),
  variable = c("var1", "var2", "var3", "var1, var2, var3", "var1, var2", "var3", "var1", "var2", "var3"),
  value = c("1", "0.2", "0.3", "1.4, 1.9, 4.1", "0.1, 2.8", "8.9", "1.7", "1.9", "7.6"),
  stringsAsFactors = FALSE
)
```

```{r}
survey_data_from_hell
```

`separate_rows()` saves the day:

```{r}
survey_data_from_hell %>%
    separate_rows(variable, value)
```

So to summarise... you can go from this:

```{r}
survey_data_from_hell
```

```{r, include=FALSE}
survey_data_clean <- survey_data2 %>%
    unite(date, year, month, day, sep = "-")
```

to this:

```{r}
survey_data_clean
```

quite easily:

```{r, include=FALSE}
survey_data_from_hell %>%
    separate_rows(variable, value, convert = TRUE) %>%
    tidyr::fill(.direction = "down", variable = "id") %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4))
```


```{r, eval=FALSE}
survey_data_from_hell %>%
    separate_rows(variable, value, convert = TRUE) %>%
    fill(.direction = "down", variable = "id") %>%
    mutate(date = rep(full_seq(c(as.Date("2018-08-01"), as.Date("2018-08-03")), 1), 4))
```


### A sneak peek to Chapter 8; `nest()`

Let's take a look at our clean survey data:

```{r}
survey_data_clean
```

`{tidyr}` has another very useful function, called `nest()`:

```{r}
survey_data_clean %>%
    nest(-id)
```

You can achieve the same result by using `group_by()` first:

```{r}
survey_data_clean %>%
    group_by(id) %>%
    nest()
```

This cerates a new tibble object, whith two columns, one with the `id` column, and a new one called
`data`. Let's take a look at the first element of this column:

```{r}
nested_survey_data <- survey_data_clean %>%
    group_by(id) %>%
    nest()

nested_survey_data$data[1]
```

As you can see, the first element of the `data` is also a tibble! You may wonder why this is useful;
I will give you a small taste of what's waiting in chapter 9. Imagine that you want to create a
barplot for each `id`. For instance, for the first `id` (we are going to learn about making plots
with `{ggplot2}` in the next chapter. For now, just follow along, even if you don't understand
everything I write):

```{r}
survey_data_id1 <- survey_data_clean %>%
    filter(id == 1)
```

Now, let's create the plot:

```{r}
ggplot(data = survey_data_id1) +
    geom_bar(aes(y = value, x = date, fill = variable), stat = "identity") +
    ggtitle("id 1")
```

Ok great. But now I want to create this plot for each `id`... so I have to copy paste this 3 times.
But copy-pasting is error prone. So there are two alternatives; I either write a function that
takes as argument the data and the `id` I want to plot and run it for each `id` (we will learn to
do this in Chapter 8), or I can use `tidyr::nest()`, combined with `purrr::map()`. `{purrr}` is
another very useful `{tidevyrse}` package, and we are going to learn about it in Chapter 9. Again,
just follow along for now:

```{r}
my_plots <- nested_survey_data %>%
    mutate(plot = map2(.x = id,
                       .y = data,
                       ~ggplot(data = .y) +
                           geom_bar(aes(y = value, x = date, fill = variable), stat = "identity") +
                           ggtitle(paste0("id", .x))))
```

This is some very advanced stuff, and again, do not worry if you don't understand everything now.
We are going to learn about this in detail in Chapter 9. Let's go through each line.
In the first line, I have started from my clean data `nested_survey_data` and then, using the `%>%`
and `mutate()` I create a new column called `plot`. Inside the `mutate()` function, I called `map2`.
`map2` is a `{purrr}` function that takes three inputs: `.x`, `.y` and a function. `.x` is the `id`
column from my data, and `.y` is the `data` column from `nested_survey_data`. The function is the
ggplot I created before. Think of `map2()` has a loop over two lists, all while applying a function.

The following illustration, taken from @vaudor_purrr_2018 by
[Lise Vaudor](https://twitter.com/LVaudor), illustrates this perfectly:

```{r, echo=FALSE}
knitr::include_graphics("assets/purrr3.png")
```

Two inputs go in at the same type, the factory, your function, does what it has to do, and an
output comes out. Forget about the factory's chimney, which represents `walk2()` for now. We'll
learn about it in due time. By the way, you really should read Lise's blog, her posts are really
great and informative. It's in French, but that's not a problem, you know how to read R code, right?
Here's a link to [her blog](http://perso.ens-lyon.fr/lise.vaudor/).

Now, let's take a look at the result:

```{r}
my_plots
```

`my_plots` is a tibble with three columns: `id`, `data` and.. `plot`! `plot` is a very interesting
column. It is a list, where each element is of type `S3: gg`. Yes, you guessed it, each element of
that list-column is a ggplot! If you now want to take a look at the plots, it is enough to do this:

```{r}
my_plots$plot
```

Ok, that was quite complicated, but again, this was only to introduce `nest()` and give you a taste
of the power of the `{tidyverse}`. By the end of Chapter 9, you'll be used to this.

That was it for a first introduction to `{tidyr}`. Now, it's time to learn about *scoped* verbs.

## Scoped `{tidyverse}` verbs

Scoped verbs are special versions of the verbs you are now familiar with.

### filter_*()

Let's go back to the `gasoline` data from the `{plm}` package.

`filter()` is not the only *filtering* verb there is. Suppose that we have a condition that we want
to use to filter out a lot of columns at once. For example, for every column that is of type
`numeric`, keep only the lines where the condition *value > -8* is satisfied. The next line does
that:

```{r}
gasoline %>% filter_if( ~all(is.numeric(.)), all_vars(. > -8))
```

This is a bit more complicated than before, but let's go through it, step by step.
`filter_if()` needs 3 arguments to work; the data, a
predicate function (a function that returns `TRUE`, or `FALSE`) which will select the columns we
want to work on, and then the condition. The condition can be applied to *all* the columns that
were selected by the predicate function (hence the `all_vars()`) or only to at least one (you'd use
`any_vars()` then). Try to change the condition, or the predicate function, to figure out how
`filter_if()` works. The dot is a placeholder that stands for whatever columns where selected.

Another scoped `filter()` verb is `filter_at()`; it allows the user to filter columns by position:

```{r}
gasoline %>% filter_at(vars(ends_with("p")), all_vars(. > -8))
```

we already know about `ends_with()` and `starts_with()`. So the above line means "for the columns
whose name end with a 'p' only keep the lines where, for all the selected columns, the values are
strictly superior to `-8`". Again, this is not very easy the first time you deal with that, so
play around with it for a bit.

Finally, there is also `filter_all()` which, as the name implies, uses all the variables for
the filtering step.

`filter_if()` and `filter_at()` are very useful when you have very large datasets with a lot of
variables and you want to apply a filtering function only to a subset of them. `filter_all()` is
useful if, for example, you only want to keep the positive values for all the columns.

### select_\*() and rename_\*()

Just as with `filter()`, `select()` also has scoped versions. `select_if()` makes it easy to
select columns that satisfy a criterium:

```{r}
gasoline %>% select_if(is.numeric)
```

You can even pass a further function to `select_if()` that will be applied to the selected columns:

```{r}
gasoline %>% select_if(is.numeric, toupper)
```

`select_at()` makes it easy to select all the variables that start with "l" for example:

```{r}
gasoline %>% select_at(vars(starts_with("l")))
```

`select_at()` also works if you specify the position of the columns you're interested in:

```{r}
gasoline %>% select_at(vars(c(1,2,5)))
```

Notice that I use a new helper function, `vars()`, which allows me to specify which variables I want
to select. This is similar to the helper functions we have seen before, `all_vars()`
and `any_vars()`. So the way to read the above line would be something like "Start with the
gasoline data, then select the variables at position 1, 2, 5". Knowing how to read your code makes
it even easier to write.

There is also a `select_all()`, and you might be wondering why it is useful:

```{r}
gasoline %>% select_all()
```

This simply returns all the columns of the data frame... but wait, `selec_all()` allows you to do
something very useful, which is rename all the columns in bulk:

```{r}
gasoline %>% select_all(toupper)
```

You can apply any arbitrary function:

```{r}
gasoline %>% select_all(funs(paste0("hello_", .)))
```

I have passed the `paste0()` function to the `funs()` helper function of `select_all()`. `funs()` is
used a lot with scoped verbs, just like `vars()`, which we have already seen.

The scoped version of `rename()` work just as you expect. But I'll leave these for the exercices
down below.

### group_by_*()

To illustrate `group_by_*()` I have to first modify the `gasoline` data a little bit. As you can
see below, the data column is of type integer:

```{r}
gasoline
```

Let's change that to character:

```{r}
gasoline <- gasoline %>%
    mutate(year = as.character(year))
```

Now, this allows me to do the following:

```{r}
gasoline %>%
    group_by_if(is.character) %>%
    summarise(mean(lincomep))
```

This is faster than having to write:

```{r}
gasoline %>%
    group_by(country, year) %>%
    summarise(mean(lincomep))
```

You may think that having two write the name of two variables is not a huge hassle, which is true.
But imagine that you have dozens of character columns that you want to group by. This is the kind
of situation where `group_by_if()` is very useful. If you prefer, you can specify the position of
the columns instead of the name of the columns, with `group_by_at()`:

```{r}
gasoline %>%
    group_by_at(vars(1, 2)) %>%
    summarise(mean(lincomep))
```

There is also a `group_by_all()`, but I fail to see the use case...

### summarise_()

Just like for `filter()`, `select()`, and `group_by()`, summarise()` comes with scoped versions:

```{r}
gasoline %>%
  group_by(country) %>%
  summarise_at(vars(starts_with("l")), mean)
```

See how I managed to summarise every variable in one simple call to `summarise_at()`? Simply by
using `vars()` and specifying that I was interested in the ones that started with "l" and then I
specified the function I wanted. But what if I wanted to use more than one function to summarise
the data? Very easy:

```{r}
gasoline %>%
  group_by(country) %>%
  summarise_at(vars(starts_with("l")), funs(mean, sd, max, min))
```

Just use `vars()` to specify the variables you want to summarise, and then `funs()` to list all the
functions you want to use on each of the columns.

But maybe you're just interested in descriptive statistics for some variables, but not all those
that start with "l"? What if you want to use another pattern? Easy to do with the `contains()`
helper:

```{r}
gasoline %>%
  group_by(country) %>%
  summarise_at(vars(dplyr::contains("car")), funs(mean, sd, max, min))
```

I used `dplyr::contains()` instead of simply `contains()` because there's also a
`purrr::contains()`. If you load `purrr` after `dplyr`, `contains()` will actually be
`purrr::contains()` and not `dplyr::contains()` which causes the above code to fail.

There's also `summarise_if()`:

```{r}
gasoline %>%
  group_by(country) %>%
  summarise_if(is.double, funs(mean, sd, min, max))
```

This allows you to summarise every column that contain real numbers. The difference between
`is.double()` and `is.numeric()` is that `is.numeric()` returns `TRUE` for integers too, whereas
`is.double()` returns `TRUE` for real numbers only (integers are real numbers too, but you know
what I mean). To go faster, you can also use `summarise_all()`:

```{r}
gasoline %>%
  select(-year) %>%
  group_by(country) %>%
  summarise_all(funs(mean, sd, min, max))
```

I removed the `year` variable because it's not a variable for which we want to have descriptive
statistics.

### mutate_*()

Of course, `mutate()` and `transmute()` also come with scoped versions:

```{r}
gasoline %>%
  mutate_if(is.double, exp)
```

```{r}
gasoline %>%
  mutate_at(vars(starts_with("l")), exp)
```

```{r}
gasoline %>%
  mutate_all(as.character)
```

I think that by now, you are able to understand the above lines quite easily. If not, try some other
functions, or mutating other variables and you'll see that it is not that complicated!

## Other useful `{tidyverse}` functions

```{r, include=FALSE}
gasoline %<>%
    mutate(year = as.numeric(year))
```

### `if_else()`, `case_when()` and `recode()`

Some other very useful `{tidyverse}` functions are `if_else()` and `case_when`. These two
functions, combined with `mutate()` make it easy to create a new variable whose values must
respect certain conditions. For instance, we might want to have a dummy that equals `1` if a country
in the European Union (to simplify, say as of 2017) and `0` if not. First let's create a list of
countries that are in the EU:

```{r}
eu_countries <- c("austria", "belgium", "bulgaria", "croatia", "republic of cyprus",
                  "czech republic", "denmark", "estonia", "finland", "france", "germany",
                  "greece", "hungary", "ireland", "italy", "latvia", "lithuania", "luxembourg",
                  "malta", "netherla", "poland", "portugal", "romania", "slovakia", "slovenia",
                  "spain", "sweden", "u.k.")
```

I've had to change "netherlands" to "netherla" because that's how the country is called in the
`gasoline` data. Now let's create a dummy variable that equals `1` for EU countries, and `0` for the others:

```{r}
gasoline %>%
  mutate(country = tolower(country)) %>%
  mutate(in_eu = if_else(country %in% eu_countries, 1, 0))
```

Instead of `1` and `0`, we can of course use strings (I add `filter(year == 1960)` at the end to
have a better view of what happened):

```{r}
gasoline %>%
  mutate(country = tolower(country)) %>%
  mutate(in_eu = if_else(country %in% eu_countries, "yes", "no")) %>%
  filter(year == 1960)
```

I think that `if_else()` is fairly straightforward, especially if you know `ifelse()` already. You
might be wondering what is the difference between these two. `if_else()` is stricter than
`ifelse()` and does not do type conversion. Compare the two next lines:

```{r}
ifelse(1 == 1, "0", 1)
```

```{r, eval = FALSE}
if_else(1 == 1, "0", 1)
```

```{r, eval = FALSE}
Error: `false` must be type string, not double
```

Type conversion, especially without a warning is very dangerous. `if_else()`'s behaviour which
consists in failing as soon as possble avoids a lot of pain and suffering, especially when
programming non-interactively.

`if_else()` also accepts an optional argument, that allows you to specify what should be returned
in case of `NA`:

```{r}
if_else(1 <= NA, 0, 1, 999)

# Or
if_else(1 <= NA, 0, 1, NA_real_)
```

`case_when()` can be seen as a generalization of `if_else()`. Whenever you want to use multiple
`if_else()`s, that's when you know you should use `case_when()` (I'm adding the filter at the end
for the same reason as before, to see the output better):

```{r}
gasoline %>%
  mutate(country = tolower(country)) %>%
  mutate(region = case_when(
           country %in% c("france", "italy", "turkey", "greece", "spain") ~ "mediterranean",
           country %in% c("germany", "austria", "switzerl", "belgium", "netherla") ~ "central europe",
           country %in% c("canada", "u.s.a.", "u.k.", "ireland") ~ "anglosphere",
           country %in% c("denmark", "norway", "sweden") ~ "nordic",
           country %in% c("japan") ~ "asia")) %>%
  filter(year == 1960)
```

If all you want is to recode values, you can use `recode()`. For example, the Netherlands is
written as "NETHERLA" in the `gasoline` data, which is quite ugly. Same for Switzerland:

```{r}
gasoline <- gasoline %>%
  mutate(country = tolower(country)) %>%
  mutate(country = recode(country, "netherla" = "netherlands", "switzerl" = "switzerland"))
```

I saved the data with these changes as they will become useful in the future. Let's take a look at
the data:

```{r}
gasoline %>%
  filter(country %in% c("netherlands", "switzerland"), year == 1960)
```

### `lead()` and `lag()`

`lead()` and `lag()` are especially useful in econometrics. When I was doing my masters, in 4 B.d.
(*Before dplyr*) lagging variables in panel data was quite tricky. Now, with `dplyr` it's really
very easy:

```{r}
gasoline %>%
  group_by(country) %>%
  mutate(lag_lgaspcar = lag(lgaspcar)) %>%
  mutate(lead_lgaspcar = lead(lgaspcar)) %>%
  filter(year %in% seq(1960, 1963))
```

To lag every variable, remember that you can use `mutate_if()`:

```{r}
gasoline %>%
  group_by(country) %>%
  mutate_if(is.double, lag) %>%
  filter(year %in% seq(1960, 1963))
```

you can replace `lag()` with `lead()`, but just keep in mind that the columns get transformed in
place.

### `ntile()`

The last helper function I will discuss is `ntile()`. There are some other, so do read `mutate()`'s
documentation with `help(mutate)`!

If you need quantiles, you need `ntile()`. Let's see how it works:

```{r}
gasoline %>%
  mutate(quintile = ntile(lgaspcar, 5)) %>%
  mutate(decile = ntile(lgaspcar, 10)) %>%
  select(country, year, lgaspcar, quintile, decile)
```

`quintile` and `decile` do not hold the values but the quantile the value lies in. If you want to
have a column that contains the median for instance, you can use good ol' `quantile()`:

```{r}
gasoline %>%
  group_by(country) %>%
  mutate(median = quantile(lgaspcar, 0.5)) %>% # quantile(x, 0.5) is equivalent to median(x)
  filter(year == 1960) %>%
  select(country, year, median)
```

### `arrange()`

`arrange()` re-orders the whole `tibble` according to values of the supplied variable:

```{r}
gasoline %>%
  arrange(lgaspcar)
```

If you want to re-order the `tibble` in descending order of the variable:

```{r}
gasoline %>%
  arrange(desc(lgaspcar))
```

`arrange`'s documentation alerts the user that re-ording by group is only possible by explicitely
specifying an option:

```{r}
gasoline %>%
  filter(year %in% seq(1960, 1963)) %>%
  group_by(country) %>%
  arrange(desc(lgaspcar), .by_group = TRUE)
```

This is especially useful for plotting. We'll see this in Chapter 6.

### `tally()` and `count()`

`tally()` and `count()` count the number of observations in your data. I believe `count()` is the
more useful of the two, as it counts the number of observations within a group that you can provide:

```{r}
gasoline %>%
  count(country)
```
There's also `add_count()` which adds the column to the data:

```{r}
gasoline %>%
  add_count(country)
```

`add_count()` is a shortcut for the following code:

```{r}
gasoline %>%
  group_by(country) %>%
  mutate(n = n())
```

where `n()` is a `dplyr` function that can only be used within `summarise()`, `mutate()` and
`filter()`.


## Special packages for special kinds of data: `{forcats}`, `{lubridate}`, and `{stringr}`

### 🐈🐈🐈🐈

Factor variables are very useful but not very easy to manipulate. `forcats` contains very useful
functions that make working on factor variables painless. In my opinion, the four following functions, `fct_recode()`, `fct_relevel()`, `fct_reorder()` and `fct_relabel()`, are the ones you must
know, so that's what I'll be showing.

Remember in chapter 3 when I very quickly explained what were `factor` variables? In this section,
we are going to work a little bit with these type of variable. `factor`s are very useful, and the
`forcats` package includes some handy functions to work with them. First, let's load the `forcats` package:

```{r}
library(forcats)
```

as an example, we are going to work with the `gss_cat` dataset that is included in `forcats`. Let's
load the data:

```{r}
data(gss_cat)

head(gss_cat)
```

as you can see, `marital`, `race`, `rincome` and `partyid` are all factor variables. Let's take a closer
look at `marital`:

```{r}
str(gss_cat$marital)
```

and let's see `rincome`:

```{r}
str(gss_cat$rincome)
```

`factor` variables have different levels and the `forcats` package includes functions that allow
you to recode, collapse and do all sorts of things on these levels. For example , using
`forcats::fct_recode()` you can recode levels:

```{r}
gss_cat <- gss_cat %>%
  mutate(marital = fct_recode(marital,
                              refuse = "No answer",
                              never_married = "Never married",
                              divorced = "Separated",
                              divorced = "Divorced",
                              widowed = "Widowed",
                              married = "Married"))

gss_cat %>%
  tabyl(marital)
```

Using `fct_recode()`, I was able to recode the levels and collapse `Separated` and `Divorced` to
a single category called `divorced`. As you can see, `refuse` and `widowed` are less than 10%, so
maybe you'd want to lump these categories together:

```{r}
gss_cat <- gss_cat %>%
  mutate(marital = fct_lump(marital, prop = 0.10, other_level = "other"))

gss_cat %>%
  tabyl(marital)
```

`fct_reorder()` is especially useful for plotting. We will explore plotting in the next chapter,
but to show you why `fct_reorder()` is so useful, I will create a barplot, first without
using `fct_reorder()` to re-order the factors, then with reordering. Do not worry if you don't
understand all the code for now:

```{r}
gss_cat %>%
    tabyl(marital) %>%
    ggplot() +
    geom_col(aes(y = n, x = marital)) +
    coord_flip()
```

It would be much better if the categories were ordered by frequency. This is easy to do with
`fct_reorder()`:

```{r}
gss_cat %>%
    tabyl(marital) %>%
    mutate(marital = fct_reorder(marital, n, .desc = FALSE)) %>%
    ggplot() +
    geom_col(aes(y = n, x = marital)) +
    coord_flip()
```

Much better! In Chapter 6, we are going to learn about `{ggplot2}`.

`{forcats}` contains other very useful functions, so I urge you to go through the documentation.

### Get your dates right with `{lubridate}`

`{lubridate}` is yet another tidyverse package, that makes dealing with dates or durations (and intervals) as
painless as possible. I do not use every function contained in the package daily, and as such will
only focus on some of the functions. However, if you have to deal with dates often,
you might want to explore the package thouroughly.

#### Defining dates, the tidy way

```{r, eval=FALSE, echo=FALSE}
page <- read_html("https://en.wikipedia.org/wiki/Decolonisation_of_Africa")

independence <- page %>%
    html_node(".wikitable") %>%
    html_table(fill = TRUE)

independence <- independence %>%
    select(-Rank) %>%
    map_df(~str_remove_all(., "\\[.*\\]")) %>%
    rename(country = `Country[a]`,
           colonial_name = `Colonial name`,
           colonial_power = `Colonial power[b]`,
           independence_date = `Independence date[c]`,
           first_head_of_state = `First head of state[d]`,
           independence_won_through = `Independence won through`)

saveRDS(independence, "independence.rds")
```

Let's load new dataset, called *independence* from the datasets folder:

```{r}
independence <- readRDS("datasets/independence.rds")
```

This dataset was scraped from the following Wikipedia [page](https://en.wikipedia.org/wiki/Decolonisation_of_Africa#Timeline).
It shows when African countries gained independence from which colonial powers. In Chapter 11, I
will show you how to scrape Wikipedia pages using R. For now, let's take a look at the contents
of the dataset:

```{r}
independence
```

as you can see, the date of independence is in a format that might make it difficult to answer questions
such as *Which African countries gained independence before 1960 ?* for two reasons. First of all,
the date uses the name of the month instead of the number of the month, and second of all the type of
the independence day column is *character* and not "date". So our first task is to correctly define the column
as being of type date, while making sure that R understands that *January* is supposed to be "01", and so
on. There are several helpful functions included in `{lubridate}` to convert columns to dates. For instance
if the column you want to convert is of the form "2012-11-21", then you would use the function `ymd()`,
for "year-month-day". If, however the column is "2012-21-11", then you would use `ydm()`. There's
a few of these helper functions, and they can handle a lot of different formats for dates. In our case,
having the name of the month instead of the number might seem quite problematic, but it turns out
that this is a case that `{lubridate}` handles painfully:

```{r}
library(lubridate)

independence <- independence %>%
  mutate(independence_date = dmy(independence_date))
```

Some dates failed to parse, for instance for Morocco. This is because these countries have several
independence dates; this means that the string to convert looks like:

```
"2 March 1956
7 April 1956
10 April 1958
4 January 1969"
```

which obviously cannot be converted by `{lubridate}` without further manipulation. I ignore these cases for
simplicity's sake.

#### Data manipulation with dates

Let's take a look at the data now:

```{r}
independence
```

As you can see, we now have a date column in the right format. We can now answer questions such as
*Which countries gained independence before 1960?* quite easily, by using the functions `year()`,
`month()` and `day()`. Let's see which countries gained independence before 1960:

```{r}
independence %>%
  filter(year(independence_date) <= 1960) %>%
  pull(country)
```

You guessed it, `year()` extracts the year of the date column and converts it as a *numeric* so that we can work
on it. This is the same for `month()` or `day()`. Let's try to see if countries gained their independence on
Christmas Eve:

```{r}
independence %>%
  filter(month(independence_date) == 12,
         day(independence_date) == 24) %>%
  pull(country)
```

Seems like Libya was the only one! You can also operate on dates. For instance, let's compute the difference between
two dates, using the `interval()` column:

```{r}
independence %>%
  mutate(today = lubridate::today()) %>%
  mutate(independent_since = interval(independence_date, today)) %>%
  select(country, independent_since)
```

The `independent_since` column now contains an *interval* object that we can convert to years:

```{r}
independence %>%
  mutate(today = lubridate::today()) %>%
  mutate(independent_since = interval(independence_date, today)) %>%
  select(country, independent_since) %>%
  mutate(years_independent = as.numeric(independent_since, "years"))
```

We can now see for how long the last country to gain independence has been independent.
Because the data is not tidy (in some cases, an African country was colonized by two powers,
see Libya), I will only focus on 4 European colonial powers: Belgium, France, Portugal and the United Kingdom:

```{r}
independence %>%
  filter(colonial_power %in% c("Belgium", "France", "Portugal", "United Kingdom")) %>%
  mutate(today = lubridate::today()) %>%
  mutate(independent_since = interval(independence_date, today)) %>%
  mutate(years_independent = as.numeric(independent_since, "years")) %>%
  group_by(colonial_power) %>%
  summarise(last_colony_independent_for = min(years_independent, na.rm = TRUE))
```

#### Arithmetic with dates

Adding or substracting days to dates is quite easy:

```{r}
ymd("2018-12-31") + 16
```

It is also possible to be more explicit and use `days()`:

```{r}
ymd("2018-12-31") + days(16)
```

To add years, you can use `years()`:

```{r}
ymd("2018-12-31") + years(1)
```

But you have to be careful with leap years:

```{r}
ymd("2016-02-29") + years(1)
```

Because 2017 is not a leap year, the above computation returns `NA`. The same goes for months with
a different number of days:

```{r}
ymd("2018-12-31") + months(2)
```

The way to solve these issues is to use the special `%m+%` infix operator:

```{r}
ymd("2016-02-29") %m+% years(1)
```

and for months:

```{r}
ymd("2018-12-31") %m+% months(2)
```

`{lubridate}` contains many more functions. If you often work with dates, duration or interval
data, `{lubridate}` is a package that you have to add to your toolbox.

### Manipulate strings with `{stringr}`

`{stringr}` contains functions to manipulate strings. In Chapter 10, I will teach you about regular
expressions, but the functions contained in `{stringr}` allow you to already do a lot of work on
strings, without needing to be a regular expression expert.

I will discuss the most common string operations: detecting, locating, matching, searching and
replacing, and exctracting/removing strings.

To introduce these operations, let us use an ALTO file of an issue of *The Winchester News* from
October 31, 1910, which you can find on this
[link](https://gist.githubusercontent.com/b-rodrigues/5139560e7d0f2ecebe5da1df3629e015/raw/e3031d894ffb97217ddbad1ade1b307c9937d2c8/gistfile1.txt) (to see
how the newspaper looked like,
[click here](https://chroniclingamerica.loc.gov/lccn/sn86069133/1910-10-31/ed-1/seq-1/)). I re-hosted
the file on a public gist for archiving purposes. While working on the book, the original site went
down several times...

ALTO is an XML schema for the description of text OCR and layout information of pages for digitzed
material, such as newspapers (source: [ALTO Wikipedia page](https://en.wikipedia.org/wiki/ALTO_(XML))).
For more details, you can read my
[blogpost](https://www.brodrigues.co/blog/2019-01-13-newspapers_mets_alto/)
on the matter, but for our current purposes, it is enough to know that the file contains the text
of newspaper articles. The file looks like this:

```
<TextLine HEIGHT="138.0" WIDTH="2434.0" HPOS="4056.0" VPOS="5814.0">
<String STYLEREFS="ID7" HEIGHT="108.0" WIDTH="393.0" HPOS="4056.0" VPOS="5838.0" CONTENT="timore" WC="0.82539684">
<ALTERNATIVE>timole</ALTERNATIVE>
<ALTERNATIVE>tlnldre</ALTERNATIVE>
<ALTERNATIVE>timor</ALTERNATIVE>
<ALTERNATIVE>insole</ALTERNATIVE>
<ALTERNATIVE>landed</ALTERNATIVE>
</String>
<SP WIDTH="74.0" HPOS="4449.0" VPOS="5838.0"/>
<String STYLEREFS="ID7" HEIGHT="105.0" WIDTH="432.0" HPOS="4524.0" VPOS="5847.0" CONTENT="market" WC="0.95238096"/>
<SP WIDTH="116.0" HPOS="4956.0" VPOS="5847.0"/>
<String STYLEREFS="ID7" HEIGHT="69.0" WIDTH="138.0" HPOS="5073.0" VPOS="5883.0" CONTENT="as" WC="0.96825397"/>
<SP WIDTH="74.0" HPOS="5211.0" VPOS="5883.0"/>
<String STYLEREFS="ID7" HEIGHT="69.0" WIDTH="285.0" HPOS="5286.0" VPOS="5877.0" CONTENT="were" WC="1.0">
<ALTERNATIVE>verc</ALTERNATIVE>
<ALTERNATIVE>veer</ALTERNATIVE>
</String>
<SP WIDTH="68.0" HPOS="5571.0" VPOS="5877.0"/>
<String STYLEREFS="ID7" HEIGHT="111.0" WIDTH="147.0" HPOS="5640.0" VPOS="5838.0" CONTENT="all" WC="1.0"/>
<SP WIDTH="83.0" HPOS="5787.0" VPOS="5838.0"/>
<String STYLEREFS="ID7" HEIGHT="111.0" WIDTH="183.0" HPOS="5871.0" VPOS="5835.0" CONTENT="the" WC="0.95238096">
<ALTERNATIVE>tll</ALTERNATIVE>
<ALTERNATIVE>Cu</ALTERNATIVE>
<ALTERNATIVE>tall</ALTERNATIVE>
</String>
<SP WIDTH="75.0" HPOS="6054.0" VPOS="5835.0"/>
<String STYLEREFS="ID3" HEIGHT="132.0" WIDTH="351.0" HPOS="6129.0" VPOS="5814.0" CONTENT="cattle" WC="0.95238096"/>
</TextLine>
```

We are interested in the strings after `CONTENT=`. We are going to use functions from the `{stringr}`
package to get the strings after `CONTENT=`. In Chapter 10, we are going to explore this file
again, but using complex regular expressions to get all the content in one go.

#### Getting text data into Rstudio

First of all, let us read in the file:

```{r, cache=TRUE}
winchester <- read_lines("https://gist.githubusercontent.com/b-rodrigues/5139560e7d0f2ecebe5da1df3629e015/raw/e3031d894ffb97217ddbad1ade1b307c9937d2c8/gistfile1.txt")
```

Even though the file is an XML file, I still read it in using `read_lines()` and not `read_xml()`
from the `{xml2}` package. This is for the purposes of the current exercise, and also because I
always have trouble with XML files, and prefer to treat them as simple text files, and use regular
expressions to get what I need.

Now that the ALTO file is read in and saved in the `winchester` variable, you might want to print
the whole thing in the console. Before that, take a look at the structure:

```{r}
str(winchester)
```

So the `winchester` variable is a character atomic vector with 43 elements. So first, we need to
understand what these elements are. Let's start with the first one:

```{r}
winchester[1]
```

Ok, so it seems like the first element is part of the header of the file. What about the second one?

```{r}
winchester[2]
```

Same. So where is the content? The file is very large, so if you print it in the console, it will
take quite some time to print, and you will not really be able to make out anything. The best
way would be to try to detect the string `CONTENT` and work from there.

#### Detecting, getting the position and locating strings

When confronted to an atomic vector of strings, you might want to know inside which elements you
can find certain strings. For example, to know which elements of `winchester` contain the string
`CONTENT`, use `str_detect()`:

```{r}
winchester %>%
  str_detect("CONTENT")
```

This returns a boolean atomic vector of the same length as `winchester`. If the string `CONTENT` is
nowhere to be found, the result will equal `FALSE`, if not it will equal `TRUE`. Here it is easy to
see that the last element contains the string `CONTENT`. But what if instead of having 43 elements,
the vector had 24192 elements? And hundreds would contain the string `CONTENT`? It would be easier
to instead have the indices of the vector where one can find the word `CONTENT`. This is possible
with `str_which()`:

```{r}
winchester %>%
  str_which("CONTENT")
```

Here, the result is 43, meaning that the 43rd element of `winchester` contains the string `CONTENT`
somewhere. If we need more precision, we can use `str_locate()` and `str_locate_all()`. To explain
how both these functions work, let's create a very small example:

```{r}
ancient_philosophers <- c("aristotle", "plato", "epictetus", "seneca the younger", "epicurus", "marcus aurelius")
```

Now suppose I am interested in philosophers whose name ends in `us`. Let us use `str_locate()` first:

```{r}
ancient_philosophers %>%
  str_locate("us")
```

You can interpret the result as follows: in the rows, the index of the vector where the
string `us` is found. So the 3rd, 5th and 6th philosopher have `us` somewhere in their name.
The result also has two columns: `start` and `end`. These give the position of the string. So the
string `us` can be found starting at position 8 of the 3rd element of the vector, and ends at position
9. Same goes for the other philisophers. However, consider Marcus Aurelius. He has two names, both
ending with `us`. However, `str_locate()` only shows the position of the `us` in `Marcus`.

To get both `us` strings, you need to use `str_locate_all()`:

```{r}
ancient_philosophers %>%
  str_locate_all("us")
```

Now we get the position of the two `us` in Marcus Aurelius. Doing this on the `winchester` vector
will give use the position of the `CONTENT` string, but this is not really important right now. What
matters is that you know how `str_locate()` and `str_locate_all()` work.

So now that we know what interests us in the 43nd element of `winchester`, let's take a closer
look at it:

```{r, eval=FALSE}
winchester[43]
```

As you can see, it's a mess:

```
<TextLine HEIGHT=\"126.0\" WIDTH=\"1731.0\" HPOS=\"17160.0\" VPOS=\"21252.0\"><String HEIGHT=\"114.0\" WIDTH=\"354.0\" HPOS=\"17160.0\" VPOS=\"21264.0\" CONTENT=\"0tV\" WC=\"0.8095238\"/><SP WIDTH=\"131.0\" HPOS=\"17514.0\" VPOS=\"21264.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"111.0\" WIDTH=\"474.0\" HPOS=\"17646.0\" VPOS=\"21258.0\" CONTENT=\"BATES\" WC=\"1.0\"/><SP WIDTH=\"140.0\" HPOS=\"18120.0\" VPOS=\"21258.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"114.0\" WIDTH=\"630.0\" HPOS=\"18261.0\" VPOS=\"21252.0\" CONTENT=\"President\" WC=\"1.0\"><ALTERNATIVE>Prcideht</ALTERNATIVE><ALTERNATIVE>Pride</ALTERNATIVE></String></TextLine><TextLine HEIGHT=\"153.0\" WIDTH=\"1689.0\" HPOS=\"17145.0\" VPOS=\"21417.0\"><String STYLEREFS=\"ID7\" HEIGHT=\"105.0\" WIDTH=\"258.0\" HPOS=\"17145.0\" VPOS=\"21439.0\" CONTENT=\"WM\" WC=\"0.82539684\"><TextLine HEIGHT=\"120.0\" WIDTH=\"2211.0\" HPOS=\"16788.0\" VPOS=\"21870.0\"><String STYLEREFS=\"ID7\" HEIGHT=\"96.0\" WIDTH=\"102.0\" HPOS=\"16788.0\" VPOS=\"21894.0\" CONTENT=\"It\" WC=\"1.0\"/><SP WIDTH=\"72.0\" HPOS=\"16890.0\" VPOS=\"21894.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"96.0\" WIDTH=\"93.0\" HPOS=\"16962.0\" VPOS=\"21885.0\" CONTENT=\"is\" WC=\"1.0\"/><SP WIDTH=\"80.0\" HPOS=\"17055.0\" VPOS=\"21885.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"102.0\" WIDTH=\"417.0\" HPOS=\"17136.0\" VPOS=\"21879.0\" CONTENT=\"seldom\" WC=\"1.0\"/><SP WIDTH=\"80.0\" HPOS=\"17553.0\" VPOS=\"21879.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"96.0\" WIDTH=\"267.0\" HPOS=\"17634.0\" VPOS=\"21873.0\" CONTENT=\"hard\" WC=\"1.0\"/><SP WIDTH=\"81.0\" HPOS=\"17901.0\" VPOS=\"21873.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"87.0\" WIDTH=\"111.0\" HPOS=\"17982.0\" VPOS=\"21879.0\" CONTENT=\"to\" WC=\"1.0\"/><SP WIDTH=\"81.0\" HPOS=\"18093.0\" VPOS=\"21879.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"96.0\" WIDTH=\"219.0\" HPOS=\"18174.0\" VPOS=\"21870.0\" CONTENT=\"find\" WC=\"1.0\"/><SP WIDTH=\"77.0\" HPOS=\"18393.0\" VPOS=\"21870.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"69.0\" WIDTH=\"66.0\" HPOS=\"18471.0\" VPOS=\"21894.0\" CONTENT=\"a\" WC=\"1.0\"/><SP WIDTH=\"77.0\" HPOS=\"18537.0\" VPOS=\"21894.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"78.0\" WIDTH=\"384.0\" HPOS=\"18615.0\" VPOS=\"21888.0\" CONTENT=\"succes\" WC=\"0.82539684\"><ALTERNATIVE>success</ALTERNATIVE></String></TextLine><TextLine HEIGHT=\"126.0\" WIDTH=\"2316.0\" HPOS=\"16662.0\" VPOS=\"22008.0\"><String STYLEREFS=\"ID7\" HEIGHT=\"75.0\" WIDTH=\"183.0\" HPOS=\"16662.0\" VPOS=\"22059.0\" CONTENT=\"sor\" WC=\"1.0\"><ALTERNATIVE>soar</ALTERNATIVE></String><SP WIDTH=\"72.0\" HPOS=\"16845.0\" VPOS=\"22059.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"90.0\" WIDTH=\"168.0\" HPOS=\"16917.0\" VPOS=\"22035.0\" CONTENT=\"for\" WC=\"1.0\"/><SP WIDTH=\"72.0\" HPOS=\"17085.0\" VPOS=\"22035.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"69.0\" WIDTH=\"267.0\" HPOS=\"17157.0\" VPOS=\"22050.0\" CONTENT=\"even\" WC=\"1.0\"><ALTERNATIVE>cen</ALTERNATIVE><ALTERNATIVE>cent</ALTERNATIVE></String><SP WIDTH=\"77.0\" HPOS=\"17434.0\" VPOS=\"22050.0\"/><String STYLEREFS=\"ID7\" HEIGHT=\"66.0\" WIDTH=\"63.0\" HPOS=\"17502.0\" VPOS=\"22044.0\"
```

The file was imported without any newlines. So we need to insert them ourselves, by splitting the
string in a clever way.

#### Splitting strings

There are two functions included in `{stringr}` to split strings, `str_split()` and `str_split_fixed()`.
Let's go back to our ancient philosophers. Two of them, Seneca the Younger and Marcus Aurelius have
something else in common than both being Roman Stoic philosophers. Their names are composed of several
words. If we want to split their names at the space character, we can use `str_split()` like this:

```{r}
ancient_philosophers %>%
  str_split(" ")
```

`str_split()` also has a `simplify = TRUE` option:

```{r}
ancient_philosophers %>%
  str_split(" ", simplify = TRUE)
```

This time, the returned object is a matrix.

What about `str_split_fixed()`? The difference is that here you can specify the number of pieces
to return. For example, you could consider the name "Aurelius" to be the middle name of Marcus Aurelius,
and the "the younger" to be the middle name of Seneca the younger. This means that you would want
to split the name only at the first space character, and not at all of them. This is easily achieved
with `str_split_fixed()`:

```{r}
ancient_philosophers %>%
  str_split_fixed(" ", 2)
```

This gives the expected result.

So how does this help in our case? Well, if you look at how the ALTO file looks like, at the beginning
of this section, you will notice that every line ends with the ">" character. So let's split at
that character!

```{r}
winchester_text <- winchester[43] %>%
  str_split(">")
```

Let's take a closer look at `winchester_text`:

```{r}
str(winchester_text)
```

So this is a list of length one, and the first, and only, element of that list is an atomic vector
with 19706 elements. Since this is a list of only one element, we can simplify it by saving the
atomic vector in a variable:

```{r}
winchester_text <- winchester_text[[1]]
```

Let's now look at some lines:

```{r}
winchester_text[1232:1245]
```

This now looks easier to handle. We can narrow it down to the lines that only contain the string
we are interested in, "CONTENT". First, let's get the indices:

```{r}
content_winchester_index <- winchester_text %>%
  str_which("CONTENT")
```

How many lines contain the string "CONTENT"?

```{r}
length(content_winchester_index)
```

As you can see, this reduces the amount of data we have to work with. Let us save this is a new
variable:

```{r}
content_winchester <- winchester_text[content_winchester_index]
```

#### Matching strings

Matching strings is useful, but only in combination with regular expressions. As stated at the
beginning of this section, we are going to learn about regular expressions in Chapter 10, but in
order to make this section useful, we are going to learn the easiest, but perhaps the most useful
regular expression: `.*`.

Let's go back to our ancient philosophers, and use `str_match()` and see what happens. Let's match
the "us" string:

```{r}
ancient_philosophers %>%
  str_match("us")
```

Not very useful, but what about the regular expression `.*`? How could it help?

```{r}
ancient_philosophers %>%
  str_match(".*us")
```

That's already very interesting! So how does `.*` work? To understand, let's first start by using
`.` alone:

```{r}
ancient_philosophers %>%
  str_match(".us")
```

This also matched whatever symbol comes just before the "u" from "us". What if we use two `.` instead?

```{r}
ancient_philosophers %>%
  str_match("..us")
```

This time, we get the two symbols that immediately precede "us". Instead of continuing like this
we now use the `*`, which matches zero or more of `.`. So by combining `*` and `.`, we can match
any symbol repeatedly, until there is nothing more to match. Note that there is also `+`, which works
similarly to `*`, but it matches one or more symbols.

There is also a `str_match_all()`:

```{r}
ancient_philosophers %>%
  str_match_all(".*us")
```

In this particular case it does not change the end result, but keep it in mind for cases like this one:

```{r}
c("haha", "huhu") %>%
  str_match("ha")
```

and:

```{r}
c("haha", "huhu") %>%
  str_match_all("ha")
```

What if we want to match names containing the letter "t"? Easy:

```{r}
ancient_philosophers %>%
  str_match(".*t.*")
```

So how does this help us with our historical newspaper? Let's try to get the strings that come
after "CONTENT":

```{r}
winchester_content <- winchester_text %>%
  str_match("CONTENT.*")
```

Let's use our faithful `str()` function to take a look:

```{r}
winchester_content %>%
  str
```

Hum, there's a lot of `NA` values! This is because a lot of the lines from the file did not have the
string "CONTENT", so there is no match possible. Let's us remove all these `NA`s. Because the
result is a matrix, we cannot use the `filter()` function from `{dplyr}`. So we need to convert it
to a tibble first:

```{r}
winchester_content <- winchester_content %>%
  as.tibble() %>%
  filter(!is.na(V1))
```

Because matrix columns do not have names, when a matrix gets converted into a tibble, the firt column
gets automatically called `V1`. This is why I filter on this column. Let's take a look at the data:

```{r}
head(winchester_content)
```

#### Searching and replacing strings

We are getting close to the final result. We still need to do some cleaning however. Since our data
is inside a nice tibble, we might as well stick with it. So let's first rename the column and 
change all the strings to lowercase:

```{r}
winchester_content <- winchester_content %>% 
  mutate(content = tolower(V1)) %>% 
  select(-V1)
```

Let's take a look at the result:

```{r}
head(winchester_content)
```

The second part of the string, "wc=...." is not really interesting. Let's search and replace this
with an empty string, using `str_replace()`:

```{r}
winchester_content <- winchester_content %>% 
  mutate(content = str_replace(content, "wc.*", ""))

head(winchester_content)
```

We need to use the regular expression from before to replace "wc" and every character that follows.
The same can be use to remove "content=":

```{r}
winchester_content <- winchester_content %>% 
  mutate(content = str_replace(content, "content=", ""))

head(winchester_content)
```

We are almost done, but some cleaning is still necessary:

#### Exctracting or removing strings

Now, because I now the ALTO spec, I know how to find words that are split between two sentences: 

```{r}
winchester_content %>% 
  filter(str_detect(content, "hyppart"))
```

For instance, the word "average" was split over two lines, the first part of the word, "aver" on the
first line, and the second part of the word, "age", on the second line. We want to keep what comes
after "subs_content". Let's extract the word "average" using `str_extract()`. However, because only
some words were split between two lines, we first need to detect where the string "hyppart1" is 
located, and only then can we extract what comes after "subs_content". Thus, we need to combine
`str_detect()` to first detect the string, and then `str_extract()` to extract what comes after 
"subs_content":

```{r}
winchester_content <- winchester_content %>% 
  mutate(content = if_else(str_detect(content, "hyppart1"), 
                           str_extract_all(content, "content=.*", simplify = TRUE), 
                           content))
```

Let's take a look at the result:

```{r}
winchester_content %>% 
  filter(str_detect(content, "content"))
```

We still need to get rid of the string "content=" and then of all the strings that contain "hyppart2",
which are not needed now:

```{r}
winchester_content <- winchester_content %>% 
  mutate(content = str_replace(content, "content=", "")) %>% 
  mutate(content = if_else(str_detect(content, "hyppart2"), NA_character_, content))

head(winchester_content)
```

Almost done! We only need to remove the `"` characters:

```{r}
winchester_content <- winchester_content %>% 
  mutate(content = str_replace_all(content, "\"", "")) 

head(winchester_content)
```

Let's remove space characters with `str_trim()`:

```{r}
winchester_content <- winchester_content %>% 
  mutate(content = str_trim(content)) 

head(winchester_content)
```

To finish off this section, let's remove stop words (words that do not add any meaning to a sentence,
such as "as", "and"...) and words that are composed of less than 3 characters. You can find a dataset
with stopwords inside the `{stopwords}` package:

```{r}
library(stopwords)

data(data_stopwords_stopwordsiso)

eng_stopwords <- tibble("content" = data_stopwords_stopwordsiso$en)

winchester_content <- winchester_content %>% 
  anti_join(eng_stopwords) %>% 
  filter(nchar(content) > 3)

head(winchester_content)
```

That's it for this section! You now know how to work with strings, but in Chapter 10 we are going
one step further by learning about regular expressions, which offer much more power.

### Tidy data frames with `{tibble}`

We have already seen and used several functions from the `{tibble}` package. Let's now go through
some more useful functions. 

#### Creating tibbles

`tribble()` makes it easy to create tibble row by row, manually:

```{r, include=FALSE}
cars <- tribble(
  ~combustion, ~doors,
  "oil", 3,
  "diesel", 5,
  "oil", 5,
  "electric", 5
)
```

It is also possible to create a tibble from a named list:

```{r}
as_tibble(list("combustion" = c("oil", "diesel", "oil", "electric"),
               "doors" = c(3, 5, 5, 5)))
```

```{r}
enframe(list("combustion" = c(1,2), "doors" = c(1,2,4), "cylinders" = c(1,8,9,10)))
```


## List-columns

To learn about list-columns, let's first focus on a single character of the `starwars` dataset:

```{r}
data(starwars)
```

```{r}
starwars %>%
  filter(name == "Luke Skywalker") %>%
  glimpse()
```

We see that the columns `films`, `vehicles` and `starships` are all lists, and in the case of
`films`, it lists all the films where Luke Skywalker has appeared. What if you want to take a closer look at this list?

```{r}
starwars %>%
  filter(name == "Luke Skywalker") %>%
  pull(films)
```

`pull()` is a `dplyr` function that extract (pulls) the column you're interested in. It is quite
useful when you want to inspect a column.

Suppose we want to create a categorical variable which counts the number of movies in which the
characters have appeared. For this we need to compute the length of the list, or count the number
of elements this list has. Let's try with `length()` a base R function:

```{r}
starwars %>%
  filter(name == "Luke Skywalker") %>%
  pull(films) %>%
  length()
```

This might be surprising at first, because we know that Luke Skywalker has appeared in more than 1
movie... the problem here is that for each individual, `films` is a list, whose single element is
a vector of characters. This means that `length(films)` computes the length of the list, which is
one, and not the length of the vector contained in the list! How can we get the length of the
vector of characters contained in the list and for each character? For this we need to use
`dplyr::rowwise()` and remove the `filter()` function and use `mutate()` to add this column to the
dataset:


```{r}
starwars <- starwars %>%
  rowwise() %>%
  mutate(n_films = length(films))
```

`dplyr::rowwise()` is useful when working with list-columns: columns that have lists as elements.

Let's take a look at the characters and the number of films they have appeared in:

```{r}
starwars %>%
  select(name, n_films)
```

Now we can create a factor variable that groups characters by asking whether they appeared only in
1 movie, or more:

```{r}
starwars <- starwars %>%
  mutate(more_1 = case_when(n_films == 1 ~ "Exactly one movie",
                            n_films != 1 ~ "More than 1 movie"))
```

`case_when()` is a `dplyr` function that works similarly to the standard `if..else..` construct of
many programming languages (R also has this, we are going to learn about it in later chapters).

You can also create list columns with your own datasets, by using `tidyr::nest()`. Remember the
fake `survey_data` I created to illustrate `spread()` and `gather()`? Let's go back to that dataset
again:

```{r}
print(survey_data)

nested_data = survey_data %>%
  nest(variable, value)

print(nested_data)
```

This creates a new tibble, with columns `id` and `data`. `data` is a list-column that contains
tibbles; each tibble is the `variable` and `value` for each individual:

```{r}
nested_data %>%
  filter(id == "1") %>%
  pull(data)
```

As you can see, for individual 1, the column data contains a 2x2 tibble with columns `variable` and
`value`. You might be wondering why this is useful, because this seems to introduce an unnecessary
layer of complexity. The usefulness of list-columns will become apparent in the next chapters,
where we are going to learn how to repeat actions over, say, individuals.

## Going beyond descriptive statistics and data manipulation

The `{tidyverse}` collection of packages can do much more than simply data manipulation and
descriptive statisics. You can use the principles we have covered and the functions you now know
to do much more. For instance, you can use a few `{tidyverse}` functions to do Monte Carlo simulations,
for example to estimate $\pi$.

Draw the unit circle inside the unit square, the ratio of the area of the circle to the area of the
square will be $\pi/4$. Then shot K arrows at the square; roughly $K*\pi/4$ should have fallen
inside the circle. So if now you shoot N arrows at the square, and M fall inside the circle, you have
the following relationship $M = N*\pi/4$. You can thus compute $\pi$ like so: $\pi = 4*M/N$.

The more arrows N you throw at the square, the better approximation of $\pi$ you'll have. Let's
try to do this with a tidy Monte Carlo simulation. First, let's randomly pick some points inside
the unit square:

```{r}
library(tidyverse)

n <- 5000

set.seed(2019)
points <- tibble("x" = runif(n), "y" = runif(n))
```

Now, to know if a point is inside the unit circle, we need to check wether $x^2 + y^2 < 1$. Let's
add a new column to the `points` tibble, called `inside` equal to 1 if the point is inside the
unit circle and 0 if not:

```{r}
points <- points %>%
    mutate(inside = map2_dbl(.x = x, .y = y, ~ifelse(.x**2 + .y**2 < 1, 1, 0))) %>%
    rowid_to_column("N")
```

Let's take a look at `points`:

```{r}
points
```

Now, I can compute the estimation
of $\pi$ at each row, by computing the cumulative sum of the 1's in the `inside` column and dividing
that by the current value of `N` column:

```{r}
points <- points %>%
    mutate(estimate = 4*cumsum(inside)/N)
```

`cumsum(inside)` is the `M` from the formula. Now, we can finish by plotting the result:

```{r}
ggplot(points) +
    geom_line(aes(y = estimate, x = N)) +
    geom_hline(yintercept = pi)
```

In the next chapter, we are going to learn all about `{ggplot2}`, the package I used in the lines
above to create this plot.

As the number of tries grows, the estimation of $\pi$ gets better.

Using a data frame as a structure to hold our simulated points and the results makes it very easy
to avoid loops, and thus write code that is more concise and easier to follow.
If you studied a quantitative field in university, you might have done a similar exercise at the
time, very likely by defining a matrix to hold your points, and an empty vector to hold whether a
particular point was inside the unit circle. Then you wrote a loop to compute whether
a point was inside the unit circle, save this result in the before-defined empty vector and then
compute the estimation of $\pi$. Again, I take this opportunity here to stress that there is nothing
wrong with this approach per se, but R is better suited for a workflow where lists or data frames
are the central objects and where the analyst operates over them with functional programming techniques.

## Exercises

### Exercise 1 {-}

Load the `LaborSupply` dataset from the `Ecdat` package and answer the following questions:

* Compute the average annual hours worked by year (plus standard deviation)
* What age group worked the most hours in the year 1982?
* Create a variable, `n_years` that equals the number of years an  individual stays in the panel. Is the panel balanced?
* Which are the individuals that do not have any kids during the whole period? Create a variable, `no_kids`, that flags these individuals (1 = no kids, 0 = kids)
* Using the `no_kids` variable from before compute the average wage, standard deviation and number of observations in each group for the year 1980 (no kids group vs kids group).
* Create the lagged logarithm of hours worked and wages. Remember that this is a panel.

```{r, eval=FALSE, include=FALSE}
library(Ecdat)
library(dplyr)

data("LaborSupply")

# Compute the average annual hours worked by year (plus standard deviation)

LaborSupply %>%
  group_by(year) %>%
  summarise(mean(lnhr), sd(lnhr))

# What age group worked the most hours in the year 1982?

LaborSupply %>%
  filter(year == 1982) %>%
  group_by(age) %>%
  mutate(total_lnhr_age = sum(lnhr)) %>%
  ungroup() %>%
  filter(total_lnhr_age == max(total_lnhr_age))

# Create a variable, `n_years` that equals the number of years an
# individual stays in the panel. Is the panel balanced?

LaborSupply %>%
  group_by(id) %>%
  mutate(n_years = n()) %>%
  ungroup() %>%
  summarise(mean(10))

# Which are the individuals that do not have any kids during the whole period?
# Create a variable, `no_kids`, that flags these individuals (1 = no kids, 0 = kids)

LaborSupply = LaborSupply %>%
  group_by(id) %>%
  mutate(n_kids = max(kids)) %>%
  mutate(no_kids = ifelse(n_kids == 0, 1, 0))

# Using the `no_kids` variable from before compute the average wage in 1980 for these two groups (no kids group vs kids group).
LaborSupply %>%
  filter(year == 1980) %>%
  group_by(no_kids) %>%
  summarise(mean(lnwg), sd(lnwg), n())

```

### Exercise 2 {-}

* What does the following code do? Copy and paste it in an R interpreter to find out!

```{r, eval=FALSE}
LaborSupply %>%
  group_by(id) %>%
  mutate_at(vars(starts_with("l")), funs(lag, lead))
```

`mutate_at()` is a scoped version of `mutate()` which allows you to specify a number of columns and
functions in one go. This also exists for `summarise()`.

* Using `summarise_at()`, compute the mean, standard deviation and number of individuals of `lnhr` and `lnwg` for each individual.

```{r, eval=FALSE, include = FALSE}
LaborSupply %>%
  group_by(id) %>%
  summarise_at(vars(starts_with("l"), funs(mean, sd)))
```

### Exercise 3 {-}

* In the dataset folder you downloaded at the beginning of the chapter, there is a folder called
"unemployment". I used the data in the section about working with lists of datasets. Using
`rio::import_list()`, read the 4 datasets into R.

```{r, eval=FALSE, echo=FALSE}
paths = Sys.glob("datasets/unemployment/*.csv")

all_datasets = import_list(paths)
```

* Using `map()`, map the `janitor::clean_names()` function to each dataset (just like in the example
in the section on working with lists of datasets). Then, still with `map()` and `mutate()` convert
all commune names in the `commune` column with the function `tolower()`, in a new column called `lcommune`.
This is not an easy exercise; so here are some hints:

    * Remember that `all_datasets` is a list of datasets. Which function do you use when you want to map a function to each element of a list?
    * Each element of `all_datasets` are `data.frame` objects. Which function do you use to add a column to a `data.frame`?
    * What symbol can you use to access a column of a `data.frame`?

```{r, eval=FALSE, echo=FALSE}
all_datasets %>%
  map(~mutate(., lcommune = tolower(.$commune)))
```
