# Functional programming

By now, you are pretty familiar with functions in R. Now, let's introduce the functional programming
paradigm, but first, let's go over some formal definitions.

## Functions definition

You should be familiar with function definitions in R. For example, suppose you want to compute
the square root of a number and want to do so using Newton's algorithm:

```{r square_root_loop}
sqrt_newton <- function(a, init, eps = 0.01){
    while(abs(init**2 - a) > eps){
        init <- 1/2 *(init + a/init)
    }
    return(init)
}
```

You can then use this function to get the square root of a number:

```{r}
sqrt_newton(16, 2)
```

We are using a `while` loop inside the body. The *body* of a function are the instructions that
define the function. You can get the body of a function with `body(some_func)` of the function.
In *pure* functional programming languages, like Haskell, you don't have loops. How can you
program without loops, you may ask? In functional programming, loops are replaced by recursion,
which we already discussed in the previous chapter. Let's rewrite our little example above 
with recursion:

```{r square_root_recur}
sqrt_newton_recur <- function(a, init, eps = 0.01){
    if(abs(init**2 - a) < eps){
        result <- init
    } else {
        init <- 1/2 * (init + a/init)
        result <- sqrt_newton_recur(a, init, eps)
    }
    return(result)
}
```

```{r}
sqrt_newton_recur(16, 2)
```

R is not a pure functional programming language though, so we can still use loops (be it `while` or
`for` loops) in the bodies of our functions. As discussed in the previous chapter, it is actually 
better, performance-wise, to use loops instead of recursion, because R is not tail-call optimized.
I won't got into the details of what tail-call optimization is but just remember that if 
performance is important a loop will be faster. However, sometimes, it is easier to write a 
function using recursion. I personally tend to avoid loops if performance is not important, 
because I find that code that avoids loops is easier to read and debug. However, knowing that 
you can use loops is reassuring. In the coming sections I will show you some built-in functions
that make it possible to avoid writing loops and that don't rely on recursion, so performance
won't be penalized.

## Properties of functions

Mathematical functions have a nice property: we always get the same output for a given input. This
is called referential transparency and we should aim to write our R functions in such a way.

For example, the following function:

```{r}
increment <- function(x){
    return(x + 1)
}
```

Is a referential transparent function. We always get the same result for any `x` that we give to
this function.

This:

```{r}
increment(10)
```

will always produce `11`.

However, this one:

```{r}
increment_opaque <- function(x){
    return(x + spam)
}
```

is not a referential transparent function, because its value depends on the global variable `spam`.

```{r}
spam <- 1

increment_opaque(10)
```

will only produce `11` if `spam = 1`. But what if `spam = 19`?


```{r}
spam <- 19

increment_opaque(10)
```

To make `increment_opaque()` a referential transparent function, it is enough to make `spam` an
argument:

```{r}
increment_not_opaque <- function(x, spam){
    return(x + spam)
}
```

Now even if there is a global variable called `spam`, this will not influence our function:

```{r}
spam <- 19

increment_not_opaque(10, 34)
```

This is because the variable `spam` defined in the body of the function is a local variable. It
could have been called anything else, really. Avoiding opaque functions makes our life easier.

Another property that adepts of functional programming value is that functions should have no, or
very limited, side-effects. This means that functions should not change the state of your program.

For example this function (which is not a referential transparent function):

```{r square_root_loop_side_effects}
count_iter <- 0

sqrt_newton_side_effect <- function(a, init, eps = 0.01){
    while(abs(init**2 - a) > eps){
        init <- 1/2 *(init + a/init)
        count_iter <<- count_iter + 1 # The "<<-" symbol means that we assign the
    }                                 # RHS value in a variable in the global environment
    return(init)
}
```

If you look in the environment pane, you will see that `count_iter` equals 0. Now call this
function with the following arguments:

```{r}
sqrt_newton_side_effect(16000, 2)

print(count_iter)
```

If you check the value of `count_iter` now, you will see that it increased! This is a side effect,
because the function changed something outside of its scope. It changed a value in the global
environment. In general, it is good practice to avoid side-effects. For example, we could make the
above function not have any side effects like this:

```{r square_root_loop_not_more_side_effects}
sqrt_newton_count <- function(a, init, count_iter = 0, eps = 0.01){
    while(abs(init**2 - a) > eps){
        init <- 1/2 *(init + a/init)
        count_iter <- count_iter + 1
    }
    return(c(init, count_iter))
}
```

Now, this function returns a list with two elements, the result, and the number of iterations it
took to get the result:

```{r}
sqrt_newton_count(16000, 2)
```

Writing to disk is also considered a side effect, because the function changes something (a file)
outside its scope. But this cannot be avoided since you *want* to write to disk. 
Just remember: try to avoid having functions changing variables in the global environment unless 
you have a very good reason of doing so.

Finally, another property of mathematical functions, is that they do one single thing. Functional
programming purists also program their functions to do one single task. This has benefits, but
can complicate things. The function we wrote previously does two things: it computes the square
root of a number and also returns the number of iterations it took to compute the result. However,
this is not a bad thing; the function is doing two tasks, but these tasks are related to each other
and it makes sense to have them together. My piece of advice: avoid having functions that do 
many *unrelated* things. This makes debugging harder.

In conclusion: you should strive for referential transparency, try to avoid side effects unless you
have a good reason to have them and try to keep your functions short and do as little tasks as
possible. This makes testing and debugging easier, as you will see.

## Functional programming with `{purrr}`

Hadley Wickham developed a package called `purrr` which contains a lot of very useful functions. 

### The `map*()` family of functions

In the previous section we saw how to map a function to each element of a list. Each version of an
`*apply()` function has a different purpose, but it is not very easy to remember which one returns
a list, which other one returns an atomic vector and so on. If you're working on data frames you
can use `apply()` to sum (for example) over columns or rows, because you can specify which `MARGIN`
you want to sum over. But you do not get a data frame back. In the `purrr` package, each of the
functions that do mapping have a similar name. The first part of these functions' names all start
with `map_` and the second part tells you what this function is going to output. For example, if
you want `double`s out, you would use `map_dbl()`. If you are working on data frames want a data
frame back, you would use `map_df()`. These are much more intuitive and easier to remember and we're
going to learn how to use them in the chapter about [The Tidyverse](#tidyverse). For now, let's just focus on
the basic functions, `map()` and `reduce()` (and some variants of `reduce()`). To map a function
to every element of a list, simply use `map()`:

```{r}
library("purrr")
numbers <- c(7, 8, 19, 64)

map(numbers, sqrt_newton, init = 1)
```


If you want print "hello" using a function from `purrr` you would need to use `rerun()`:

```{r}
rerun(10, "hello")
```

`rerun()` simply runs an expression (which can be arbitrarily complex) `n` times, whereas `map()`
maps a function to a list of inputs, so to achieve the same with `map()`, you need to map the `print()`
function to a vector of characters:

```{r}
map(rep("hello", 10), print)
```

`rep()` is a function that creates a vector by repeating something, in this case the string "hello",
as many times as needed, here 10. The output here is a bit different that before though, because first
you will see "hello" printed 10 times, but `map()` always returns a list, this means that you 
will also get a list where each element is the string "hello".

We know the standard `map()` function, which returns a list, but there are a number of 
variants of this function. `map_dbl()` returns an atomic vector of doubles:

```{r}
map_dbl(numbers, sqrt_newton, init = 1)
```

`map_chr()` returns an atomic vector of strings:

```{r}
map_chr(numbers, sqrt_newton, init = 1)
```

`map_lgl()` returns an atomic vector of `TRUE` or `FALSE`:

```{r}
divisible <- function(x, y){
  if_else(x %% y == 0, TRUE, FALSE)
}

map_lgl(seq(1:100), divisible, 3)
```

There are also other interesting variants, such as `map_if()`:

```{r}
a <- seq(1,10)

map_if(a, (function(x) divisible(x, 2)), sqrt)
```

I used `map_if()` to take the square root of only those numbers in vector `a` that are divisble by 2,
by using an anonymous function that checks if a number is divisible by 2 (by wrapping `divisible()`).

`map_at()` is similar to `map_if()` but maps the function at a position specified by the user:

```{r}
map_at(numbers, c(1, 3), sqrt)
```

or if you have a named list:

```{r}
recipe <- list("spam" = 1, "eggs" = 3, "bacon" = 10)

map_at(recipe, "bacon", `*`, 2)
```

I used `map_at()` to double the quantity of bacon in the recipe (by using the `*` function, and specifying
its second argument, `2`. Try the following in the command prompt: `` `*`(3, 4) ``).

`map2()` is the equivalent of `mapply()` and `pmap()` is the generalisation of `map2()` for more
than 2 arguments:

```{r}
print(a)

b <- seq(1, 2, length.out = 10)

print(b)

map2(a, b, `*`)
```

```{r}
n <- seq(1:10)

pmap(list(a, b, n), rnorm)
```


### Reducing with `purrr`

In the `purrr` package, you can find two more functions for folding: `reduce()` and
`reduce_right()`. The difference between `reduce()` and `reduce_right()` is pretty obvious:
`reduce_right()` starts from the right!

```{r, include=FALSE}
a <- seq(1, 10)

purrr::reduce(a, `-`)
purrr::reduce_right(a, `-`)
```

For operations that are not commutative, this makes a difference. Other interesting folding
functions are `accumulate()` and `accumulate_right()`:

```{r, eval=FALSE}
a <- seq(1, 10)

accumulate(a, `-`)
accumulate_right(a, `-`)
```

These two functions keep the intermediary results.

In the previous chapter, we wrote a loop to compute the sum of the 100 first integers. We can do
the same with `purrr::reduce()`:

```{r}
result = reduce(seq(1,100), `+`)

print(result)
```

You certainly agree with me that is simpler to understand. You can even see what happens in more
detail using `accumulate`:

```{r, include=FALSE}
purrr::accumulate(seq(1, 100), `+`)
```

```{r, eval=FALSE}
accumulate(seq(1, 100), `+`)
```

### `safely()` and `possibly()`

`safely()` and `possibly()` are very useful functions. Consider the following situation:

```{r, eval = FALSE}

a <- list("a", 4, 5)

sqrt(a)
```

```{r, eval = FALSE}
Error in sqrt(a) : non-numeric argument to mathematical function
```

Using `map()` or `Map()` will result in a similar error. `safely()` is an higher-order function that
takes one function as an argument and executes it... *safely*, meaning the execution of the function
will not stop if there is an error. The error message gets captured alongside valid results.

```{r}

a <- list("a", 4, 5)

safe_sqrt <- safely(sqrt)

map(a, safe_sqrt)
```
`possibly()` works similarly, but also allows you to specify a return value in case of an error:

```{r}
possible_sqrt <- possibly(sqrt, otherwise = NA_real_)

map(a, possible_sqrt)
```

Of course, in this particular example, the same effect could be obtained way more easily:

```{r}
sqrt(as.numeric(a))
```

However, in some situations, this trick does not work as intended (or at all), so `possibly()` and
`safely()` are the way to go.

### `is_*()` and `as_*()` functions

Remember in Chapter 3, when I introduced `is.*()` and `as.*()` functions? I told you then that we
were going to learn about `is_*()` and `as_*()` in Chapter 9. This is it!

### «Transposing lists»

Another interesting function is `transpose()`. It is not an alternative to the function `t()` from
`base` but, has a similar effect. `transpose()` works on lists. Let's take a look at the example
from before:

```{r}
safe_sqrt <- safely(sqrt, otherwise = NA_real_)

map(a, safe_sqrt)
```

The output is a list with the first element being a list with a result and an error message. One
might want to have all the results in a single list, and all the error messages in another list.
This is possible with `transpose()`:

```{r}
purrr::transpose(map(a, safe_sqrt))
```

I explicitely call `purrr::transpose()` because there is also a `data.table::transpose()`, which
is not the same function. You have to be careful about that sort of thing, because it can cause
errors in your programs and debuging this type of error is a nightmare.

## Using your functions inside `mutate()`

Once you wrote a function, you can easily use it inside a pipe workflow:

```{r}
double_number <- function(x){
  x+x
}
```

```{r}
mtcars %>%
  mutate(double_mpg = double_number(mpg))
```

Granted this example is stupid, but it shows you, again, that functions you define are nothing
special. You can use them just as any other.

You can also avoid to define a function altogether, especially if you need an operation only once,
by using the `.` like this:

```{r}
mtcars %>%
  mutate(double_mpg = .$mpg + .$mpg)
```


## Working with a list of datasets

### Using `map()` to work on lists of datasets

This is our first encouter with a typical functional programming function, `map()`.

Let's read the list of datasets from the previous chapter:


```{r, cache=TRUE}
paths <- Sys.glob("datasets/unemployment/*.csv")

all_datasets <- import_list(paths)

str(all_datasets)
```

For working with lists, another package from the `tidyverse` that is very useful, is called
`purrr`. `purrr` will be presented in-depth in Chapter 9.

The first thing we are going to do is use a function to clean the names of the datasets. These
names are not very easy to work with; there are spaces, and it would be better if the names of the
columns would be all lowercase. For this we are going to use the function `clean_names()` from the
`janitor` package. For a single dataset, I would write this:

```{r, include=FALSE}
library(janitor)
```

```{r, eval = FALSE}
library(janitor)

one_dataset = one_dataset %>%
  clean_names()
```

and I would get a dataset with column names in lowercase and spaces replaced by `_` (and other
corrections). How can I apply, or map, this function to each dataset in the list? To do this I need
to use `purrr::map()`:

```{r, cache=TRUE}
library(purrr)

all_datasets <- all_datasets %>%
  map(clean_names)

all_datasets %>%
  glimpse()
```

So now, what if I want to know, for each dataset, which *communes* have an unemployment rate that is
less than, say, 3%? For a single dataset I would do something like this:

```{r, eval=FALSE}
one_dataset %>%
  filter(unemployment_rate_in_percent < 3)
```

But for a list of datasets, `map()` is needed (and as you will see, that is not all that is needed):

```{r, cache=TRUE}
all_datasets %>%
  map(~filter(., unemployment_rate_in_percent < 3))
```

I know what you're thinking... *what the hell?*. Let me explain: `map()` needs a function to map to
each element of the list. `all_datasets` is the list to which I want to map the function. But what
function? `filter()` is the function I need, so why doesn't:

```{r, eval = FALSE}
all_datasets %>%
  map(filter(unemployment_rate_in_percent < 3))
```
work? This is a bit complicated, and has to do with what is called environments. If you try to run
the code above, you will get this error message:

```
Error in filter(unemployment_rate_in_percent < 3) :
  object 'unemployment_rate_in_percent' not found
```

I won't go into details, but by writing `~filter(., unemployment_rate_in_percent < 3)`, which is a
formula (`~` is the symbol to define formulas, more on this in the later chapters), `map()`
converts it to a function that it can use. If you want to know more about this, you can read it in
[Advanced R](http://adv-r.had.co.nz/Functional-programming.html#closures) by Hadley Wickham, but it
is an advanced topic.


## Mapping your homebrewed functions to lists of datasets

Before merging these datasets together, we would need them to have a `year` column indicating the
year. It would also be helpful if gave names to these datasets. For this task, we can use
`purrr::set_names()`:

```{r, eval=FALSE}
all_datasets = set_names(all_datasets, as.character(seq(2013, 2016)))
```

Let's take a look at the list now:

```{r, eval=FALSE}
str(all_datasets)
```

As you can see, each `data.frame` object contained in the list has been renamed. You can thus
access them with the `$` operator:

```{r, echo=FALSE}
knitr::include_graphics("pics/all_datasets_names.png")
```

### Data frames and `reduce`

Using `map()` we now know how to apply a function to each dataset of a list. But maybe it would be
easier to merge all the datasets first, and then manipulate them? Before that though, I am going to
teach you how to use `purrr::reduce()`, another very powerful function that works on lists. This is
a function that you can find in other programming languages, but sometimes it is called fold.

I think that the following example illustrates the power of `reduce()` well:

```{r, cache=TRUE}
numbers = seq(1, 5) # Create a vector with the numbers 1 to 5

reduce(numbers, `+`, .init = 0)
```

`reduce()` takes a function as an argument, here the function `+`^[This is simply the `+` operator
you're used to. Try this out: `` `+`(1, 5) `` and you'll see `+` is a function like any other. You
just have to write backticks around the plus symbol to make it work.] and then does the following
computation:

```
0 + numbers[1] + numbers[2] + numbers[3]...
```

It applies the user supplied function successively but has to start with something, so we give it
the argument `init` also. This argument is actually optional, but I show it here because in some
cases it might be useful to start the computations at another value than `0`.`reduce()`
generalizes functions that only take two arguments. If you were to write a function that returns
the minimum between two numbers:

```{r, cache=TRUE}
my_min = function(a, b){
    if(a < b){
        return(a)
    } else {
        return(b)
    }
}
```

You could use `reduce()` to get the minimum of a list of numbers:

```{r, cache=TRUE}
numbers2 = c(3, 1, -8, 9)

reduce(numbers2, my_min)
```

As long as you provide a function and a list of elements to `reduce()`, you will get a single
output. So how could `reduce()` help us with merging all the datasets that are in the list? `dplyr`
comes with a lot of function to merge *two* datasets. Remember that I said before that `reduce()`
allows you to generalize a function of two arguments? Let's try it with our list of datasets:


```{r, cache=TRUE}
unemp_lux = reduce(all_datasets, full_join)

glimpse(unemp_lux)
```

`full_join()` is one of the `dplyr` function that merges data. There are others that might be
useful depending on the kind of join operation you need. Let's write this data to disk as we're
going to keep using it for the next chapters:

```{r, cache=TRUE}
export(unemp_lux, "datasets/unemp_lux.csv")
```


## Functional programming and plotting

In this section, we are going to learn how to use the possibilities offered by the `purrr` package
and how it can work together with `ggplot2` to generate many plots. This is a more advanced topic,
but what comes next is also what makes R, and the functional programming paradigm so powerful.

For example, suppose that instead of wanting a single plot with the unemployment rate of each
commune, you need one unemployment plot, per commune:

```{r, cache=TRUE}
unemp_lux_data %>%
  filter(division == "Luxembourg") %>%
  ggplot(aes(year, unemployment_rate_in_percent, group = division)) +
  theme_minimal() +
  labs(title = "Unemployment in Luxembourg", x = "Year", y = "Rate") +
  geom_line()
```

and then you would write the same for "Esch-sur-Alzette" and also for "Wiltz". If you only have to
make to make these 3 plots, copy and pasting the above lines is no big deal:

```{r, cache=TRUE}
unemp_lux_data %>%
  filter(division == "Esch-sur-Alzette") %>%
  ggplot(aes(year, unemployment_rate_in_percent, group = division)) +
  theme_minimal() +
  labs(title = "Unemployment in Esch-sur-Alzette", x = "Year", y = "Rate") +
  geom_line()

unemp_lux_data %>%
  filter(division == "Wiltz") %>%
  ggplot(aes(year, unemployment_rate_in_percent, group = division)) +
  theme_minimal() +
  labs(title = "Unemployment in Esch-sur-Alzette", x = "Year", y = "Rate") +
  geom_line()
```

Put copy and pasting is error prone. Can you spot the copy-paste mistake I made? And what if you
have to create the above plots for all 108 Luxembourguish communes? That's a lot of copy pasting.
What if, once you are done copy pasting, you have to change something, for example, the theme? You
could use the search and replace function of RStudio, true, but sometimes search and replace can
also introduce bugs and typos. You can avoid all these issues by using `purrr::map()`. What do you
need to map over? The commune names. So let's create a vector of commune names:

```{r, cache=TRUE}
communes <- list("Luxembourg", "Esch-sur-Alzette", "Wiltz")
```

Now we can create the graphs using `map()`, or `map2()` to be exact:

```{r, cache=TRUE}
plots_tibble = unemp_lux_data %>%
  filter(division %in% communes) %>%
  group_by(division) %>%
  nest() %>%
  mutate(plot = map2(.x = data, .y = division, ~ggplot(data = .x) +
       theme_minimal() +
       geom_line(aes(year, unemployment_rate_in_percent, group = 1)) +
       labs(title = paste("Unemployment in", .y))))
```

Let's study this line by line: the first line is easy, we simply use `filter()` to keep only the
communes we are interested in. Then we group by `division` and use `tidyr::nest()`. As a refresher,
let's take a look at what this does:

```{r, cache=TRUE}
unemp_lux_data %>%
  filter(division %in% communes) %>%
  group_by(division) %>%
  nest()
```

This creates a tibble with two columns, `division` and `data`, where each individual (or
commune in this case) is another tibble with all the original variables. This is very useful,
because now we can pass these tibbles to `map2()`, to generate the plots. But why `map2()` and
what's the difference with `map()`? `map2()` works the same way as `map()`, but maps over two
inputs:

```{r, cache=TRUE}
numbers1 <- list(1, 2, 3, 4, 5)

numbers2 <- list(9, 8, 7, 6, 5)

map2(numbers1, numbers2, `*`)
```

In our example with the graphs, the two inputs are the data, and the names of the communes. This is
useful to create the title with `labs(title = paste("Unemployment in", .y))))` where `.y` is the
second input of `map2()`, the commune names contained in variable `division`.

So what happened? We now have a tibble called `plots_tibble` that looks like this:

```{r, cache=TRUE}
print(plots_tibble)
```

This tibble contains three columns, `division`, `data` and now a new one called `plot`, that we
created before using the last line `mutate(plot = ...)` (remember that `mutate()` adds columns to
tibbles). `plot` is a list-column, with elements... being plots! Yes you read that right, the
elements of the column `plot` are literally plots. This is what I meant with list columns.
Let's see what is inside the `data` and the `plot` columns exactly:

```{r, cache=TRUE}
plots_tibble %>%
  pull(data)
```

each element of data is a tibble for the specific country with columns `year`, `active_population`,
etc, the original columns. But obviously, there is no `division` column. So to plot the data, and
join all the dots together, we need to add `group = 1` in the call to `ggplot2()` (whereas if you
plot multiple lines in the same graph, you need to write `group = division`).

But more interestingly, how can you actually see the plots? If you want to simply look at them, it
is enough to use `pull()`:

```{r, cache=TRUE}
plots_tibble %>%
  pull(plot)
```

And if we want to save these plots, we can do so using `map2()`:

```{r, eval=FALSE}
map2(paste0(plots_tibble$division, ".pdf"), plots_tibble$plot, ggsave)
```

```
Saving 7 x 5 in image
Saving 6.01 x 3.94 in image
Saving 6.01 x 3.94 in image
```

This was probably the most advanced topic we have studied yet; but you probably agree with me that
it is among the most useful ones. This section is a perfect illustration of the power of functional
programming; you can mix and match functions as long as you give them the correct arguments.
You can pass data to functions that use data and then pass these functions to other functions that
use functions as arguments, such as `map()`.^[Functions that have other functions as input are
called *higher order functions*] `map()` does not care if the functions you pass to it produces tables,
graphs or even another function. `map()` will simply map this function to a list of inputs, and as
long as these inputs are correct arguments to the function, `map()` will do its magic. If you
combine this with list-columns, you can even use `map()` alongside `dplyr` functions and map your
function by first grouping, filtering, etc...

## Exercises

1. Suppose you have an Excel workbook that contains data on three sheets. Create a function that
   reads entire workbooks, and that returns a list of tibbles, where each tibble is the data of one
   sheet (download the example Excel workbook, `example_workbook.xlsx`, from the `assets` folder on
   the books github).

2. Use one of the `map()` functions to combine two lists into one. Consider the following two lists:

```{r, eval = FALSE}
mediterranean <- list("starters" = list("humous", "lasagna"), "dishes" = list("sardines", "olives"))

continental <- list("starters" = list("pea soup", "terrine"), "dishes" = list("frikadelle", "sauerkraut"))
```

The result we'd like to have would look like this:
<!-- map2(mediterranean, continental, append) -->


```{r, eval = FALSE}
$starters
$starters[[1]]
[1] "humous"

$starters[[2]]
[1] "olives"

$starters[[3]]
[1] "pea soup"

$starters[[4]]
[1] "terrine"


$dishes
$dishes[[1]]
[1] "sardines"

$dishes[[2]]
[1] "lasagna"

$dishes[[3]]
[1] "frikadelle"

$dishes[[4]]
[1] "sauerkraut"
```
