---
title: "Fundamentals of Data Visualization"
author: "Claus O. Wilke"
description: "A guide to making visualizations that accurately reflect the data, tell a story, and look professional."
github-repo: clauswilke/dataviz
site: bookdown::bookdown_site
output:
  bookdown::gitbook:
    includes:
      in_header: html_includes/google_analytics.html
bibliography: "bibliography.bib"
biblio-style: "apalike"
link-citations: true
cover-image: cover.png
---

# Welcome {-}

<a href="https://www.amazon.com/gp/product/1492031089"><img src="cover.png" width="304" height="398" alt="Cover image" align="right" style="margin: 0 0 1em 1em; border-width: 1px; border-style: solid; border-color: black;" /></a> This is the website for the book "Fundamentals of Data Visualization," published by O’Reilly Media, Inc. The website contains the complete author manuscript before final copy-editing and other quality control. If you would like to order an official hardcopy or ebook, you can do so at various resellers, including [Amazon,](https://www.amazon.com/gp/product/1492031089) [Barnes and Noble,](https://www.barnesandnoble.com/w/fundamentals-of-data-visualization-claus-o-wilke/1128580869) [Google Play,](https://play.google.com/store/books/details/Claus_O_Wilke_Fundamentals_of_Data_Visualization?id=WmmNDwAAQBAJ) or [Powells.](https://www.powells.com/book/-9781492031086)

The book is meant as a guide to making visualizations that accurately reflect the data, tell a story, and look professional. It has grown out of my experience of working with students and postdocs in my laboratory on thousands of data visualizations. Over the years, I have noticed that the same issues arise over and over. I have attempted to collect my accumulated knowledge from these interactions in the form of this book.

The entire book is written in R Markdown, using RStudio as my text editor and the **bookdown** package to turn a collection of markdown documents into a coherent whole. The book's source code is hosted on GitHub, at https://github.com/clauswilke/dataviz. If you notice typos or other issues, feel free to open an issue on GitHub or submit a pull request. If you do the latter, in your commit message, please add the sentence "I assign the copyright of this contribution to Claus O. Wilke," so that I can maintain the option of publishing this book in other forms.

This work is licensed under the [Attribution-NonCommercial-NoDerivatives 4.0 International](https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode) License. 

<!--chapter:end:index.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(forcats)
library(patchwork)
library(lubridate)
```


# (PART\*) Part I: From data to visualization {-}


# Visualizing data: Mapping data onto aesthetics {#aesthetic-mapping}

Whenever we visualize data, we take data values and convert them in a systematic and logical way into the visual elements that make up the final graphic. Even though there are many different types of data visualizations, and on first glance a scatter plot, a pie chart, and a heatmap don't seem to have much in common, all these visualizations can be described with a common language that captures how data values are turned into blobs of ink on paper or colored pixels on screen. The key insight is the following: All data visualizations map data values into quantifiable features of the resulting graphic. We refer to these features as *aesthetics.*

## Aesthetics and types of data

Aesthetics describe every aspect of a given graphical element. A few examples are provided in Figure \@ref(fig:common-aesthetics). A critical component of every graphical element is of course its *position,* which describes where the element is located. In standard 2d graphics, we describe positions by an *x* and  *y* value, but other coordinate systems and one- or three-dimensional visualizations are possible. Next, all graphical elements have a *shape*, a *size*, and a *color.* Even if we are preparing a black-and-white drawing, graphical elements need to have a color to be visible, for example black if the background is white or white if the background is black. Finally, to the extent we are using lines to visualize data, these lines may have different widths or dash--dot patterns. Beyond the examples shown in Figure \@ref(fig:common-aesthetics), there are many other aesthetics we may encounter in a data visualization. For example, if we want to display text, we may have to specify font family, font face, and font size, and if graphical objects overlap, we may have to specify whether they are partially transparent.

(ref:common-aesthetics) Commonly used aesthetics in data visualization: position, shape, size, color, line width, line type. Some of these aesthetics can represent both continuous and discrete data (position, size, line width, color) while others can usually only represent discrete data (shape, line type).

```{r common-aesthetics, fig.width = 6, fig.asp = 0.45, fig.cap = '(ref:common-aesthetics)'}
aes_pos <- ggdraw() + 
  geom_segment(data = data.frame(x = c(0, 0.5),
                                 xend = c(1, 0.5),
                                 y = c(0.5, 0),
                                 yend = c(0.5, 1)),
                aes(x = x, y = y, xend = xend, yend = yend),
                arrow = arrow(length = grid::unit(12, "pt")), size = .75) +
  draw_text("y", .5, 1, size = 12, vjust = 1, hjust = 2.5, family = dviz_font_family) +
  draw_text("x", 1, .5, size = 12, vjust = 2, hjust = 1, family = dviz_font_family) + 
  coord_cartesian(xlim = c(-.2, 1.2), ylim = c(-.2, 1.2))

aes_color <- ggdraw() +
  geom_tile(data = data.frame(x = 0.15 + .2333*(0:3)),
            aes(x, y = .5, fill = factor(x)), width = .2, height = .6) +
  scale_fill_OkabeIto(guide = "none")

aes_shape <- ggdraw() +
  geom_point(data = data.frame(x = (.5 + 0:3)/4),
             aes(x, y = .5, shape = factor(x)), size = 8, fill = "grey80") +
  scale_shape_manual(values = 21:24)

aes_size <- ggdraw() +
  geom_point(data = data.frame(x = (.5 + 0:3)/4),
             aes(x, y = .5, size = factor(x)), shape = 21, fill = "grey80") +
  scale_size_manual(values = c(2, 5, 8, 11))

aes_lwd <- ggdraw() +
  geom_segment(data = data.frame(x = rep(0.05, 4),
                                 xend = rep(0.95, 4),
                                 y = (1.5 + 0:3)/6,
                                 yend = (1.5 + 0:3)/6,
                                 size = 4:1),
               aes(x = x, y = y, xend = xend, yend = yend, size = size)) +
  scale_size_identity()

aes_ltp <- ggdraw() +
  geom_segment(data = data.frame(x = rep(0.05, 4),
                                 xend = rep(0.95, 4),
                                 y = (1.5 + 0:3)/6,
                                 yend = (1.5 + 0:3)/6,
                                 linetype = 4:1),
               aes(x = x, y = y, xend = xend, yend = yend, linetype = linetype), size = 1) +
  scale_linetype_identity()


plot_grid(aes_pos, aes_shape, aes_size,
          aes_color, aes_lwd, aes_ltp,
          ncol = 3,
          labels = c("position", "shape", "size", "color", "line width", "line type"),
          label_x = 0.05, label_y = 0.95, hjust = 0, vjust = 1)
```


All aesthetics fall into one of two groups: Those that can represent continuous data and those that can not. Continuous data values are values for which arbitrarily fine intermediates exist. For example, time duration is a continuous value. Between any two durations, say 50 seconds and 51 seconds, there are arbitrarily many intermediates, such as 50.5 seconds, 50.51 seconds, 50.50001 seconds, and so on. By contrast, number of persons in a room is a discrete value. A room can hold 5 persons or 6, but not 5.5. For the examples in Figure \@ref(fig:common-aesthetics), position, size, color, and line width can represent continuous data, but shape and line type can usually only represent discrete data.

Next we'll consider the types of data we may want to represent in our visualization. You may think of data as numbers, but numerical values are only two out of several types of data we may encounter. In addition to continuous and discrete numerical values, data can come in the form of discrete categories, in the form of dates or times, and as text (Table \@ref(tab:basic-data-types)). When data is numerical we also call it *quantitative* and when it is categorical we call it *qualitative*. Variables holding qualitative data are *factors*, and the different categories are called *levels*. The levels of a factor are most commonly without order (as in the example of "dog", "cat", "fish" in Table \@ref(tab:basic-data-types)), but factors can also be ordered, when there is an intrinsic order among the levels of the factor (as in the example of "good", "fair", "poor" in Table \@ref(tab:basic-data-types)).

Table: (\#tab:basic-data-types) Types of variables encountered in typical data visualization scenarios.

---------------------------------------------------------------------------------------------------------------------
Type of variable         Examples              Appropriate scale       Description
------------------------ --------------------- ----------------------- ----------------------------------------------
quantitative/numerical   1.3, 5.7, 83,         continuous              Arbitrary numerical values. These can be
continuous               1.5x10^-2^                                    integers, rational numbers, or real numbers.
 
quantitative/numerical   1, 2, 3, 4            discrete                Numbers in discrete units. These are most
discrete                                                               commonly but not necessarily integers.
                                                                       For example, the numbers 0.5, 1.0, 1.5 could
                                                                       also be treated as discrete if intermediate
                                                                       values cannot exist in the given dataset.
 
qualitative/categorical  dog, cat, fish        discrete                Categories without order. These are discrete
unordered                                                              and unique categories that have no inherent
                                                                       order. These variables are
                                                                       also called *factors*.
                                                                       
qualitative/categorical  good, fair, poor      discrete                Categories with order. These are discrete
ordered                                                                and unique categories with an order. For
                                                                       example, "fair" always lies between "good"
                                                                       and "poor". These variables are
                                                                       also called *ordered factors*.

date or time             Jan. 5 2018, 8:03am   continuous or discrete  Specific days and/or times. Also
                                                                       generic dates, such as July 4 or Dec. 25
                                                                       (without year).

text                     The quick brown fox   none, or discrete       Free-form text. Can be treated
                         jumps over the lazy                           as categorical if needed.
                         dog.
---------------------------------------------------------------------------------------------------------------------

To examine a concrete example of these various types of data, take a look at Table \@ref(tab:data-example). It shows the first few rows of a dataset providing the daily temperature normals (average daily temperatures over a 30-year window) for four U.S. locations. This table contains five variables: month, day, location, station ID, and temperature (in degrees Fahrenheit). Month is an ordered factor, day is a discrete numerical value, location is an unordered factor, station ID is similarly an unordered factor, and temperature is a continuous numerical value.

Table: (\#tab:data-example) First 12 rows of a dataset listing daily temperature normals for four weather stations. Data source: NOAA.

 Month   Day  Location      Station ID   Temperature
------- ----- ------------ ------------ -------------
  Jan     1   Chicago      USW00014819        25.6
  Jan     1   San Diego    USW00093107        55.2
  Jan     1   Houston      USW00012918        53.9
  Jan     1   Death Valley USC00042319        51.0
  Jan     2   Chicago      USW00014819        25.5
  Jan     2   San Diego    USW00093107        55.3
  Jan     2   Houston      USW00012918        53.8
  Jan     2   Death Valley USC00042319        51.2
  Jan     3   Chicago      USW00014819        25.3
  Jan     3   San Diego    USW00093107        55.3
  Jan     3   Death Valley USC00042319        51.3
  Jan     3   Houston      USW00012918        53.8



## Scales map data values onto aesthetics

To map data values onto aesthetics, we need to specify which data values correspond to which specific aesthetics values. For example, if our graphic has an *x* axis, then we need to specify which data values fall onto particular positions along this axis. Similarly, we may need to specify which data values are represented by particular shapes or colors. This mapping between data values and aesthetics values is created via *scales*. A scale defines a unique mapping between data and aesthetics (Figure \@ref(fig:basic-scales-example)). Importantly, a scale must be one-to-one, such that for each specific data value there is exactly one aesthetics value and vice versa. If a scale isn't one-to-one, then the data visualization becomes ambiguous.

(ref:basic-scales-example) Scales link data values to aesthetics. Here, the numbers 1 through 4 have been mapped onto a position scale, a shape scale, and a color scale. For each scale, each number corresponds to a unique position, shape, or color and vice versa.

```{r basic-scales-example, fig.width = 5.5, fig.asp = 0.3, fig.cap = '(ref:basic-scales-example)'}
df <- data.frame(x = c(1:4))

scale_num <- ggplot(df, aes(x)) + 
  geom_point(size = 3, color = "#0072B2", y = 1) + 
  scale_y_continuous(limits = c(0.8, 1.2), expand = c(0, 0), breaks = 1, label = "position  ") +
  scale_x_continuous(limits = c(.7, 4.4), breaks = 1:5, labels = c("1", "2", "3", "4", "5"), name = NULL, position = "top") +
  theme_dviz_grid() +
  theme(axis.ticks.length = grid::unit(0, "pt"),
        axis.text = element_text(size = 14),
        axis.title.y = element_blank(),
        axis.ticks.y = element_blank())

scale_color <- ggplot(df, aes(x, color = factor(x), fill = factor(x))) + 
  geom_point(size = 5, shape = 22, y = 1) + 
  scale_y_continuous(limits = c(0.8, 1.2), expand = c(0, 0), breaks = 1, label = "color  ") +
  scale_x_continuous(limits = c(.7, 4.4), breaks = NULL) +
  scale_color_manual(values = darken(c("#0082A6", "#4EBBB9", "#9CDFC2", "#D8F0CD"), .1), guide = "none") +
  scale_fill_manual(values = c("#0082A6", "#4EBBB9", "#9CDFC2", "#D8F0CD"), guide = "none") +
  theme_dviz_grid() +
  theme(axis.ticks.length = grid::unit(0, "pt"),
        axis.text.x = element_blank(),
        axis.text.y = element_text(size = 14),
        axis.title = element_blank(),
        axis.ticks = element_blank(),
        panel.grid.major = element_blank()) 

scale_shape <- ggplot(df, aes(x, shape = factor(x))) + 
  geom_point(size = 4, color = "grey30", y = 1, fill = "grey80") + 
  scale_y_continuous(limits = c(0.8, 1.2), expand = c(0, 0), breaks = 1, label = "shape  ") +
  scale_x_continuous(limits = c(.7, 4.4), breaks = NULL) +
  scale_shape_manual(values = 21:24, guide = "none") +
  theme_dviz_grid() +
  theme(axis.ticks.length = grid::unit(0, "pt"),
        axis.text.x = element_blank(),
        axis.text.y = element_text(size = 14),
        axis.title = element_blank(),
        axis.ticks = element_blank(),
        panel.grid.major = element_blank()) 

scale_num + scale_shape + scale_color + plot_layout(ncol = 1)
```

Let's put things into practice. We can take the dataset shown in Table \@ref(tab:data-example), map temperature onto the *y* axis, day of the year onto the *x* axis, location onto color, and visualize these aesthetics with solid lines. The result is a standard line plot showing the temperature normals at the four locations as they change during the year (Figure 
\@ref(fig:temp-normals-vs-time)).

(ref:temp-normals-vs-time) Daily temperature normals for four selected locations in the U.S. Temperature is mapped to the *y* axis, day of the year to the *x* axis, and location to line color. Data source: NOAA.

```{r temp-normals-vs-time, fig.cap = '(ref:temp-normals-vs-time)'}
temps_long <- filter(ncdc_normals,
                station_id %in% c(
                  "USW00014819", # Chicago, IL 60638
                  #"USC00516128", # Honolulu, HI 96813
                  #"USW00027502", # Barrow, AK 99723, coldest point in the US
                  "USC00042319", # Death Valley, CA 92328 hottest point in the US
                  "USW00093107", # San Diego, CA 92145
                  #"USC00427606"  # Salt Lake City, UT 84103
                  "USW00012918" # Houston, TX 77061
                )) %>%
  mutate(location = fct_recode(factor(station_id),
                               "Chicago" = "USW00014819",
                               #"Honolulu, HI" = "USC00516128",
                               #"Barrow, AK" = "USW00027502",
                               "Death Valley" = "USC00042319",
                               "San Diego" = "USW00093107",
                               #"Salt Lake City, UT" = "USC00427606",
                               "Houston" = "USW00012918")) %>%
  mutate(location = factor(location, levels = c("Death Valley", "Houston", "San Diego", "Chicago")))

ggplot(temps_long, aes(x = date, y = temperature, color = location)) +
  geom_line(size = 1) +
  scale_x_date(name = "month", limits = c(ymd("0000-01-01"), ymd("0001-01-04")),
               breaks = c(ymd("0000-01-01"), ymd("0000-04-01"), ymd("0000-07-01"),
                          ymd("0000-10-01"), ymd("0001-01-01")),
               labels = c("Jan", "Apr", "Jul", "Oct", "Jan"), expand = c(1/366, 0)) + 
  scale_y_continuous(limits = c(19.9, 107),
                     breaks = seq(20, 100, by = 20),
                     name = "temperature (°F)") +
  scale_color_OkabeIto(order = c(1:3, 7), name = NULL) +
  theme_dviz_grid() +
  theme(legend.title.align = 0.5)
```

Figure \@ref(fig:temp-normals-vs-time) is a fairly standard visualization for a temperature curve and likely the visualization most data scientists would intuitively choose first. However, it is up to us which variables we map onto which scales. For example, instead of mapping temperature onto the *y* axis and location onto color, we can do the opposite. Because now the key variable of interest (temperature) is shown as color, we need to show sufficiently large colored areas for the color to convey useful information [@Stone_et_al_2014]. Therefore, for this visualization I have chosen squares instead of lines, one for each month and location, and I have colored them by the average temperature normal for each month (Figure \@ref(fig:four-locations-temps-by-month)).

(ref:four-locations-temps-by-month) Monthly normal mean temperatures for four locations in the U.S. Data source: NOAA 

```{r four-locations-temps-by-month, fig.width = 5.5*6/4.2, fig.asp = .3, fig.cap = '(ref:four-locations-temps-by-month)'}
month_names <- c("01" = "Jan", "02" = "Feb", "03" = "Mar", "04" = "Apr", "05" = "May", "06" = "Jun",
                   "07" = "Jul", "08" = "Aug", "09" = "Sep", "10" = "Oct", "11" = "Nov", "12" = "Dec")


mean_temps <- temps_long %>%
  group_by(location, month) %>%
  summarize(mean = mean(temperature)) %>%
  ungroup() %>%
  mutate(month = month_names[month]) %>%
  mutate(month = factor(month, levels = unname(month_names)))

p <- ggplot(mean_temps, aes(x = month, y = location, fill = mean)) + 
  geom_tile(width = .95, height = 0.95) + 
  scale_fill_viridis_c(option = "B", begin = 0.15, end = 0.98,
                       name = "temperature (°F)") + 
  scale_y_discrete(name = NULL) +
  coord_fixed(expand = FALSE) +
  theme_dviz_open() +
  theme(axis.line = element_blank(),
        axis.ticks = element_blank(),
        #axis.text.y = element_text(size = 14),
        legend.title = element_text(size = 12)
        )

# fix legend (make it centered)
ggdraw(align_legend(p))
```

I would like to emphasize that Figure \@ref(fig:four-locations-temps-by-month) uses two position scales (month along the *x* axis and location along the *y* axis) but neither is a continuous scale. Month is an ordered factor with 12 levels and location is an unordered factor with four levels. Therefore, the two position scales are both discrete. For discrete position scales, we generally place the different levels of the factor at an equal spacing along the axis. If the factor is ordered (as is here the case for month), then the levels need to placed in the appropriate order. If the factor is unordered (as is here the case for location), then the order is arbitrary, and we can choose any order we want. I have ordered the locations from overall coldest (Chicago) to overall hottest (Death Valley) to generate a pleasant staggering of colors. However, I could have chosen any other order and the figure would have been equally valid.

Both Figures \@ref(fig:temp-normals-vs-time) and \@ref(fig:four-locations-temps-by-month) used three scales in total, two position scales and one color scale. This is a typical number of scales for a basic visualization, but we can use more than three scales at once. Figure \@ref(fig:mtcars-five-scale) uses five scales, two position scales, one color scale, one size scale, and one shape scale, and all scales represent a different variable from the dataset.


(ref:mtcars-five-scale) Fuel efficiency versus displacement, for 32 cars (1973--74 models). This figure uses five separate scales to represent data: (i) the *x* axis (displacement); (ii) the *y* axis (fuel efficiency); (iii) the color of the data points (power); (iv) the size of the data points (weight); and (v) the shape of the data points (number of cylinders). Four of the five variables displayed (displacement, fuel efficiency, power, and weight) are numerical continuous. The remaining one (number of cylinders) can be considered to be either numerical discrete or qualitative ordered. Data source: *Motor Trend*, 1974.

```{r mtcars-five-scale, fig.width = 6, fig.asp = .8, fig.cap = '(ref:mtcars-five-scale)'}
p_mtcars <- ggplot(mtcars, aes(disp, mpg, fill = hp, shape = factor(cyl), size = wt)) + 
  geom_point(color = "white") +
  scale_shape_manual(values = c(23, 24, 21), name = "cylinders") +
  scale_fill_continuous_sequential(
    palette = "Emrld", name = "power (hp)",
    breaks = c(100, 200, 300),
    rev = FALSE
  ) +
  xlab("displacement (cu. in.)") +
  ylab("fuel efficiency (mpg)") +
  guides(
    shape = guide_legend(override.aes = list(size = 4, fill = "#329D84")),
    size = guide_legend(override.aes = list(shape = 21, fill = "#329D84"),
    title = "weight (1000 lbs)")
  ) +
  theme_dviz_open() + background_grid() +
  theme(
    #legend.title = element_text(size = 12),
    legend.box.background = element_rect(fill = "white", color = "white"),
    legend.position = "top",
    legend.direction = "vertical",
    legend.justification = "center",
    legend.box.margin = margin(7, 7, 7, 7)
  )

legend <- get_legend(align_legend(p_mtcars))

ggdraw() + 
  draw_plot(p_mtcars + theme(legend.position = "none")) + 
  draw_grob(legend, x = .36, y = .7, width = .7, height = .3)
```



<!--chapter:end:aesthetic_mapping.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")
```

# Annotated bibliography {#bibliography}

No single book can cover everything there is to know about a topic. I encourage you to read other texts on data visualization to deepen your understanding and to develop your technical skills in making figures. Here, I provide a limited selection of books that I have personally found interesting, thought provoking, or helpful. Books listed in Section \@ref(bibliography-thinking) are the most similar in scope to the present book, and may provide complementary or alternative perspectives on the topics I have covered. Books listed in Section \@ref(bibliography-programming) address the important topic of how to make visualizations using programming approaches and available software libraries. The remaining sections list other books that will expand your knowledge of data visualization and help you communicate with visuals and data.

## Thinking about data and visualization {#bibliography-thinking}

The following books discuss the thought processes and decision making required for turning data into visualizations. They serve as introductory texts into how to choose what visualizations to make and what pitfalls to look out for. 

* Alberto Cairo. *"The Truthful Art."* New Riders, 2016.  
*Excellent all-around introduction to data visualization, in particular for journalists. The book covers many important concepts of data visualization, such as how to visualize distributions, trends, uncertainty, and maps. In many chapters, the book also serves as an introduction to basic statistical principles, explaining concepts such as population, sample, and confidence level.* 

* Stephen Few. *"Show Me the Numbers."* Analytics Press, 2012.  
*A book about data visualization for the business professional. It is similar in scope and target audience to the book by Nussbaumer Knaflic (see below). However, Few’s book contains more material and covers many topics in more depth. At the same time, the book is not as well written and as carefully produced as the Nussbaumer Knaflic book.*

* Cole Nussbaumer Knaflic. *"Storytelling with Data."* John Wiley & Sons, 2015.  
*A well written and carefully produced book on how to turn data into visuals. The book’s primary audience are people making business graphics, and the book is excellent for the topics it covers. However, the book is missing many topics of importance to scientists, such as the visualization of distributions, trends, or uncertainty.*


## Programming books {#bibliography-programming}

The following books are all how-to books that teach programming approaches to data visualization.

* Kieran Healy. *"Data Visualization: A Practical Introduction."* Princeton University Press, 2018.  
*Introduction to using ggplot2 for data visualization. Recommended as follow-up after Wickham and Grolemund's "R for Data Science" (see below).*

* Scott Murray. *"Interactive Data Visualization for the Web: An Introduction to Designing with D3, 2nd Edition."* O'Reilly Media, 2017.  
*Introduction to making interactive online visualizations with D3, using HTML, CSS, JavaScript, and SVG.*

* Jake VanderPlas. *"Python Data Science Handbook: Essential Tools for Working with Data."* O'Reilly Media, 2016.  
*Introduction to using the programming language Python for data science. Has extensive material on data visualization using Python's Matplotlib and Seaborn.*

* Hadley Wickham, Garrett Grolemund. *"R for Data Science."* O'Reilly Media, 2017.  
*All-around introduction to using the programming language R for data science. Contains several chapters on using ggplot2 for data visualization.*


## Statistics texts

Introductory texts in statistics will generally contain material on data visualization, covering topics such as scatter plots, histograms, box plots, and line graphs. There are many such texts that could be listed. Here, I mention just a few recent additions that are worth a closer look.

* David M. Diez, Christopher D. Barr, Mine Çetinkaya-Rundel. *"OpenIntro Statistics, 3rd Edition."* OpenIntro, Inc., 2015.  
*Open source introductory statistics text book. The entire book is freely available, as are the LaTeX files and R code used to compile the book and make the figures.*

* Susan Holmes, Wolfgang Huber. *"Modern Statistics for Modern Biology."* 2018.  
*A statistics text that emphasizes computational tools needed for modern biology. The entire book is freely available, and R code for all examples is provided.*


## Historical texts

The books in this section are of interest primarily for historical reasons. They were influential at the time of their publication, but similar material can now be found elsewhere or in more modern form.


* William S. Cleveland. *"Visualizing Data."* Hobart Press, 1993.  
*Companion book to "The Elements of Graphing Data" by the same author (see below). This one is more mathematical and doesn't talk about human perception.*

* William S. Cleveland. *"The Elements of Graphing Data, 2nd Edition."* Hobart Press, 1994.  
*One of the first books about information design written for statisticians. The book contains many examples of scatter plots, line graphs, histograms, and boxplots, and it discusses them in the context of data analysis and statistical modeling. It also popularized the Cleveland dot plot.*

* Edward R. Tufte. *"Envisioning Information."* Graphics Press, 1990.  
*This book popularized the concept of the small multiple.*

* Edward R. Tufte. *"The Visual Display of Quantitative Information, 2nd Edition."* Graphics Press, 2001.  
*First published in 1983, this book has been highly influential in the field of data visualization. It introduced concepts such as chart junk, data-to-ink ratio, and sparklines. The book also showed the first slopegraph (but didn't name it). However, the book also contains numerous recommendations that have not stood the test of time. In particular, it recommends an excessively minimalistic plot design.*


## Books on broadly related topics

The following books are all broadly related to the topics of data visualization and effective communication. 

* Joshua Schimel. *"Writing Science."* Oxford University Press, 2011.  
*Teaches how to write scientific and other technical topics in an engaging way, by telling a story. While not primarily a book about data visualization, this is an indispensable text for anybody who needs to write technical articles and/or proposals.*

* Jonathan Schwabish. *"Better Presentations."* Columbia University Press, 2016.  
*Short and informative guide for making presentations. A must-read for anybody who routinely uses slides to give talks or presentations.*

* Maureen C. Stone. *"A Field Guide to Digital Color."* A K Peters, 2003.  
*Comprehensive guide to how colors are captured, processed, and reproduced by computers.*

* Colin Ware. *"Information Visualization, 3rd Edition."* Morgan Kaufmann, 2012.  
*A book about principles of visualization, specifically addressing topics such as how the human visual system works and how different graphical patterns are perceived. The book covers many different visualization scenarios, including user interfaces and virtual worlds, but it puts comparatively less emphasis on visualizing data in the form of 2D figures.*


<!--chapter:end:annotated_bibliography.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(forcats)
library(ggridges) # for geom_density_line
```


# Avoid line drawings {#avoid-line-drawings}

Whenever possible, visualize your data with solid, colored shapes rather than with lines that outline those shapes. Solid shapes are more easily perceived as coherent objects, are less likely to create visual artifacts or optical illusions, and do more immediately convey amounts than do outlines. In my experience, visualizations using solid shapes are both clearer and more pleasant to look at than equivalent versions that use line drawings. Thus, I avoid line drawings as much as possible. However, I want to emphasize that this recommendation does not supersede the principle of proportional ink (Chapter \@ref(proportional-ink)).

Line drawings have a long history in the field of data visualization because throughout most of the 20th century, scientific visualizations were drawn by hand and had to be reproducible in black-and-white. This precluded the use of areas filled with solid colors, including solid gray-scale fills. Instead, filled areas were sometimes simulated by applying hatch, cross-hatch, or stipple patterns. Early plotting software imitated the hand-drawn simulations and similarly made extensive use of line drawings, dashed or dotted line patterns, and hatching. While modern visualization tools and modern reproduction and publishing platforms have none of the earlier limitations, many plotting applications still default to outlines and empty shapes rather than filled areas. To raise your awareness of this issue, here I'll show you several examples of the same figures drawn with both lines and filled shapes.

The most common and at the same time most inappropriate use of line drawings is seen in histograms and bar plots. The problem with bars drawn as outlines is that it is not immediately apparent which side of any given line is inside a bar and which side is outside. As a consequence, in particular when there are gaps between bars, we end up with a confusing visual pattern that detracts from the main message of the figure (Figure \@ref(fig:titanic-ages-lines)). Filling the bars with a light color, or with gray if color reproduction is not possible, avoids this problem (Figure \@ref(fig:titanic-ages-filled)).

(ref:titanic-ages-lines) Histogram of the ages of Titanic passengers, drawn with empty bars. The empty bars create a confusing visual pattern. In the center of the histogram, it is difficult to tell which parts are inside of bars and which parts are outside. 

```{r titanic-ages-lines, fig.cap='(ref:titanic-ages-lines)'}
titanic <- titanic_all
age_hist_3 <- data.frame(age = (1:25)*3-1.5, 
                         count = hist(titanic$age, breaks=(0:25)*3 + .01, plot = FALSE)$counts)

h3_bad <- ggplot(age_hist_3, aes(x = age, y = count)) + geom_col(width = 2., fill = "transparent",
                                                             color = "black")  + 
  scale_y_continuous(limits = c(0, 86), expand = c(0, 0), breaks = 25*(0:5)) +
  scale_x_continuous(limits = c(0, 75), expand = c(0, 0)) +
  theme_dviz_open() +
  background_grid(major = "y", minor = "none") +
  theme(plot.margin = margin(3, 7, 3, 1.5))

stamp_bad(h3_bad)
```

(ref:titanic-ages-filled) The same histogram of Figure \@ref(fig:titanic-ages-lines), now drawn with filled bars. The shape of the age distribution is much more easily discernible in this variation of the figure.

```{r titanic-ages-filled, fig.cap='(ref:titanic-ages-filled)'}
h3_good <- ggplot(age_hist_3, aes(x = age, y = count)) + geom_col(width = 2.7, fill = "#56B4E9")  + 
  scale_y_continuous(limits = c(0, 86), expand = c(0, 0), breaks = 25*(0:5)) +
  scale_x_continuous(limits = c(0, 75), expand = c(0, 0)) +
  theme_dviz_open() +
  background_grid(major = "y", minor = "none") +
  theme(plot.margin = margin(3, 7, 3, 1.5))

h3_good
```

Next, let's take a look at an old-school density plot. I'm showing density estimates for the sepal-length distributions of three species of iris, drawn entirely in black-and-white as a line drawing (Figure \@ref(fig:iris-densities-lines)). The distributions are shown just by their outlines, and because the figure is in black-and-white, we're using different line styles to distinguish them. This figure has two main problems. First, the dashed line styles do not provide a clear separation between the area under the curve and the area above it. While our visual system is quite good at connecting the individual line elements into a continuous line, the dashed lines nevertheless look porous and do not serve as a strong boundary of the enclosed area. Second, because the lines intersect and the areas they enclose are not shaded, it is difficult to segment the different densities from the six distinct shape outlines. This effect would have been even stronger had I used solid rather than dashed lines for all three distributions.

(ref:iris-densities-lines) Density estimates of the sepal lengths of three different iris species. The broken line styles used for versicolor and virginica detract from the perception that the areas under the curves are distinct from the areas above them.

```{r iris-densities-lines, fig.cap='(ref:iris-densities-lines)'}
# compute densities for sepal lengths
iris_dens <- group_by(iris, Species) %>%
  do(ggplot2:::compute_density(.$Sepal.Length, NULL)) %>%
  rename(Sepal.Length = x)

# get the maximum values
iris_max <- filter(iris_dens, density == max(density)) %>%
  ungroup() %>%
  mutate(
    hjust = c(0, 0.4, 0),
    vjust = c(1, 0, 1),
    nudge_x = c(0.11, 0, 0.24),
    nudge_y = c(-0.02, 0.03, -0.02) 
  )

iris_lines <- ggplot(iris_dens, aes(x = Sepal.Length, y = density, linetype = Species)) + 
  geom_density_line(stat = "identity", alpha = 0., size = 0.75) +
  geom_text(
    data = iris_max,
    aes(
      label = paste0("Iris ", Species),
      hjust = hjust, vjust = vjust,
      x = Sepal.Length + nudge_x, 
      y = density + nudge_y
    ),
    family = dviz_font_family,
    fontface = "italic",
    inherit.aes = FALSE, size = 12/.pt
  ) +
  scale_linetype_manual(
    values = c(1, 3, 4),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_x_continuous(expand = c(0, 0), name = "sepal length") +
  scale_y_continuous(limits = c(0, 1.5), expand = c(0, 0)) +
  theme_dviz_open() +
  theme(plot.margin = margin(14, 7, 3, 1.5))

stamp_ugly(iris_lines)
```

We can attempt to address the problem of porous boundaries by using colored lines rather than dashed lines (Figure \@ref(fig:iris-densities-colored-lines)). However, the density areas in the resulting plot still have little visual presence. Overall, I find the version with filled areas (Figure \@ref(fig:iris-densities-filled)) the most clear and intuitive. It is important, however, to make the filled areas partially transparent, so that the complete distribution for each species is visible.

(ref:iris-densities-colored-lines) Density estimates of the sepal lengths of three different iris species. By using solid, colored lines we have solved the problem of Figure \@ref(fig:iris-densities-lines) that the areas below and above the lines seem to be connected. However, we still don't have a strong sense of the size of the area under each curve.

```{r iris-densities-colored-lines, fig.cap='(ref:iris-densities-colored-lines)'}
iris_colored_lines <- ggplot(iris_dens, aes(x = Sepal.Length, y = density, color = Species)) + 
  geom_density_line(stat = "identity", alpha = 0., size = 1) +
  geom_text(
    data = iris_max,
    aes(
      label = paste0("Iris ", Species),
      hjust = hjust, vjust = vjust,
      x = Sepal.Length + nudge_x, 
      y = density + nudge_y
    ),
    family = dviz_font_family,
    fontface = "italic",
    inherit.aes = FALSE, size = 12/.pt
  ) +
  scale_color_manual(
    values = c("#56B4E9", "#E69F00", "#009E73"),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_x_continuous(expand = c(0, 0), name = "sepal length") +
  scale_y_continuous(limits = c(0, 1.5), expand = c(0, 0)) +
  theme_dviz_open() +
  theme(plot.margin = margin(14, 7, 3, 1.5))

stamp_ugly(iris_colored_lines)
```


(ref:iris-densities-filled) Density estimates of the sepal lengths of three different iris species, shown as partially transparent shaded areas.

```{r iris-densities-filled, fig.cap='(ref:iris-densities-filled)'}
iris_filled <- ggplot(iris_dens, aes(x = Sepal.Length, y = density, fill = Species, color = Species)) + 
  geom_density_line(stat = "identity") +
  geom_text(
    data = iris_max,
    aes(
      label = paste0("Iris ", Species),
      hjust = hjust, vjust = vjust,
      x = Sepal.Length + nudge_x, 
      y = density + nudge_y
    ),
    family = dviz_font_family,
    fontface = "italic",
    inherit.aes = FALSE, size = 12/.pt
  ) +
  scale_color_manual(
    values = darken(c("#56B4E9", "#E69F00", "#009E73"), 0.3),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_fill_manual(
    values = c("#56B4E950", "#E69F0050", "#009E7350"),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_x_continuous(expand = c(0, 0), name = "sepal length") +
  scale_y_continuous(limits = c(0, 1.5), expand = c(0, 0)) +
  theme_dviz_open() +
  theme(plot.margin = margin(14, 7, 3, 1.5))

iris_filled
```

Line drawings also arise in the context of scatter plots, when different point types are drawn as open circles, triangles, crosses, etc. As an example, consider Figure \@ref(fig:mpg-linespoints). The figure contains a lot of visual noise, and the different point types do not strongly separate from each other. Drawing the same figure with solidly colored shapes addresses this issue (Figure \@ref(fig:mpg-filledpoints)).

(ref:mpg-linespoints) City fuel economy versus engine displacement, for cars with front-wheel drive (FWD), rear-wheel drive (RWD), and all-wheel drive (4WD). The different point styles, all black-and-white line-drawn symbols, create substantial visual noise and make it difficult to read the figure.

```{r mpg-linespoints, fig.width=5.5, fig.asp=.7416, fig.cap='(ref:mpg-linespoints)'}
mpg_linespoints <- ggplot(mpg, aes(y=cty, x=displ, shape=drv)) +
  geom_point(
    size = 2,
    position = position_jitter(
      width = 0.01 * diff(range(mpg$displ)),
      height = 0.01 * diff(range(mpg$cty)),
      seed = 7384
    )
  ) + 
  ylab("fuel economy (mpg)") +
  xlab("displacement (l)") +
  scale_shape_manual(
    values=c(4, 1, 2), 
    name="drive train",
    breaks=c("f", "r", "4"),
    labels=c("FWD", "RWD", "4WD")
  ) +
  theme_dviz_open() +
  theme(
    legend.position = c(.7, .8),
    plot.margin = margin(3, 7, 3, 1.5)
  )

stamp_ugly(mpg_linespoints)
```


(ref:mpg-filledpoints) City fuel economy versus engine displacement. By using both different colors and different solid shapes for the different drive-train variants, this figure clearly separates the drive-train variants while remaining reproducible in gray scale if needed.

```{r mpg-filledpoints, fig.width=5.5, fig.asp=.7416, fig.cap='(ref:mpg-filledpoints)'}
mpg_filledpoints <- ggplot(mpg, aes(y = cty, x = displ, color = drv, fill = drv, shape = drv)) +
  geom_point(
    size = 3,
    position = position_jitter(
      width = 0.01 * diff(range(mpg$displ)),
      height = 0.01 * diff(range(mpg$cty)),
      seed = 7384
    )
  ) + 
  ylab("fuel economy (mpg)") +
  xlab("displacement (l)") +
  scale_color_manual(
    values = c("#202020", "#E69F00", "#56B4E9"), 
    name = "drive train",
    breaks = c("f", "r", "4"),
    labels = c("FWD", "RWD", "4WD")
  ) +
  scale_fill_manual(
    values = c("#20202080", "#E69F0080", "#56B4E980"), 
    name = "drive train",
    breaks = c("f", "r", "4"),
    labels = c("FWD", "RWD", "4WD")
  ) +
  scale_shape_manual(
    values = c(22, 21, 25), 
    name = "drive train",
    breaks = c("f", "r", "4"),
    labels = c("FWD", "RWD", "4WD")
  ) +
  theme_dviz_open() +
  theme(
    legend.position = c(.7, .8),
    plot.margin = margin(3, 7, 3, 1.5)
  )
        
mpg_filledpoints
```

I strongly prefer solid points over open points, because the solid points have much more visual presence. The argument that I sometimes hear in favor of open points is that they help with overplotting, since the empty areas in the middle of each point allow us to see other points that may be lying underneath. In my opinion, the benefit from being able to see overplotted points does not, in general, outweigh the detriment from the added visual noise of open symbols. There are other approaches for dealing with overplotting, see Chapter \@ref(overlapping-points) for some suggestions.

Finally, let's consider boxplots. Boxplots are commonly drawn with empty boxes, as in Figure \@ref(fig:lincoln-weather-box-empty). I prefer a light shading for the box, as in Figure \@ref(fig:lincoln-weather-box-filled). The shading separates the box more clearly from the plot background, and it helps in particular when we're showing many boxplots right next to each other, as is the case in Figures \@ref(fig:lincoln-weather-box-empty) and  \@ref(fig:lincoln-weather-box-filled). In Figure \@ref(fig:lincoln-weather-box-empty), the large number of boxes and lines can again create the illusion of background areas outside of boxes being actually on the inside of some other shape, just as we saw in Figure \@ref(fig:titanic-ages-lines). This problem is eliminated in Figure \@ref(fig:lincoln-weather-box-filled). I have sometimes heard the critique that shading the inside of the box gives too much weight to the center 50% of the data, but I don't buy that argument. It is inherent to the boxplot, shaded box or not, to give more weight to the center 50% of the data than to the rest. If you don't want this emphasis, then don't use a boxplot. Instead, use a violin plot, jittered points, or a sina plot (Chapter \@ref(boxplots-violins)).

(ref:lincoln-weather-box-empty) Distributions of daily mean temperatures in Lincoln, Nebraska, in 2016. Boxes are drawn in the traditional way, without shading.

```{r lincoln-weather-box-empty, fig.cap='(ref:lincoln-weather-box-empty)'}
lincoln_weather %>% 
  mutate(
    month_short = fct_recode(
      Month,
      Jan = "January",
      Feb = "February",
      Mar = "March",
      Apr = "April",
      May = "May",
      Jun = "June",
      Jul = "July",
      Aug = "August",
      Sep = "September",
      Oct = "October",
      Nov = "November",
      Dec = "December"
    )
  ) %>%
  mutate(month_short = fct_rev(month_short)) -> lincoln_df

lincoln_box_empty <- ggplot(lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)) +
  geom_boxplot() + 
  xlab("Month") +
  ylab("mean temperature (°F)") +
  theme_dviz_open() +
  theme(plot.margin = margin(3, 7, 3, 1.5))

lincoln_box_empty
```


(ref:lincoln-weather-box-filled) Distributions of daily mean temperatures in Lincoln, Nebraska, in 2016. By giving the boxes a light gray shading, we can make them stand out better against the background.

```{r lincoln-weather-box-filled, fig.cap='(ref:lincoln-weather-box-filled)'}
lincoln_box_filled <- ggplot(lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)) +
  geom_boxplot(fill = 'grey90') + 
  xlab("Month") +
  ylab("mean temperature (°F)") +
  theme_dviz_open() +
  theme(plot.margin = margin(3, 7, 3, 1.5))

lincoln_box_filled
```

<!--chapter:end:avoid_line_drawings.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(readr)
library(lubridate)
library(ggrepel)
```

# Balance the data and the context {#balance-data-context}

We can broadly subdivide the graphical elements in any visualization into elements that represent data and elements that do not. The former are elements such as the points in a scatter plot, the bars in a histogram or barplot, or the shaded areas in a heatmap. The latter are elements such as plot axes, axis ticks and labels, axis titles, legends, and plot annotations. These elements generally provide context for the data and/or visual structure to the plot. When designing a plot, it can be helpful to think about the amount of ink (Chapter \@ref(proportional-ink)) used to represent data and context. A common recommendation is to reduce the amount of non-data ink, and following this advice can often yield less cluttered and more elegant visualizations. At the same time, context and visual structure are important, and overly minimizing the plot elements that provide them can result in figures that are difficult to read, confusing, or simply not that compelling.

## Providing the appropriate amount of context

The idea that distinguishing between data and non-data ink may be useful was popularized by Edward Tufte in his book "The Visual Display of Quantitative Information" [@TufteQuantDispl]. Tufte introduces the concept of the "data--ink ratio", which he defines as the "proportion of a graphic's ink devoted to the non-redundant display of data-information." He then writes (emphasis mine):

> Maximize the data--ink ratio, *within reason.*

I have emphasized the phrase "within reason" because it is critical and frequently forgotten. In fact, I think that Tufte himself forgets it in the remainder of his book, where he advocates overly minimalistic designs that, in my opinion, are neither elegant nor easy to decipher. If we interpret the phrase "maximize the data--ink ratio" to mean "remove clutter and strive for clean and elegant designs," then I think it is reasonable advice. But if we interpret it as "do everything you can to remove non-data ink" then it will result in poor design choices. If we go too far in either direction we will end up with ugly figures. However, away from the extremes there is wide range of designs that are all acceptable and may be appropriate in different settings.

To explore the extremes, let's consider a figure that clearly has too much non-data ink (Figure \@ref(fig:Aus-athletes-grid-bad)). The colored points in the plot panel (the framed center area containing data points) are data ink. Everything else is non-data ink. The non-data ink includes a frame around the entire figure, a frame around the plot panel, and a frame around the legend. None of these frames are needed. We also see a prominent and dense background grid that draws attention away from the actual data points. By removing the frames and minor grid lines and by drawing the major grid lines in a light gray, we arrive at Figure \@ref(fig:Aus-athletes-grid-good). In this version of the figure, the actual data points stand out much more clearly, and they are perceived as the most important component of the figure.

(ref:Aus-athletes-grid-bad) Percent body fat versus height in professional male Australian athletes. Each point represents one athlete. This figure devotes way too much ink to non-data. There are unnecessary frames around the entire figure, around the plot panel, and around the legend. The coordinate grid is very prominent, and its presence draws attention away from the data points. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-grid-bad, fig.cap='(ref:Aus-athletes-grid-bad)'}
male_Aus <- filter(Aus_athletes, sex=="m") %>%
    filter(sport %in% c("basketball", "field", "swimming", "track (400m)",
                        "track (sprint)", "water polo")) %>%
    mutate(sport = case_when(sport == "track (400m)" ~ "track",
                             sport == "track (sprint)" ~ "track",
                             TRUE ~ sport))

male_Aus$sport <- factor(male_Aus$sport,
                         levels = c("field", "water polo", "basketball", "swimming", "track"))

p_Aus_base <- ggplot(male_Aus, aes(x=height, y=pcBfat, color=sport, fill=sport, shape=sport)) +
  geom_point(size = 2.5) +
  scale_shape_manual(values = 21:25) +
  scale_color_OkabeIto(order=c(2, 1, 3, 4, 5), darken = 0.3) +
  scale_fill_OkabeIto(order=c(2, 1, 3, 4, 5), darken = 0.1, alpha = 0.7) +
  scale_x_continuous(limits = c(169, 210), name = "height (cm)") +
  scale_y_continuous(limits = c(5, 20), name = "% body fat")

p <- p_Aus_base +
  theme_dviz_open() +
  theme(
    axis.line = element_blank(),
    panel.grid.major = element_line(color = "black", size = 0.3),
    panel.grid.minor = element_line(color = "black", size = 0.15),
    panel.border = element_rect(color = "black", size = 1),
    legend.background = element_rect(color = "black", size = 0.5),
    legend.margin = margin(7, 7, 7, 7),
    plot.background = element_rect(color = "black", size = 1),
    plot.margin = margin(7, 7, 7, 7)
  )

# width: 6 in = 433.62pt
# height: 0.618 * 6 in = 3.71 in = 268.12

stamp_ugly(ggdraw() + draw_plot(p, x = 1/434, y = 1/268, width = 426/434, height = 252/268))
```

(ref:Aus-athletes-grid-good) Percent body fat versus height in professional male Australian athletes. This figure is a cleaned-up version of Figure \@ref(fig:Aus-athletes-grid-bad). Unnecessary frames have been removed, minor grid lines have been removed, and major grid lines have been drawn in light gray to stand back relative to the data points. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-grid-good, fig.cap='(ref:Aus-athletes-grid-good)'}
p_Aus_base + 
  theme_dviz_grid()# + panel_border(colour = "grey70") + theme(axis.ticks = element_line(colour = "grey70"))
```

At the other extreme, we might end up with a figure such as Figure \@ref(fig:Aus-athletes-min-bad), which is a minimalist version of Figure \@ref(fig:Aus-athletes-grid-good). In this figure, the axis tick labels and titles have been made so faint that they are hard to see. If we just glance at the figure we will not immediately perceive what data is actually shown. We only see points floating in space. Moreover, the legend annotations are so faint that the points in the legend could be mistaken for data points. This effect is amplified because there is no clear visual separation between the plot area and the legend. Notice how the background grid in Figure \@ref(fig:Aus-athletes-grid-good) both anchors the points in space and sets off the data area from the legend area. Both of these effects have been lost in Figure \@ref(fig:Aus-athletes-min-bad).

(ref:Aus-athletes-min-bad) Percent body fat versus height in professional male Australian athletes. In this example, the concept of removing non-data ink has been taken too far. The axis tick labels and title are too faint and are barely visible. The data points seem to float in space. The points in the legend are not sufficiently set off from the data points, and the casual observer might think they are part of the data. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-min-bad, fig.cap='(ref:Aus-athletes-min-bad)'}
axis_col <- "grey70"
title_col <- "grey70"

p <- p_Aus_base + theme_dviz_open(11) +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_line(color = axis_col),
    axis.text = element_text(color = axis_col),
    axis.title = element_text(color = title_col),
    legend.text = element_text(color = title_col),
    legend.title = element_text(color = title_col),
    plot.margin = margin(3.5, 7, 3.5, 1.5)
  )

stamp_ugly(p)
```

In Figure \@ref(fig:Aus-athletes-grid-good), I am using an open background grid and no axis lines or frame around the plot panel. I like this design because it conveys to the reader that range of possible data values extends beyond the axis limits. For example, even though Figure \@ref(fig:Aus-athletes-grid-good) shows no athlete taller than 210 cm, such an athlete could conceivably exist. However, some authors prefer to delineate the extent of the plot panel, by drawing a frame around it (Figure \@ref(fig:Aus-athletes-grid-good-frame)). Both options are reasonable, and which is preferable is primarily a matter of personal opinion. One advantage of the framed version is that it clearly separates the legend from the plot panel.

(ref:Aus-athletes-grid-good-frame) Percent body fat versus height in professional male Australian athletes. This figure adds a frame around the plot panel of Figure \@ref(fig:Aus-athletes-grid-good), and this frame helps separate the legend from the data. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-grid-good-frame, fig.cap='(ref:Aus-athletes-grid-good-frame)'}
p_Aus_base + 
  theme_dviz_grid() + panel_border(colour = "grey70") + theme(axis.ticks = element_line(colour = "grey70"))
```


Figures with too little non-data ink commonly suffer from the effect that figure elements appear to float in space, without clear connection or reference to anything. This problem tends to be particularly severe in small multiples plots. Figure \@ref(fig:titanic-survival-by-gender-class-bad) shows a small-multiples plot comparing six different bar plots, but it looks more like a piece of modern art than a useful data visualization. The bars are not anchored to a clear baseline and the individual plot facets are not clearly delineated. We can resolve these issues by adding a light gray background and thin horizontal grid lines to each facet (Figure \@ref(fig:titanic-survival-by-gender-class)).

(ref:titanic-survival-by-gender-class-bad) Survival of passengers on the Titanic, broken down by gender and class. This small-multiples plot is too minimalistic. The individual facets are not framed, so it's difficult to see which part of the figure belongs to which facet. Further, the individual bars are not anchored to a clear baseline, and they seem to float.

```{r titanic-survival-by-gender-class-bad, fig.width = 5, fig.asp = 3/4, fig.cap = '(ref:titanic-survival-by-gender-class-bad)'}
titanic %>% mutate(surv = ifelse(survived == 0, "died", "survived")) %>%
  ggplot(aes(sex, fill = sex)) + geom_bar() +
    facet_grid(class ~ surv, scales = "free_x") +
    scale_x_discrete(name = NULL) + 
    scale_y_continuous(limits = c(0, 195), expand = c(0, 0)) +
    scale_fill_manual(values = c("#D55E00D0", "#0072B2D0"), guide = "none") +
    theme_dviz_open(rel_small = 1) +
    theme(axis.line.x = element_blank(),
          axis.ticks.x = element_blank(),
          #axis.text.x = element_text(margin = margin(7, 0, 0, 0)),
          strip.text = element_text(margin = margin(3.5, 3.5, 3.5, 3.5)),
          strip.background  = element_blank(),
          plot.margin = margin(3.5, 14, 3.5, 1.5)) -> p

stamp_ugly(p)
```


(ref:titanic-survival-by-gender-class) Survival of passengers on the Titanic, broken down by gender and class. This is an improved version of Figure \@ref(fig:titanic-survival-by-gender-class-bad). The gray background in each facet clearly delineates the six groupings (survived or died in first, second, or third class) that make up this plot. Thin horizontal lines in the background provide a reference for the bar heights and facility comparison of bar heights among facets.

```{r titanic-survival-by-gender-class, fig.width = 5, fig.asp = 3/4, fig.cap = '(ref:titanic-survival-by-gender-class)'}
colors <- desaturate(lighten(c("#56B4E9", "#56B4E9"), c(.8, -.4)), .8)

titanic %>% mutate(surv = ifelse(survived == 0, "died", "survived")) %>%
  ggplot(aes(sex, fill = sex)) + geom_bar() +
    facet_grid(class ~ surv, scales = "free_x") +
    scale_x_discrete(name = NULL) + 
    scale_y_continuous(limits = c(0, 195), expand = c(0, 0)) +
    scale_fill_manual(values = c("#D55E00D0", "#0072B2D0"), guide = "none") +
    theme_dviz_open(rel_small = 1) +
    theme(axis.line = element_blank(),
          #axis.line.y = element_line(color = colors[2]),
          axis.ticks = element_blank(),
          axis.ticks.length = grid::unit(0, "pt"),
          #axis.ticks.y = element_line(color = colors[2]),
          axis.text.x = element_text(margin = margin(7, 0, 3.5, 0)),
          axis.text.y = element_text(color = colors[2]),
          strip.text = element_text(margin = margin(3.5, 3.5, 3.5, 3.5)),
          strip.background  = element_blank(),
          panel.background = element_rect(fill = colors[1]),
          panel.grid.major.y = element_line(color = "white"),
          plot.margin = margin(3.5, 14, 3.5, 1.5)) -> p

p
```



## Background grids

Gridlines in the background of a plot can help the reader discern specific data values and compare values in one part of a plot to values in another part. At the same time, gridlines can add visual noise, in particular when they are prominent or densely spaced. Reasonable people can disagree about whether to use a grid or not, and if so how to format it and how densely to space it. Throughout this book I am using a variety of different grid styles, to highlight that there isn't necessarily one best choice.

The R software ggplot2 has popularized a style using a fairly prominent background grid of white lines on a gray background. Figure \@ref(fig:price-plot-ggplot-default) shows an example in this style. The figure displays the change in stock price of four major tech companies over a five-year window, from 2012 to 2017. With apologies to the ggplot2 author Hadley Wickham, for whom I have the utmost respect, I don't find the white-on-gray background grid particularly attractive. To my eye, the gray background can detract from the actual data, and a grid with major and minor lines can be too dense. I also find the gray squares in the legend confusing.

(ref:price-plot-ggplot-default) Stock price over time for four major tech companies. The stock price for each company has been normalized to equal 100 in June 2012. This figure mimics the ggplot2 default look, with white major and minor grid lines on a gray background. In this particular example, I think the grid lines overpower the data lines, and the result is a figure that is not well balanced and that doesn't place sufficient emphasis on the data. Data source: Yahoo Finance

```{r price-plot-ggplot-default, fig.cap='(ref:price-plot-ggplot-default)'}
price_plot <- ggplot(tech_stocks, aes(x=date, y=price_indexed, color=ticker)) +
  geom_line(size = 0.66, na.rm = TRUE) +
  scale_color_manual(
    values = c("#000000", "#E69F00", "#56B4E9", "#009E73"),
    name = NULL,
    breaks = c("FB", "GOOG", "MSFT", "AAPL"),
    labels = c("Facebook", "Alphabet", "Microsoft", "Apple")
  ) +
  scale_x_date(
    name="year",
    limits=c(ymd("2012-06-01"), ymd("2017-05-31")),
    expand=c(0,0)
  ) + 
  scale_y_continuous(
    name = "stock price, indexed",
    limits = c(0, 560),
    expand = c(0,0)
  ) +
  theme_dviz_grid() +
  theme(
    panel.background = element_rect(fill = "grey90"),
    panel.grid.major = element_line(color = "white"),
    panel.grid.minor = element_line(color = "white", size = rel(0.5)),
    legend.key = element_rect(fill = "grey90", color = "white"),
    axis.ticks = element_line(color = "black"),
    plot.margin = margin(7, 7, 3, 1.5)
  )

stamp_ugly(price_plot)
```

Arguments in favor of the gray background include that it (i) helps the plot to be perceived as a single visual entity and (ii) prevents the plot to appear as a white box in surrounding dark text [@Wickham2016]. I completely agree with the first point, and it was the reason I used gray backgrounds in Figure \@ref(fig:titanic-survival-by-gender-class). For the second point, I'd like to caution that the perceived darkness of text will depend on the font size, fontface, and line spacing, and the perceived darkness of a figure will depend on the absolute amount and color of ink used, including all data ink. A scientific paper typeset in dense, 10-point Times New Roman will look much darker than a coffee-table book typeset in 14-point Palatino with one-and-a-half line spacing. Likewise, a scatter plot of five points in yellow will look much lighter than a scatter plot of 10,000 points in black. If you want to use a gray figure background, consider the color intensity of your figure foreground, as well as the expected layout and typography of the text around your figures, and adjust the choice of your background gray accordingly. Otherwise, it could happen that your figures end up standing out as dark boxes among the surrounding lighter text. Also, keep in mind that the colors you use to plot your data need to work with the gray background. We tend to perceive colors differently against different backgrounds, and a gray background requires darker and more saturated foreground colors than a white background.

We can go all the way in the opposite direction and remove both the background and the grid lines (Figure \@ref(fig:price-plot-no-grid)). In this case, we need visible axis lines to frame the plot and keep it as a single visual unit. For this particular figure, I think this choice is a worse option, and I have labeled it as "bad". In the absence of any background grid whatsoever, the curves seem to float in space, and it's difficult to reference the final values on the right to the axis ticks on the left.

(ref:price-plot-no-grid) Indexed stock price over time for four major tech companies. In this variant of Figure \@ref(fig:price-plot-ggplot-default), the data lines are not sufficiently anchored. This makes it difficult to ascertain to what extent they have deviated from the index value of 100 at the end of the covered time interval. Data source: Yahoo Finance

```{r price-plot-no-grid, fig.cap='(ref:price-plot-no-grid)'}
stamp_bad(
  price_plot + 
    theme_dviz_open() +
    theme(plot.margin = margin(7, 7, 3, 1.5))
)
```

At the absolute minimum, we need to add one horizontal reference line. Since the stock prices in Figure \@ref(fig:price-plot-no-grid) indexed to 100 in June 2012, marking this value with a thin horizontal line at *y* = 100 helps a lot (Figure \@ref(fig:price-plot-refline)). Alternatively, we can use a minimal "grid" of horizontal lines. For a plot where we are primarily interested in the change in *y* values, vertical grid lines are not needed. Moreover, grid lines positioned at only the major axis ticks will often be sufficient. And, the axis line can be omitted or made very thin, since the horizontal lines clearly mark the extent of the plot (Figure \@ref(fig:price-plot-hgrid)).

(ref:price-plot-refline) Indexed stock price over time for four major tech companies. Adding a thin horizontal line at the index value of 100 to Figure \@ref(fig:price-plot-no-grid) helps provide an important reference throughout the entire time period the plot spans. Data source: Yahoo Finance

```{r price-plot-refline, fig.cap='(ref:price-plot-refline)'}
price_plot2 <- ggplot(tech_stocks, aes(x=date, y=price_indexed, color=ticker)) +
  geom_hline(yintercept = 100, size = 0.5, color="grey70") +
  geom_line(size = 0.66, na.rm = TRUE) +
  scale_color_manual(
    values = c("#000000", "#E69F00", "#56B4E9", "#009E73"),
    name = NULL,
    breaks = c("FB", "GOOG", "MSFT", "AAPL"),
    labels = c("Facebook", "Alphabet", "Microsoft", "Apple")
  ) +
  scale_x_date(
    name = "year",
    limits = c(ymd("2012-06-01"), ymd("2017-05-31")),
    expand = c(0,0)
  ) + 
  scale_y_continuous(
    name = "stock price, indexed",
    limits = c(0, 560),
    expand = c(0,0)
  ) +
  theme_dviz_open() +
  theme(plot.margin = margin(7, 7, 3, 1.5))

price_plot2
```


(ref:price-plot-hgrid) Indexed stock price over time for four major tech companies. Adding thin  horizontal lines at all major *y* axis ticks provides a better set of reference points than just the one horizontal line of Figure \@ref(fig:price-plot-refline). This design also removes the need for prominent *x* and *y* axis lines, since the evenly spaced horizontal lines create a visual frame for the plot panel. Data source: Yahoo Finance

```{r price-plot-hgrid, fig.cap='(ref:price-plot-hgrid)'}
price_plot + 
  theme_dviz_hgrid() +
  theme(plot.margin = margin(7, 7, 3, 1.5))
```

For such a minimal grid, we generally draw the lines orthogonally to direction along which the numbers of interest vary. Therefore, if instead of plotting the stock price over time we plot the five-year increase, as horizontal bars, then we will want to use vertical lines instead (Figure \@ref(fig:price-increase)).

(ref:price-increase) Percent increase in stock price from June 2012 to June 2017, for four major tech companies. Because the bars run horizontally, vertical grid lines are appropriate here. Data source: Yahoo Finance

```{r price-increase, fig.cap='(ref:price-increase)'}
perc_increase <- filter(tech_stocks, date == ymd("2017-06-01")) %>%
  mutate(perc=100*(price-index_price)/index_price,
         label=paste(as.character(round(perc)), "%", sep="")) %>%
  arrange(perc)

perc_increase$ticker <- factor(perc_increase$ticker, levels=perc_increase$ticker)

perc_plot <- ggplot(perc_increase, aes(x=ticker, y=perc)) +
  geom_col(fill="#56B4E9") +
  geom_text(aes(label=label), color="white", hjust=1.1, size=5,
            family = dviz_font_family) +
  scale_y_continuous(#name="percent increase\n(June 2012 to June 2017)",
                     name="percent increase",
                     limits=c(0, 499),
                     expand=c(0, 0)) +
  scale_x_discrete(name = NULL,
                     breaks = c("FB", "GOOG", "MSFT", "AAPL"),
                     labels = c("Facebook ", "Alphabet ", "Microsoft ", "Apple ")) +
  coord_flip(clip = "off") +
  theme_dviz_vgrid(rel_small = 1) +
  theme(axis.line.y = element_blank(),
        axis.ticks = element_blank(),
        axis.ticks.length = grid::unit(0, "pt"))

perc_plot
```


```{block type='rmdtip', echo=TRUE}
Grid lines that run perpendicular to the key variable of interest tend to be the most useful.
```

For bar graphs such as Figure \@ref(fig:price-increase), @TufteQuantDispl recommends to draw white grid lines on top of the bars instead of dark grid lines underneath (Figure \@ref(fig:price-increase-tufte)). These white grid lines have the effect of separating the bars into distinct segments of equal length. I'm of two minds on this style. On the one hand, research into human perception suggests that breaking bars into discrete segments helps the reader to perceive bar lengths [@Haroz_et_al_2015]. On the other hand, to my eye the bars look like they are falling apart and don't form a clear visual unit. In fact, I used this style purposefully in Figure \@ref(fig:titanic-passengers-by-class-sex) to visually separate stacked bars representing male and female passengers. Which effect dominates may depend on the specific choices of bar width, distance between bars, and thickness of the white grid lines. Thus, if you intend to use this style, I encourage you to vary these parameters until you have a figure that creates the desired visual effect.

(ref:price-increase-tufte) Percent increase in stock price from June 2012 to June 2017, for four major tech companies. White grid lines on top of bars can help the reader perceive the relative lengths of the bars. At the same time, they can also create the perception that the bars are falling apart. Data source: Yahoo Finance

```{r price-increase-tufte, fig.cap='(ref:price-increase-tufte)'}
perc_plot2 <- ggplot(perc_increase, aes(x = ticker, y = perc)) +
  geom_col(fill = "#56B4E9", width = 0.8) +
  geom_hline(yintercept = c(100, 200, 300, 400), color = "white", size = 1) +
  geom_text(
    aes(label = label), color = "black", hjust = -0.1, size = 5,
    family = dviz_font_family
  ) +
  scale_y_continuous(
    name = NULL,
    #name = "percent increase",
    limits = c(0, 496),
    expand = c(0, 0)
  ) +
  scale_x_discrete(
    name = NULL,
    breaks = c("FB", "GOOG", "MSFT", "AAPL"),
    labels = c("Facebook", "Alphabet", "Microsoft", "Apple")
  ) +
  coord_flip() +
  theme_dviz_vgrid(rel_small = 1) +
  theme(
    axis.ticks = element_blank(),
    axis.line = element_blank(),
    axis.text.x = element_blank(),
    axis.text.y = element_text(color = "black"),
    panel.grid.major = element_blank(),
    plot.margin = margin(3, 7, 3, 1.5)
  )

perc_plot2
```

I would like to point out another downside of Figure \@ref(fig:price-increase-tufte). I had to move the percentage values outside the bars, because the labels didn't fit into the final segments of several of the bars. However, this choice inappropriately visually elongates the bars and should be avoided whenever possible. 

Background grids along both axis directions are most appropriate for scatter plots where there is no primary axis of interest. Figure \@ref(fig:Aus-athletes-grid-good) at the beginning of this chapter provides an example. When a figure has a full background grid, axis lines are generally not needed (Figure \@ref(fig:Aus-athletes-grid-good)).

## Paired data

For figures where the relevant comparison is the *x* = *y* line, such as in scatter plots of paired data, I prefer to draw a diagonal line rather than a grid. For example, consider Figure \@ref(fig:gene-expression), which compares gene expression levels in a mutant virus to the non-mutated (wild-type) variant. By drawing the diagonal line, we can see immediately which genes are expressed higher or lower in the mutant relative to the wild type. The same observation is much harder to make when the figure has a background grid and no diagonal line (Figure \@ref(fig:gene-expression-bad)). Thus, even though Figure \@ref(fig:gene-expression-bad) looks pleasing, I label it as bad. In particular, gene *10A*, which clearly has a reduced expression level in the mutant relative to the wild-type (Figure \@ref(fig:gene-expression)), does not visually stand out in Figure \@ref(fig:gene-expression-bad). 

(ref:gene-expression) Gene expression levels in a mutant bacteriophage T7 relative to wild-type. Gene expression levels are measured by mRNA abundances, in transcripts per million (TPM). Each dot corresponds to one gene. In the mutant bacteriophage T7, the promoter in front of gene *9* was deleted, and this resulted in reduced mRNA abundances of gene *9* as well as the neighboring genes *8* and *10A* (highlighted). Data source: @Paffetal2018

```{r gene-expression, fig.width = 4.5, fig.asp = 1, fig.cap = '(ref:gene-expression)'}
df_segment <- data.frame(
  x = c(3.2e-4, 2e-4), 
  xend = c(3.2e-1, 2e-4),
  y = c(2e-4, 3.2e-4),
  yend = c(2e-4, 3.2e-1)
)

gene_expression %>% 
  filter(strain == "phi9v2" & background == "wt") %>%
  ggplot(aes(x=tpm_wt, y=tpm_mutant)) +
  geom_abline(slope = 1, color = "grey") +
  geom_point(pch = 21, fill = "#0072B2D0", color = "white", size = 3) +
  geom_text_repel(
    aes(label = label),
    nudge_y = -0.005,
    fontface = "italic",
    family = "Myriad Pro",
    box.padding = 0.35,
    size = 12/.pt,
    segment.color = NA) +
  geom_segment(
    data = df_segment,
    aes(x = x, xend = xend, y = y, yend = yend),
    size = 0.5, inherit.aes = FALSE) +
  scale_x_log10(
    limits = c(2e-4, 3.3e-1),
    name = "wild-type mRNA abundance (TPM)",
    breaks = c(3.2e-4, 1e-3, 3.2e-3, 1e-2, 3.2e-2, 1e-1, 3.2e-1),
    labels = expression("", 10^-3, "", 10^-2, "", 10^-1, "")
  ) + 
  scale_y_log10(
    limits = c(2e-4, 3.3e-1),
    name = "mutant mRNA abundance (TPM)",
    breaks = c(3.2e-4, 1e-3, 3.2e-3, 1e-2, 3.2e-2, 1e-1, 3.2e-1),
    labels = expression("", 10^-3, "", 10^-2, "", 10^-1, "")
  ) + 
  coord_fixed(expand = FALSE, clip = "off") +
  theme_dviz_open() +
  theme(
    axis.line = element_blank(),
    plot.margin = margin(17.5, 3.5, 3.5, 1.5)
  ) -> gene_expression_plot

gene_expression_plot
```


(ref:gene-expression-bad) Gene expression levels in a mutant bacteriophage T7 relative to wild-type. By plotting this dataset against a background grid, instead of a diagonal line, we are obscuring which genes are higher or lower in the mutant than in the wild-type bacteriophage.  Data source: @Paffetal2018

```{r gene-expression-bad, fig.width = 4.5, fig.asp = 1, fig.cap = '(ref:gene-expression-bad)'}
gene_expression %>% 
  filter(strain == "phi9v2" & background == "wt") %>%
  ggplot(aes(x=tpm_wt, y=tpm_mutant)) +
  geom_point(pch = 21, fill = "#0072B2D0", color = "white", size = 3) +
  geom_text_repel(
    aes(label = label),
    nudge_y = -0.005,
    fontface = "italic",
    family = "Myriad Pro",
    box.padding = 0.35,
    size = 12/.pt,
    segment.color = NA) +
  scale_x_log10(
    limits = c(2e-4, 3.3e-1),
    name = "wild-type mRNA abundance (TPM)",
    breaks = c(3.2e-4, 1e-3, 3.2e-3, 1e-2, 3.2e-2, 1e-1, 3.2e-1),
    labels = expression("", 10^-3, "", 10^-2, "", 10^-1, "")
  ) + 
  scale_y_log10(
    limits = c(2e-4, 3.3e-1),
    name = "mutant mRNA abundance (TPM)",
    breaks = c(3.2e-4, 1e-3, 3.2e-3, 1e-2, 3.2e-2, 1e-1, 3.2e-1),
    labels = expression("", 10^-3, "", 10^-2, "", 10^-1, "")
  ) + 
  coord_fixed(expand = FALSE, clip = "off") +
  theme_dviz_open() +
  theme(
    panel.grid.major = element_line(colour = "grey80", size = 0.5, linetype = 1),
    plot.margin = margin(17.5, 3.5, 3.5, 1.5)
  ) -> gene_expression_plot_bad


stamp_bad(gene_expression_plot_bad)
```


Of course we could take the diagonal line from Figure \@ref(fig:gene-expression) and add it on top of the background grid of Figure \@ref(fig:gene-expression-bad), to ensure that the relevant visual reference is present. However, the resulting figure is getting quite busy (Figure \@ref(fig:gene-expression-ugly)). I had to make the diagonal line darker so it would stand out against the background grid, but now the data points almost seem to fade into the background. We could ameliorate this issue by making the data points larger or darker, but all considered I'd rather choose Figure \@ref(fig:gene-expression).

(ref:gene-expression-ugly) Gene expression levels in a mutant bacteriophage T7 relative to wild-type. This figure combines the background grid from Figure \@ref(fig:gene-expression-bad) with the diagonal line from Figure \@ref(fig:gene-expression). In my opinion, this figure is visually too busy compared to Figure \@ref(fig:gene-expression), and I would prefer Figure \@ref(fig:gene-expression). Data source: @Paffetal2018

```{r gene-expression-ugly, fig.width = 4.5, fig.asp = 1, fig.cap = '(ref:gene-expression-ugly)'}
gene_expression %>% 
  filter(strain == "phi9v2" & background == "wt") %>%
  ggplot(aes(x=tpm_wt, y=tpm_mutant)) +
  geom_abline(slope = 1, color = "black") +
  geom_point(pch = 21, fill = "#0072B2D0", color = "white", size = 3) +
  geom_text_repel(
    aes(label = label),
    nudge_y = -0.005,
    fontface = "italic",
    family = "Myriad Pro",
    box.padding = 0.35,
    size = 12/.pt,
    segment.color = NA) +
  scale_x_log10(
    limits = c(2e-4, 3.3e-1),
    name = "wild-type mRNA abundance (TPM)",
    breaks = c(3.2e-4, 1e-3, 3.2e-3, 1e-2, 3.2e-2, 1e-1, 3.2e-1),
    labels = expression("", 10^-3, "", 10^-2, "", 10^-1, "")
  ) + 
  scale_y_log10(
    limits = c(2e-4, 3.3e-1),
    name = "mutant mRNA abundance (TPM)",
    breaks = c(3.2e-4, 1e-3, 3.2e-3, 1e-2, 3.2e-2, 1e-1, 3.2e-1),
    labels = expression("", 10^-3, "", 10^-2, "", 10^-1, "")
  ) + 
  coord_fixed(expand = FALSE, clip = "off") +
  theme_dviz_open() +
  theme(
    panel.grid.major = element_line(colour = "grey80", size = 0.5, linetype = 1),
    plot.margin = margin(17.5, 3.5, 3.5, 1.5)
  ) -> gene_expression_plot_ugly


stamp_ugly(gene_expression_plot_ugly)
```

## Summary

Both overloading a figure with non-data ink and excessively erasing non-data ink can result in poor figure design. We need to find a healthy medium, where the data points are the main emphasis of the figure while sufficient context is provided about what data is shown, where the points lie relative to each other, and what they mean.

With respect to backgrounds and background grids, there is no one choice that is preferable in all contexts. I recommend to be judicious about grid lines. Think carefully about which specific grid or guide lines are most informative for the plot you are making, and then only show those. I prefer minimal, light grids on a white background, since white is the default neutral color on paper and supports nearly any foreground color. However, a shaded background can help the plot appear as a single visual entity, and this may be particularly useful in small multiples plots. Finally, we have to consider how all these choices relate to visual branding and identity. Many magazines and websites like to have an immediately recognizable in-house style, and a shaded background and specific choice of background grid can help create a unique visual identity.



<!--chapter:end:balance_data_context.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(forcats)
library(ggridges)
```

# Visualizing many distributions at once {#boxplots-violins}

There are many scenarios in which we want to visualize multiple distributions at the same time. For example, consider weather data. We may want to visualize how temperature varies across different months while also showing the distribution of observed temperatures within each month. This scenario requires showing twelve temperature distributions at once, one for each month. None of the visualizations discussed in Chapters \@ref(histograms-density-plots) or \@ref(ecdf-qq) work well in this case. Instead, viable approaches include boxplots, violin plots, and ridgeline plots.

Whenever we are dealing with many distributions, it is helpful to think in terms of the response variable and one or more grouping variables. The response variable is the variable whose distributions we want to show. The grouping variables define subsets of the data with distinct distributions of the response variable. For example, for temperature distributions across months, the response variable is the temperature and the grouping variable is the month. All techniques discussed in this chapter draw the response variable along one axis and the grouping variables along the other. In the following, I will first describe approaches that show the response variable along the vertical axis, and then I will describe approaches that show the response variable along the horizontal axis. In all cases discussed, we could flip the axes and arrive at an alternative and viable visualization. I am showing here the canonical forms of the various visualizations.


## Visualizing distributions along the vertical axis {#boxplots-violins-vertical}

The simplest approach to showing many distributions at once is to show their mean or median as points, with some indication of the variation around the mean or median shown by error bars. Figure \@ref(fig:lincoln-temp-points-errorbars) demonstrates this approach for the distributions of monthly temperatures in Lincoln, Nebraska, in 2016. I have labeled this figure as bad because there are multiple problems with this approach. First, by representing each distribution by only one point and two error bars, we are losing a lot of information about the data. Second, it is not immediately obvious what the points represent, even though most readers would likely guess that they represent either the mean or the median. Third, it is definitely not obvious what the error bars represent. Do they represent the standard deviation of the data, the standard error of the mean, a 95% confidence interval, or something else altogether? There is no commonly accepted standard. By reading the figure caption of Figure \@ref(fig:lincoln-temp-points-errorbars), we can see that they represent here twice the standard deviation of the daily mean temperatures, meant to indicate the range that contains approximately 95% of the data. However, error bars are more commonly employed to visualize the standard error (or twice the standard error for a 95% confidence interval), and it is easy for readers to confuse the standard error with the standard deviation. The standard error quantifies how accurate our estimate of the mean is, whereas the standard deviation estimates how much spread there is in the data around the mean. It is possible for a dataset to have both a very small standard error of the mean and a very large standard deviation. Fourth, symmetric error bars are misleading if there is any skew in the data, which is the case here and almost always for real-world datasets.

(ref:lincoln-temp-points-errorbars) Mean daily temperatures in Lincoln, Nebraska in 2016. Points represent the average daily mean temperatures for each month, averaged over all days of the month, and error bars represent twice the standard deviation of the daily mean temperatures within each month. This figure has been labeled as "bad" because because error bars are conventionally used to visualize the uncertainty of an estimate, not the variability in a population. Data source: Weather Underground

```{r lincoln-temp-points-errorbars, fig.cap = '(ref:lincoln-temp-points-errorbars)'}
lincoln_weather %>%
  mutate(
    month_short = fct_recode(
      Month,
      Jan = "January",
      Feb = "February",
      Mar = "March",
      Apr = "April",
      May = "May",
      Jun = "June",
      Jul = "July",
      Aug = "August",
      Sep = "September",
      Oct = "October",
      Nov = "November",
      Dec = "December"
    )
  ) %>%
  mutate(month_short = fct_rev(month_short)) -> lincoln_df


lincoln_errbar <- ggplot(lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)) +
  stat_summary(
    fun.y = mean, fun.ymax = function(x) {mean(x) + 2*sd(x)},
    fun.ymin = function(x) {mean(x) - 2*sd(x)}, geom = "pointrange",
    fatten = 5
  ) +
  xlab("month") + 
  ylab("mean temperature (°F)") +
  theme_dviz_open() +
  theme(plot.margin = margin(3, 7, 3, 1.5))

stamp_bad(lincoln_errbar)
```

We can address all four shortcomings of Figure \@ref(fig:lincoln-temp-points-errorbars) by using a traditional and commonly used method for visualizing distributions, the boxplot. A boxplot divides the data into quartiles and visualizes them in a standardized manner (Figure \@ref(fig:boxplot-schematic)).

(ref:boxplot-schematic) Anatomy of a boxplot. Shown are a cloud of points (left) and the corresponding boxplot (right). Only the *y* values of the points are visualized in the boxplot. The line in the middle of the boxplot represents the median, and the box encloses the middle 50% of the data. The top and bottom whiskers extend either to the maximum and minimum of the data or to the maximum or minimum that falls within 1.5 times the height of the box, whichever yields the shorter whisker. The distances of 1.5 times the height of the box in either direction are called the upper and the lower fences. Individual data points that fall beyond the fences are referred to as outliers and are usually shown as individual dots.

```{r boxplot-schematic, fig.width = 5*6/4.2, fig.cap = '(ref:boxplot-schematic)'}
set.seed(3423)
y <- c(rnorm(100), 3.4)
s <- boxplot.stats(y)
df <- data.frame(y = c(s$stats, max(y)),
                 x = c(1.03, 1.405, 1.405, 1.405, 1.03, 1.04),
                 label = c("minimum", "first quartile", "median", "third quartile", "maximum within upper fence", "outlier"))
p_boxplot <- ggplot(data.frame(y), aes(x = 1, y = y)) + geom_boxplot(fill = 'grey90') +
  geom_text(data = df, aes(x, y, label = label), hjust = 0,
            size = 14/.pt, family = dviz_font_family) +
  scale_x_continuous(limits = c(0, 3.5), expand = c(0, 0)) +
  scale_y_continuous(limits = c(-2.55, 3.5), expand = c(0, 0)) +
  theme_nothing()

p_points <- ggplot(data.frame(y), aes(x = 0, y = y)) + 
  geom_point(position = position_jitter(width = .4, height = 0, seed = 320)) +
  scale_x_continuous(limits = c(-1.8, .4), expand = c(0, 0)) +
  scale_y_continuous(limits = c(-2.55, 3.5), expand = c(0, 0)) +
  theme_nothing()

plot_grid(p_points, p_boxplot, rel_widths = c(.65, 1), nrow = 1)
```

Boxplots are simple yet informative, and they work well when plotted next to each other to visualize many distributions at once. For the Lincoln temperature data, using boxplots leads to Figure \@ref(fig:lincoln-temp-boxplots). In that figure, we can now see that temperature is highly skewed in December (most days are moderately cold and a few are extremely cold) and not very skewed at all in some other months, for example in July.

(ref:lincoln-temp-boxplots) Mean daily temperatures in Lincoln, Nebraska, visualized as boxplots.

```{r lincoln-temp-boxplots, fig.cap = '(ref:lincoln-temp-boxplots)'}
lincoln_box <- ggplot(lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)) +
  geom_boxplot(fill = 'grey90') + 
  xlab("month") + 
  ylab("mean temperature (°F)") +
  theme_dviz_open() +
  theme(plot.margin = margin(3, 7, 3, 1.5))

lincoln_box
```

Boxplots were invented by the statistician John Tukey in the early 1970s, and they quickly gained popularity because they were highly informative while being easy to draw by hand. Most data visualizations were drawn by hand at that time. However, with modern computing and visualization capabilities, we are not limited to what is easily drawn by hand. Therefore, more recently, we see boxplots being replaced by violin plots, which are equivalent to the density estimates discussed in Chapter \@ref(histograms-density-plots) but rotated by 90 degrees and then mirrored (Figure \@ref(fig:violin-schematic)). Violins can be used whenever one would otherwise use a boxplot, and they provide a much more nuanced picture of the data. In particular, violin plots will accurately represent bimodal data whereas a boxplot will not.


(ref:violin-schematic) Anatomy of a violin plot. Shown are a cloud of points (left) and the corresponding violin plot (right). Only the *y* values of the points are visualized in the violin plot. The width of the violin at a given *y* value represents the point density at that *y* value. Technically, a violin plot is a density estimate rotated by 90 degrees and then mirrored. Violins are therefore symmetric. Violins begin and end at the minimum and maximum data values, respectively. The thickest part of the violin corresponds to the highest point density in the dataset.

```{r violin-schematic, fig.width = 5*6/4.2, fig.cap = '(ref:violin-schematic)'}
set.seed(3423)
y <- c(rnorm(100), 3.4)
d <- density(y)
df <- data.frame(y = c(min(y), d$x[which(d$y == max(d$y))], max(y)),
                 x = c(1.1, 1.5, 1.08),
                 label = c("minimum data value", "maximum point density", "maximum data value"))
p_violin <- ggplot(data.frame(y), aes(x = 1, y = y)) + geom_violin(fill = 'grey90') +
  geom_text(data = df, aes(x, y, label = label), hjust = 0,
            size = 14/.pt, family = dviz_font_family) +
  scale_x_continuous(limits = c(0, 3.5), expand = c(0, 0)) +
  scale_y_continuous(limits = c(-2.55, 3.5), expand = c(0, 0)) +
  theme_nothing()

p_points <- ggplot(data.frame(y), aes(x = 0, y = y)) + 
  geom_point(position = position_jitter(width = .4, height = 0, seed = 320)) +
  scale_x_continuous(limits = c(-1.8, .4), expand = c(0, 0)) +
  scale_y_continuous(limits = c(-2.55, 3.5), expand = c(0, 0)) +
  theme_nothing()

plot_grid(p_points, p_violin, rel_widths = c(.65, 1), nrow = 1)
```

```{block type='rmdtip', echo=TRUE}
Before using violins to visualize distributions, verify that you have sufficiently many data points in each group to justify showing the point densities as smooth lines.
```


When we visualize the Lincoln temperature data with violins, we obtain Figure \@ref(fig:lincoln-temp-violins). We can now see that some months do have moderately bimodal data. For example, the month of November seems to have had two temperature clusters, one around 50 degrees and one around 35 degrees Fahrenheit.


(ref:lincoln-temp-violins) Mean daily temperatures in Lincoln, Nebraska, visualized as violin plots.

```{r lincoln-temp-violins, fig.cap = '(ref:lincoln-temp-violins)'}
lincoln_violin <- ggplot(lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)) +
  geom_violin(fill = 'grey90') +
  xlab("month") + 
  ylab("mean temperature (°F)") +
  theme_dviz_open() +
  theme(plot.margin = margin(3, 7, 3, 1.5))

lincoln_violin
```


Because violin plots are derived from density estimates, they have similar shortcomings (Chapter \@ref(histograms-density-plots)). In particular, they can generate the appearance that there is data where none exists, or that the data set is very dense when actually it is quite sparse. We can try to circumvent these issues by simply plotting all the individual data points directly, as dots (Figure \@ref(fig:lincoln-temp-all-points)). Such a figure is called a *strip chart.* Strip charts are fine in principle, as long as we make sure that we don't plot too many points on top of each other. A simple solution to overplotting is to spread out the points somewhat along the *x* axis, by adding some random noise in the *x* dimension (Figure \@ref(fig:lincoln-temp-jittered)). This technique is also called *jittering.*

(ref:lincoln-temp-all-points) Mean daily temperatures in Lincoln, Nebraska, visualized as strip chart. Each point represents the mean temperature for one day. This figure is labeled as "bad" because so many points are plotted on top of each other that it is not possible to ascertain which temperatures were the most common in each month. 

```{r lincoln-temp-all-points, fig.cap = '(ref:lincoln-temp-all-points)'}
lincoln_points <- ggplot(lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)) +
  geom_point(size = 0.75) + 
  xlab("month") + 
  ylab("mean temperature (°F)") +
  theme_dviz_open() +
  theme(plot.margin = margin(3, 7, 3, 1.5))

stamp_bad(lincoln_points)
```


(ref:lincoln-temp-jittered) Mean daily temperatures in Lincoln, Nebraska, visualized as strip chart. The points have been jittered along the *x* axis to better show the density of points at each temperature value.

```{r lincoln-temp-jittered, fig.cap = '(ref:lincoln-temp-jittered)'}
lincoln_jitter <- ggplot(lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)) +
  geom_point(position = position_jitter(width = .15, height = 0, seed = 320), size = 0.75) +
  xlab("month") + 
  ylab("mean temperature (°F)") +
  theme_dviz_open() +
  theme(plot.margin = margin(3, 7, 3, 1.5))

lincoln_jitter
```

```{block type='rmdtip', echo=TRUE}
Whenever the dataset is too sparse to justify the violin visualization, plotting the raw data as individual points will be possible.
```


Finally, we can combine the best of both worlds by spreading out the dots in proportion to the point density at a given *y* coordinate. This method, called a *sina plot* [@Sidiropoulos_et_al_2018], can be thought of as a hybrid between a violin plot and jittered points, and it shows each individual point while also visualizing the distributions. I have here drawn the sina plots on top of the violins to highlight the relationship between these two approaches (Figure \@ref(fig:lincoln-temp-sina)).

(ref:lincoln-temp-sina) Mean daily temperatures in Lincoln, Nebraska, visualized as a sina plot (combination of individual points and violins). The points have been jittered along the *x* axis in proportion to the point density at the respective temperature. The name *sina plot* is meant to honor Sina Hadi Sohi, a student at the University of Copenhagen, Denmark, who wrote the first version of the code that researchers at the university used to make such plots (Frederik O. Bagger, personal communication).

```{r lincoln-temp-sina, fig.cap = '(ref:lincoln-temp-sina)'}
lincoln_sina <- ggplot(lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)) +
  geom_violin(color = "transparent", fill = "gray90") + 
  stat_sina(size = 0.75) + 
  xlab("month") + 
  ylab("mean temperature (°F)") +
  theme_dviz_open() +
  theme(plot.margin = margin(3, 7, 3, 1.5))

lincoln_sina
```


## Visualizing distributions along the horizontal axis {#boxplots-violins-horizontal}

In Chapter \@ref(histograms-density-plots), we visualized distributions along the horizontal axis using histograms and density plots. Here, we will expand on this idea by staggering the distribution plots in the vertical direction. The resulting visualization is called a ridgeline plot, because these plots look like mountain ridgelines. Ridgeline plots tend to work particularly well if want to show trends in distributions over time.

The standard ridgeline plot uses density estimates (Figure \@ref(fig:temp-ridgeline)). It is quite closely related to the violin plot, but frequently evokes a more intuitive understanding of the data. For example, the two clusters of temperatures around 35 degrees and 50 degrees Fahrenheit in November are much more obvious in Figure \@ref(fig:temp-ridgeline) than in Figure \@ref(fig:lincoln-temp-violins).

(ref:temp-ridgeline) Temperatures in Lincoln, Nebraska, in 2016, visualized as a ridgeline plot. For each month, we show the distribution of daily mean temperatures measured in Fahrenheit. Original figure concept: @Wehrwein-Lincoln-weather.

```{r temp-ridgeline, fig.cap = '(ref:temp-ridgeline)'}
bandwidth <- 3.4

ggplot(lincoln_df, aes(x = `Mean Temperature [F]`, y = `Month`)) +
  geom_density_ridges(
    scale = 3, rel_min_height = 0.01,
    bandwidth = bandwidth, fill = lighten("#56B4E9", .3), color = "white"
  ) +
  scale_x_continuous(
    name = "mean temperature (°F)",
    expand = c(0, 0), breaks = c(0, 25, 50, 75)
  ) +
  scale_y_discrete(name = NULL, expand = c(0, .2, 0, 2.6)) +
  theme_dviz_grid() +
  theme(
    axis.text.y = element_text(vjust = 0),
    plot.margin = margin(3, 7, 3, 1.5)
  )
```

Because the *x* axis shows the response variable and the *y* axis shows the grouping variable, there is no separate axis for the density estimates in a ridgeline plot. Density estimates are shown alongside the grouping variable. This is no different from the violin plot, where densities are also shown alongside the grouping variable, without a separate, explicit scale. In both cases, the purpose of the plot is not to show specific density values but instead to allow for easy comparison of density shapes and relative heights across groups.


In principle, we can use histograms instead of density plots in a ridgeline visualization. However, the resulting figures often don't look very good (Figure \@ref(fig:temp-binline)). The problems are similar to those of stacked or overlapping histograms (Chapter \@ref(histograms-density-plots)). Because the vertical lines in these ridgeline histograms appear always at the exact same *x* values, the bars from different histograms align with each other in confusing ways. In my opinion, it is better to not draw such overlapping histograms.

(ref:temp-binline) Temperatures in Lincoln, Nebraska, in 2016, visualized as a ridgeline plot of histograms. The individual histograms don't separate well visually, and the overall figure is quite busy and confusing.

```{r temp-binline, fig.cap = '(ref:temp-binline)'}
bandwidth <- 3.4

p_binline <- ggplot(lincoln_df, aes(x = `Mean Temperature [F]`, y = `Month`)) +
  stat_binline(
    scale = 3, bins = 17, draw_baseline = FALSE,
    fill = lighten("#56B4E9", .3), color = "black", alpha = 1
  ) +
  scale_x_continuous(
    name = "mean temperature (°F)",
    expand = c(0, 0), breaks = c(0, 25, 50, 75)
  ) +
  scale_y_discrete(
    name = NULL,
    expand = c(0, .2, 0, 2.6)
  ) +
  theme_dviz_grid() +
  theme(
    axis.text.y = element_text(vjust = 0),
    plot.margin = margin(3, 7, 3, 1.5)
  )

stamp_ugly(p_binline) 
```



Ridgeline plots scale to very large numbers of distributions. For example, Figure \@ref(fig:movies-ridgeline) shows the distributions of movie lengths from 1913 to 2005. This figure contains almost 100 distinct distributions and yet it is very easy to read. We can see that in the 1920s, movies came in many different lengths, but since about 1960 movie length has standardized to approximately 90 minutes.

(ref:movies-ridgeline) Evolution of movie lengths over time. Since the 1960s, the majority of all movies are approximately 90 minutes long. Data source: Internet Movie Database, IMDB

```{r movies-ridgeline, fig.width = 5, fig.asp = 1, fig.cap = '(ref:movies-ridgeline)'}
ggplot(movie_lengths, aes(x = length, y = year, group = year)) +
  geom_density_ridges(scale = 10, size = 0.25, rel_min_height = 0.03, fill = "grey85", na.rm = TRUE) +
  scale_x_continuous(limits = c(0, 200), expand = c(0, 0), name = "length (minutes)") +
  scale_y_reverse(
    breaks = c(2000, 1980, 1960, 1940, 1920),
    limits = c(2005, 1903), expand = c(0, 0)
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_grid() +
  theme(
    plot.margin = margin(3, 14, 3, 1.5)
    )
```

Ridgeline plots also work well if we want to compare two trends over time. This is a scenario that arises commonly if we want to analyze the voting patterns of the members of two different parties. We can make this comparison by staggering the distributions vertically by time and drawing two differently colored distributions at each time point, representing the two parties (Figure \@ref(fig:dw-nominate-ridgeline)).

(ref:dw-nominate-ridgeline) Voting patterns in the U.S. House of Representatives have become increasingly polarized. DW-NOMINATE scores are frequently used to compare voting patterns of representatives between parties and over time. Here, score distributions are shown for each Congress from 1963 to 2013 separately for Democrats and Republicans. Each Congress is represented by its first year. Original figure concept: @McDonald-DW-NOMINATE.

```{r dw-nominate-ridgeline, fig.width = 5.5*6/4.2, fig.asp = 0.5, fig.cap = '(ref:dw-nominate-ridgeline)'}
# U.S. House 1963-2013
all_house_88_113 <- dw_nominate_house %>% 
  filter(congress >= 88 & cd !=0 & cd != 98 & cd != 99) %>%
  filter(party_code == 100 | party_code == 200) %>%
  arrange(desc(congress)) %>% mutate(year1 = congress * 2 + 1787) %>%
  arrange(desc(year1))

ggplot(all_house_88_113,
       aes(
         x = dim_1,
         y = year1,
         group = interaction(party_code, factor(year1)),
         fill = interaction(party_code, factor(year1))
        )
      ) +
  geom_density_ridges(scale = 5, size = 0.25, rel_min_height = 0.01, alpha=0.9, color = "white") +
  scale_x_continuous(
    name = "DW-NOMINATE score",
    limits = c(-.8, 1.3),
    breaks = c(-1,-.75,-.5,-.25,0,.25,.5,.75,1)
  ) +
  scale_y_reverse(
    name = "year",
    expand = c(0, 0), breaks=c(seq(2013, 1963, -10))
  ) +
  scale_fill_cyclical(
    breaks = c("100.1963", "200.1963"),
    labels = c(`100.1963` = "Democrats  ", `200.1963` = "Republicans"),
    values = c("#4040ff", "#ff4040", "#6060ff", "#ff6060"),
    name = NULL,
    guide = "legend"
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_grid() +
  theme(
    axis.text.y = element_text(vjust = 0),
    legend.position = c(1, 1),
    legend.justification = c(1, 1),
    legend.direction = "horizontal",
    legend.background = element_rect(fill = "white")
  )
```

<!--chapter:end:boxplots_violins.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(ggthemes)
library(forcats)
```

# Choosing the right visualization software {#choosing-visualization-software}

Throughout this book, I have purposefully avoided one critical question of data visualization: How do we actually generate our figures? What tools should we use? This question can generate heated discussions, as many people have strong emotional bonds to the specific tools they are familiar with. I have often seen people vigorously defend their own preferred tools instead of investing time to learn a new approach, even if the new approach has objective benefits. And I will say that sticking with the tools you know is not entirely unreasonable. Learning any new tool will require time and effort, and you will have to go through a painful transition period where getting things done with the new tool is much more difficult than it was with the old tool. Whether going through this period is worth the effort can usually only be evaluated in retrospect, after one has made the investment to learn the new tool. Therefore, regardless of the pros and cons of different tools and approaches, the overriding principle is that you need to pick a tool that works for you. If you can make the figures you want to make, without excessive effort, then that's all that matters. 

```{block type='rmdtip', echo=TRUE}
The best visualization software is the one that allows you to make the figures you need.
```

Having said this, I do think there are general principles we can use to assess the relative merits of different approaches to producing visualizations. These principles roughly break down by how reproducible the visualizations are, how easy it is to rapidly explore the data, and to what extent the visual appearance of the output can be tweaked.

## Reproducibility and repeatability

In the context of scientific experiments, we refer to work as reproducible if the overarching scientific finding of the work will remain unchanged if a different research group performs the same type of study. For example, if one research group finds that a new pain medication reduces perceived headache pain significantly without causing noticeable side effects and a different group subsequently studies the same medication on a different patient group and has the same findings, then the work is reproducible. By contrast, work is repeatable if very similar or identical measurements can be obtained by the same person repeating the exact same measurement procedure on the same equipment. For example, if I weigh my dog and find she weighs 41 lbs and then I weigh her again on the same scales and find again that she weighs 41 lbs, then this measurement is repeatable.

With minor modifications, we can apply these concepts to data visualization. A visualization is reproducible if the plotted data are available and any data transformations that may have been applied are exactly specified. For example, if you make a figure and then send me the exact data that you plotted, then I can prepare a figure that looks substantially similar. We may be using slightly different fonts or colors or point sizes to display the same data, so the two figures may not be exactly identical, but your figure and mine convey the same message and therefore are reproductions of each other. A visualization is repeatable, on the other hand, if it is possible to recreate the exact same visual appearance, down to the last pixel, from the raw data. Strictly speaking, repeatability requires that even if there are random elements in the figure, such as jitter (Chapter \@ref(overlapping-points)), those elements were specified in a repeatable way and can be regenerated at a future date. For random data, repeatability generally requires that we specify a particular random number generator for which we set and record a seed.

Throughout this book, we have seen many examples of figures that reproduce but don't repeat other figures. For example, Chapter \@ref(avoid-line-drawings) shows several sets of figures where all figures in each set show the same data but each figure in each set looks somewhat different. Similarly, Figure \@ref(fig:lincoln-repro)a is a repeat of Figure \@ref(fig:lincoln-temp-jittered), down to the random jitter that was applied to each data point, whereas Figure \@ref(fig:lincoln-repro)b is only a reproduction of that figure. Figure \@ref(fig:lincoln-repro)b has different jitter than Figure \@ref(fig:lincoln-temp-jittered), and it also uses a sufficiently different visual design that the two figures look quite distinct, even if they clearly convey the same information about the data.

(ref:lincoln-repro) Repeat and reproduction of a figure. Part (a) is a repeat of Figure \@ref(fig:lincoln-temp-jittered). The two figures are identical down to the random jitter that was applied to each point. By contrast, part (b) is a reproduction but not a repeat. In particular, the jitter in part (b) differs from the jitter in part (a) or in Figure \@ref(fig:lincoln-temp-jittered).

```{r lincoln-repro, fig.width = 5.5*6/4.2, fig.asp = .32, fig.cap = '(ref:lincoln-repro)'}
ggridges::lincoln_weather %>% 
  mutate(
    month_short = fct_recode(
      Month,
      Jan = "January",
      Feb = "February",
      Mar = "March",
      Apr = "April",
      May = "May",
      Jun = "June",
      Jul = "July",
      Aug = "August",
      Sep = "September",
      Oct = "October",
      Nov = "November",
      Dec = "December"
    )
  ) %>%
  mutate(month_short = fct_rev(month_short)) -> lincoln_df

lincoln1 <- ggplot(
  lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)
) +
  geom_point(
    position = position_jitter(width = .15, height = 0, seed = 320),
    size = .35
  ) +
  xlab("month") + 
  ylab("mean temperature (°F)") +
  theme_dviz_open(.655*14) + 
  theme(
    axis.line = element_line(size = .655*.5),
    axis.ticks = element_line(size = .655*.5),
    plot.margin = margin(2, 6, 2, 1.5)
  )

lincoln2 <- ggplot(
  lincoln_df, aes(x = month_short, y = `Mean Temperature [F]`)
) +
  geom_point(
    position = position_jitter(width = .2, height = 0, seed = 321),
    size = .5, color = darken("#0072B2", .3)
  ) +
  xlab("month") + 
  ylab("mean temperature (°F)") +
  theme_dviz_grid(12) + 
  theme(
    axis.ticks.length = grid::unit(0, "pt"),
    axis.ticks = element_blank()
  )

plot_grid(lincoln1, lincoln2, labels = "auto", ncol = 2)
```

Both reproducibility and repeatability can be difficult to achieve when we're working with interactive plotting software. Many interactive programs allow you to transform or otherwise manipulate the data but don't keep track of every individual data transformation you perform, only of the final product. If you make a figure using this kind of a program, and then somebody asks you to reproduce the figure or create similar one with a different data set, you might have difficulty to do so. During my years as a postdoc and a young assistant professor, I used an interactive program for all my scientific visualizations, and this exact issue happened to me several times. For example, I had made several figures for a scientific manuscript. When I wanted to revise the manuscript a few months later and needed to reproduce a slightly altered version of one of the figures, I realized that I wasn't quite sure anymore how I had made the original figure in the first place. This experience has taught me to stay away from interactive programs as much as possible. I now make figures programmatically, by writing code (scripts) that generates the figures from the raw data. Programmatically generated figures will generally be repeatable by anybody who has access to the generating scripts and the programming language and specific libraries used.


## Data exploration versus data presentation

There are two distinct phases of data visualization, and they have very different requirements. The first is data exploration. Whenever you start working with a new dataset, you need to look at it from different angles and try various ways of visualizing it, just to develop an understanding of the dataset's key features. In this phase, speed and efficiency are of the essence. You need to try different types of visualizations, different data transformations, and different subsets of the data. The faster you can iterate through different ways of looking at the data, the more you will explore, and the higher the likelihood that you will notice an important feature in the data that you might otherwise have overlooked. The second phase is data presentation. You enter it once you understand your dataset and know what aspects of it you want to show to your audience. The key objective in this phase is to prepare a high-quality, publication-ready figure that can be printed in an article or book, included in a presentation, or posted on the internet.

In the exploration stage, whether the figures you make look appealing is secondary. It's fine if the axis labels are missing, the legend is messed up, or the symbols are too small, as long as you can evaluate the various patterns in the data. What is critical, however, is how easy it is for you to change how the data are shown. To truly explore the data, you should be able to rapidly move from a scatter plot to overlapping density distribution plots to boxplots to a heatmap. In Chapter \@ref(aesthetic-mapping), we have discussed how all visualizations consist of mappings from data onto aesthetics. A well-designed data exploration tool will allow you to easily change which variables are mapped onto which aesthetics, and it will provide a wide range of different visualization options within a single coherent framework. In my experience, however, many visualization tools (and in particular libraries for programmatic figure generation)  are not set up in this way. Instead, they are organized by plot type, where each different type of plot requires somewhat different input data and has its own idiosyncratic interface. Such tools can get in the way of efficient data exploration, because it's difficult to remember how all the different plot types work. I encourage you to carefully evaluate whether your visualization software allows for rapid data exploration or whether it tends to get in the way. If it more frequently tends to get in the way, you may benefit from exploring alternative visualization options.

Once we have determined how exactly we want to visualize our data, what data transformations we want to make, and what type of plot to use, we will commonly want to prepare a high-quality figure for publication. At this point, we have several different avenues we can pursue. First, we can finalize the figure using same software platform we used for initial exploration. Second, we can switch platform to one that provides us finer control over the final product, even if that platform makes it harder to explore. Third, we can produce a draft figure with a visualization software and then manually post-process with an image manipulation or illustration program such as Photoshop or Illustrator. Fourth, we can manually redraw the entire figure from scratch, either with pen and paper  or using an illustration program.

All these avenues are reasonable. However, I would like to caution against manually sprucing up figures in routine data analysis pipelines or for scientific publications. Manual steps in the figure preparation pipeline make repeating or reproducing a figure inherently difficult and time-consuming. And in my experience from working in the natural sciences, we rarely make a figure just once. Over the course of a study, we may redo experiments, expand the original dataset, or repeat an experiment several times with slightly altered conditions. I've seen it many times that late in the publication process, when we think everything is done and finalized, we end up introducing a small modification to how we analyze our data, and consequently all figures have to be redrawn. And I've also seen, in similar situations, that a decision is made not to redo the analysis or not to redraw the figures, either due to the effort involved or because the people who had made the original figure have moved on and aren't available anymore. In all these scenarios, an unnecessarily complicated and non-reproducible data visualization pipeline interferes with producing the best possible science.

Having said this, I have no principled concern about hand-drawn figures or figures that have been manually post-processed, for example to change axis labels, add annotations, or modify colors. These approaches can yield beautiful and unique figures that couldn't easily be made in any other way. In fact, as sophisticated and polished computer-generated visualizations are becoming increasingly commonplace, I observe that manually drawn figures are making somewhat of a resurgence (see Figure \@ref(fig:sequencing-cost) for an example). I think this is the case because such figures represent a unique and personalized take on what might otherwise be a somewhat sterile and routine presentation of data.

(ref:sequencing-cost) After the introduction of next-gen sequencing methods, the sequencing cost per genome has declined much more rapidly than predicted by Moore's law. This hand-drawn figure reproduces a widely publicized visualization prepared by the National Institutes of Health. Data source: National Human Genome Research Institute

```{r sequencing-cost, out.width = "70%", fig.cap='(ref:sequencing-cost)'}
knitr::include_graphics("figures/sequencing_costs.png", auto_pdf = FALSE)
```

## Separation of content and design

A good visualization software should allow you to think separately about the content and the design of your figures. By content, I refer to the specific data set shown, the data transformations applied (if any), the specific mappings from data onto aesthetics, the scales, the axis ranges, and the type of plot (scatter plot, line plot, bar plot, boxplot, etc.). Design, on the other hand, describes features such as the foreground and background colors, font specifications (e.g. font size, face, and family), symbol shapes and sizes, the placement of legends, axis ticks, axis titles, and plot titles, and whether or not the figure has a background grid. When I work on a new visualization, I usually determine first what the contents should be, using the kind of rapid exploration described in the previous subsection. Once the contents is set, I may tweak the design, or more likely I will apply a pre-defined design that I like and/or that gives the figure a consistent look in the context of a larger body of work.

In the software I have used for this book, ggplot2, separation of content and design is achieved via themes. A theme specifies the visual appearance of a figure, and it is easy to take an existing figure and apply different themes to it (Figure \@ref(fig:unemploy-themes)). Themes can be written by third parties and distributed as R packages. Through this mechanism, a thriving ecosystem of add-on themes has developed around ggplot2, and it covers a wide range of different styles and application scenarios. If you're making figures with ggplot2, you can almost certainly find an existing theme that satisfies your design needs.

(ref:unemploy-themes) Number of unemployed persons in the U.S. from 1970 to 2015. The same figure is displayed using four different ggplot2 themes: (a) the default theme for this book; (b) the default theme of ggplot2, the plotting software I have used to make all figures in this book; (c) a theme that mimics visualizations shown in the Economist; (d) a theme that mimics visualizations shown by FiveThirtyEight. FiveThirtyEight often forgoes axis labels in favor of plot titles and subtitles, and therefore I have adjusted the figure accordingly. Data source: U.S. Bureau of Labor Statistics

```{r unemploy-themes, fig.width = 5.5*6/4.2, fig.asp = 0.75, fig.cap = '(ref:unemploy-themes)'}
unemploy_base <- ggplot(economics, aes(x = date, y = unemploy)) +
  scale_y_continuous(
    name = "unemployed (x1000)",
    limits = c(0, 17000),
    breaks = c(0, 5000, 10000, 15000),
    labels = c("0", "5000", "10,000", "15,000"),
    expand = c(0.04, 0)
  ) +
  scale_x_date(
    name = "year",
    expand = c(0.01, 0)
  )

unemploy_p1 <- unemploy_base + 
  theme_dviz_grid(12) +
  geom_line(color = "#0072B2") +
  theme(
    axis.ticks.length = grid::unit(0, "pt"),
    axis.ticks = element_blank(),
    plot.margin = margin(6, 6, 6, 2)
  )

unemploy_p2 <- unemploy_base + geom_line() + theme_gray()

unemploy_p3 <- unemploy_base + 
  geom_line(aes(color = "unemploy"), show.legend = FALSE, size = .75) +
  theme_economist() + 
  scale_color_economist() +
  theme(panel.grid.major = element_line(size = .75))

unemploy_p4 <- unemploy_base + 
  geom_line(aes(color = "unemploy"), show.legend = FALSE) +
  scale_color_fivethirtyeight() +
  labs(
    title = "United States unemployment",
    subtitle = "Unemployed persons (in thousands) from\n1967 to 2015"
  ) +
  theme_fivethirtyeight() +
  theme(
    plot.title = element_text(size = 12, margin = margin(0, 0, 3, 0)),
    plot.subtitle = element_text(size = 10, lineheight = 1)
  )

plot_grid(
  unemploy_p1, NULL, unemploy_p2, 
  NULL, NULL, NULL,
  unemploy_p3, NULL, unemploy_p4,
  labels = c("a", "", "b", "", "", "", "c", "", "d"),
  hjust = -.5,
  vjust = 1.5,
  rel_widths = c(1, .02, 1),
  rel_heights = c(1, .02, 1)
)
```

Separation of content and design allows data scientists and designers to each focus on what they do best. Most data scientists are not designers, and therefore their primary concern should be the data, not the design of a visualization. Likewise, most designers are not data scientists, and they should be able provide a unique and appealing visual language for figures without having to worry about specific data, appropriate transformations, and so on. The same principle of separating content and design has long been followed in the publishing world of books, magazines, newspapers, and websites, where writers provide content but not layout or design. Layout and design are created by a separate group of people who specialize in this area and who ensure that the publication appears in a visually consistent and appealing style. This principle is logical and useful, but it is not yet that widespread in the data visualization world.

In summary, when choosing your visualization software, think about how easily you can reproduce figures and redo them with updated or otherwise changed datasets, whether you can rapidly explore different visualizations of the same data, and to what extent you can tweak the visual design separately from generating the figure content. Depending on your skill level and comfort with programming, it may be beneficial to use different visualization tools at the data exploration and the data presentation stages, and you may prefer to do the final visual tweaking interactively or by hand. If you have to make figures interactively, in particular with a software that does not keep track of all data transformations and visual tweaks you have applied, consider taking careful notes on how you make each figure, so that all your work remains reproducible. 

<!--chapter:end:choosing_visualization_software.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")

library(sf)
```

# Color scales {#color-basics}


There are three fundamental use cases for color in data visualizations: (i) we can use color to distinguish groups of data from each other; (ii) we can use color to represent data values; and (iii) we can use color to highlight. The types of colors we use and the way in which we use them are quite different for these three cases.


## Color as a tool to distinguish

We frequently use color as a means to distinguish discrete items or groups that do not have an intrinsic order, such as different countries on a map or different manufacturers of a certain product. In this case, we use a *qualitative* color scale. Such a scale contains a finite set of specific colors that are chosen to look clearly distinct from each other while also being equivalent to each other. The second condition requires that no one color should stand out relative to the others. And, the colors should not create the impression of an order, as would be the case with a sequence of colors that get successively lighter. Such colors would create an apparent order among the items being colored, which by definition have no order. 

Many appropriate qualitative color scales are readily available. Figure \@ref(fig:qualitative-scales) shows three representative examples. In particular, the ColorBrewer project provides a nice selection of qualitative color scales, including both fairly light and fairly dark colors [@ColorBrewer]. 

(ref:qualitative-scales) Example qualitative color scales. The Okabe Ito scale is the default scale used throughout this book [@Okabe-Ito-CUD]. The ColorBrewer Dark2 scale is provided by the ColorBrewer project [@ColorBrewer]. The ggplot2 hue scale is the default qualitative scale in the widely used plotting software ggplot2.

```{r qualitative-scales, fig.width=5*6/4.2, fig.asp=3*.14, fig.cap = '(ref:qualitative-scales)'}
p1 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_OkabeIto() + ggtitle("Okabe Ito") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
p2 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_brewer(type = "qual", palette = "Dark2") + ggtitle("ColorBrewer Dark2") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
p3 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_hue() + ggtitle("ggplot2 hue") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
plot_grid(p1, p2, p3, ncol = 1)
```

As an example of how we use qualitative color scales, consider Figure \@ref(fig:popgrowth-US). It shows the percent population growth from 2000 to 2010 in U.S. states. I have arranged the states in order of their population growth, and I have colored them by geographic region. This coloring highlights that states in the same regions have experienced similar population growth. In particular, states in the West and the South have seen the largest population increases whereas states in the Midwest and the Northeast have grown much less.

(ref:popgrowth-US) Population growth in the U.S. from 2000 to 2010. States in the West and South have seen the largest increases, whereas states in the Midwest and Northeast have seen much smaller increases or even, in the case of Michigan, a decrease. Data source: U.S. Census Bureau

```{r popgrowth-US, fig.width = 6, fig.asp = 1.2, fig.cap = '(ref:popgrowth-US)'}
popgrowth_df <- left_join(US_census, US_regions) %>%
    group_by(region, division, state) %>%
    summarize(pop2000 = sum(pop2000, na.rm = TRUE),
              pop2010 = sum(pop2010, na.rm = TRUE),
              popgrowth = (pop2010-pop2000)/pop2000,
              area = sum(area)) %>%
    arrange(popgrowth) %>%
    ungroup() %>%
    mutate(state = factor(state, levels = state),
           region = factor(region, levels = c("West", "South", "Midwest", "Northeast")))

# make color vector in order of the state
region_colors <- c("#E69F00", "#56B4E9", "#009E73", "#F0E442")
region_colors_dark <- darken(region_colors, 0.4)
state_colors <- region_colors_dark[as.numeric(popgrowth_df$region[order(popgrowth_df$state)])]

ggplot(popgrowth_df, aes(x = state, y = 100*popgrowth, fill = region)) + 
  geom_col() + 
  scale_y_continuous(
    limits = c(-.6, 37.5), expand = c(0, 0),
    labels = scales::percent_format(accuracy = 1, scale = 1),
    name = "population growth, 2000 to 2010"
  ) +
  scale_fill_manual(values = region_colors) +
  coord_flip() + 
  theme_dviz_vgrid(12, rel_small = 1) +
  theme(axis.title.y = element_blank(),
        axis.line.y = element_blank(),
        axis.ticks.length = unit(0, "pt"),
        axis.text.y = element_text(size = 10, color = state_colors),
        legend.position = c(.56, .68),
        #legend.text = element_text(color = region_colors),
        legend.background = element_rect(fill = "#ffffffb0"))
```



## Color to represent data values

Color can also be used to represent data values, such as income, temperature, or speed. In this case, we use a *sequential* color scale. Such a scale contains a sequence of colors that clearly indicate (i) which values are larger or smaller than which other ones and (ii) how distant two specific values are from each other. The second point implies that the color scale needs to be perceived to vary uniformly across its entire range.

Sequential scales can be based on a single hue (e.g., from dark blue to light blue) or on multiple hues (e.g., from dark red to light yellow) (Figure \@ref(fig:sequential-scales)). Multi-hue scales tend to follow color gradients that can be seen in the natural world, such as dark red, green, or blue to light yellow, or dark purple to light green. The reverse, e.g. dark yellow to light blue, looks unnatural and doesn't make a useful sequential scale.

(ref:sequential-scales) Example sequential color scales. The ColorBrewer Blues scale is a monochromatic scale that varies from dark to light blue. The Heat and Viridis scales are multi-hue scales that vary from dark red to light yellow and from dark blue via green to light yellow, respectively. 

```{r sequential-scales, fig.width=5*6/4.2, fig.asp=3*.14, fig.cap = '(ref:sequential-scales)'}
p1 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_brewer(type = "seq", palette = "Blues", direction = -1) + ggtitle("ColorBrewer Blues") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
p2 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_discrete_sequential("Heat", rev = FALSE) + ggtitle("Heat") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
p3 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_viridis_d() + ggtitle("Viridis") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
plot_grid(p1, p2, p3, ncol = 1)
```

Representing data values as colors is particularly useful when we want to show how the data values vary across geographic regions. In this case, we can draw a map of the geographic regions and color them by the data values. Such maps are called *choropleths*. Figure \@ref(fig:map-Texas-income) shows an example where I have mapped annual median income within each county in Texas onto a map of those counties. 

(ref:map-Texas-income) Median annual income in Texas counties. The highest median incomes are seen in major Texas metropolitan areas, in particular near Houston and Dallas. No median income estimate is available for Loving County in West Texas and therefore that county is shown in gray. Data source: 2015 Five-Year American Community Survey


```{r map-Texas-income, fig.width = 6, fig.asp = 0.75, fig.cap = '(ref:map-Texas-income)'}
# B19013_001: Median household income in the past 12 months (in 2015 Inflation-adjusted dollars)

# EPSG:3083
# NAD83 / Texas Centric Albers Equal Area
# http://spatialreference.org/ref/epsg/3083/
texas_crs <- "+proj=aea +lat_1=27.5 +lat_2=35 +lat_0=18 +lon_0=-100 +x_0=1500000 +y_0=6000000 +ellps=GRS80 +datum=NAD83 +units=m +no_defs"

# -110, -93.5 transformed using texas_crs
texas_xlim <- c(558298.7, 2112587)

texas_income %>% st_transform(crs = texas_crs) %>%
  ggplot(aes(fill = estimate)) + 
  geom_sf(color = "white") + 
  coord_sf(xlim = texas_xlim, datum = NA) +
  theme_dviz_map() + 
  scale_fill_distiller(
    palette = "Blues", type = 'seq', na.value = "grey60", direction = 1,
    name = "annual median income (USD)",
    limits = c(18000, 90000),
    breaks = 20000*c(1:4),
    labels = c("$20,000", "$40,000", "$60,000", "$80,000"),
    guide = guide_colorbar(
      direction = "horizontal",
      label.position = "bottom",
      title.position = "top",
      ticks = FALSE,
      barwidth = grid::unit(3.0, "in"),
      barheight = grid::unit(0.2, "in")
    )
  ) +
  theme(
    legend.title.align = 0.5,
    legend.text.align = 0.5,
    legend.justification = c(0, 0),
    legend.position = c(0.02, 0.1)
  )
```

In some cases, we need to visualize the deviation of data values in one of two directions relative to a neutral midpoint. One straightforward example is a dataset containing both positive and negative numbers. We may want to show those with different colors, so that it is immediately obvious whether a value is positive or negative as well as how far in either direction it deviates from zero. The appropriate color scale in this situation is a *diverging* color scale. We can think of a diverging scale as two sequential scales stitched together at a common midpoint, which usually is represented by a light color (Figure \@ref(fig:diverging-scales)). Diverging scales need to be balanced, so that the progression from light colors in the center to dark colors on the outside is approximately the same in either direction. Otherwise, the perceived magnitude of a data value would depend on whether it fell above or below the midpoint value.

(ref:diverging-scales) Example diverging color scales. Diverging scales can be thought of as two sequential scales stitched together at a common midpoint color. Common color choices for diverging scales include brown to greenish blue, pink to yellow-green, and blue to red.

```{r diverging-scales, fig.width=5*6/4.2, fig.asp=3*.14, fig.cap = '(ref:diverging-scales)'}
p1 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_discrete_divergingx(palette = "Earth") + ggtitle("CARTO Earth") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
p2 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_brewer(type = "div", palette = "PiYG") + ggtitle("ColorBrewer PiYG") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
p3 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_discrete_diverging("Blue-Red") + ggtitle("Blue-Red") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
plot_grid(p1, p2, p3, ncol = 1)
```

As an example application of a diverging color scale, consider Figure \@ref(fig:map-Texas-race), which shows the percentage of people identifying as white in Texas counties. Even though percentage is always a positive number, a diverging scale is justified here, because 50% is a meaningful midpoint value. Numbers above 50% indicate that whites are in the majority and numbers below 50% indicate the opposite. The visualization clearly shows in which counties whites are in the majority, in which they are in the minority, and in which whites and non-whites occur in approximately equal proportions.

(ref:map-Texas-race) Percentage of people identifying as white in Texas counties. Whites are in the majority in North and East Texas but not in South or West Texas. Data source: 2010 Decennial U.S. Census

```{r map-Texas-race, fig.width = 6, fig.asp = 0.75, fig.cap = '(ref:map-Texas-race)'}
texas_race %>% st_sf() %>%
  st_transform(crs = texas_crs) %>%
  filter(variable == "White") %>%
  ggplot(aes(fill = pct)) +
  geom_sf(color = "white") +
  coord_sf(xlim = texas_xlim, datum = NA) + 
  theme_dviz_map() +
  scale_fill_continuous_divergingx(
    palette = "Earth",
    mid = 50,
    limits = c(0, 100),
    breaks = 25*(0:4),
    labels = c("0% ", "25%", "50%", "75%", " 100%"),
    name = "percent identifying as white",
    guide = guide_colorbar(
      direction = "horizontal",
      label.position = "bottom",
      title.position = "top",
      ticks = FALSE,
      barwidth = grid::unit(3.0, "in"),
      barheight = grid::unit(0.2, "in"))) +
  theme(
    legend.title.align = 0.5,
    legend.text.align = 0.5,
    legend.justification = c(0, 0),
    legend.position = c(0.02, 0.1)
  )
```



## Color as a tool to highlight

Color can also be an effective tool to highlight specific elements in the data. There may be specific categories or values in the dataset that carry key information about the story we want to tell, and we can strengthen the story by emphasizing the relevant figure elements to the reader. An easy way to achieve this emphasis is to color these figure elements in a color or set of colors that vividly stand out against the rest of the figure. This effect can be achieved with *accent* color scales, which are color scales that contain both a set of subdued colors and a matching set of stronger, darker, and/or more saturated colors (Figure \@ref(fig:accent-scales)).

(ref:accent-scales) Example accent color scales, each with four base colors and three accent colors. Accent color scales can be derived in several different ways: (top) we can take an existing color scale (e.g., the Okabe Ito scale, Fig \@ref(fig:qualitative-scales)) and lighten and/or partially desaturate some colors while darkening others; (middle) we can take gray values and pair them with colors; (bottom) we can use an existing accent color scale, e.g. the one from the ColorBrewer project.

```{r accent-scales, fig.width=5*6/4.2, fig.asp=3*.14, fig.cap = '(ref:accent-scales)'}
accent_OkabeIto <- palette_OkabeIto[c(1, 2, 7, 4, 5, 3, 6)]
accent_OkabeIto[1:4] <- desaturate(lighten(accent_OkabeIto[1:4], .4), .8)
accent_OkabeIto[5:7] <- darken(accent_OkabeIto[5:7], .3)


p1 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_manual(values = accent_OkabeIto) + ggtitle("Okabe Ito Accent") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
p2 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_manual(values = c("gray60", "gray70","gray80", "gray90", "#C95C4F",   '#83A121', '#6B8AD5')) + ggtitle("Grays with accents") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
p3 <- gg_color_swatches(7, title_family = dviz_font_family) + 
  scale_fill_brewer(type = "qual", palette = "Accent") + ggtitle("ColorBrewer Accent") +
  theme(plot.margin = margin(7, 1.5, 7, 1.5))
plot_grid(p1, p2, p3, ncol = 1)
```

As an example of how the same data can support differing stories with different coloring approaches, I have created a variant of Figure \@ref(fig:popgrowth-US) where now I highlight two specific states, Texas and Louisiana (Figure \@ref(fig:popgrowth-US-highlight)). Both states are in the South, they are immediate neighbors, and yet one state (Texas) was the fifth-fastest growing state within the U.S. whereas the other was the third slowest growing from 2000 to 2010.

(ref:popgrowth-US-highlight) From 2000 to 2010, the two neighboring southern states Texas and Louisiana have experienced among the highest and lowest population growth across the U.S. Data source: U.S. Census Bureau

```{r popgrowth-US-highlight, fig.width = 6, fig.asp = 1.2, fig.cap = '(ref:popgrowth-US-highlight)'}
popgrowth_hilight <- left_join(US_census, US_regions) %>%
    group_by(region, division, state) %>%
    summarize(pop2000 = sum(pop2000, na.rm = TRUE),
              pop2010 = sum(pop2010, na.rm = TRUE),
              popgrowth = (pop2010-pop2000)/pop2000,
              area = sum(area)) %>%
    arrange(popgrowth) %>%
    ungroup() %>%
    mutate(region = ifelse(state %in% c("Texas", "Louisiana"), "highlight", region)) %>%
    mutate(state = factor(state, levels = state),
           region = factor(region, levels = c("West", "South", "Midwest", "Northeast", "highlight")))

# make color and fontface vector in order of the states
region_colors_bars <- c(desaturate(lighten(c("#E69F00", "#56B4E9", "#009E73", "#F0E442"), .4), .8), darken("#56B4E9", .3))
region_colors_axis <- c(rep("gray30", 4), darken("#56B4E9", .4))
region_fontface <- c(rep("plain", 4), "bold")
state_colors <- region_colors_axis[as.numeric(popgrowth_hilight$region[order(popgrowth_hilight$state)])]
state_fontface <- region_fontface[as.numeric(popgrowth_hilight$region[order(popgrowth_hilight$state)])]

ggplot(popgrowth_hilight, aes(x = state, y = 100*popgrowth, fill = region)) + 
  geom_col() + 
  scale_y_continuous(
    limits = c(-.6, 37.5), expand = c(0, 0),
    labels = scales::percent_format(accuracy = 1, scale = 1),
    name = "population growth, 2000 to 2010"
  ) +
  scale_fill_manual(
    values = region_colors_bars,
    breaks = c("West", "South", "Midwest", "Northeast")
  ) +
  coord_flip() + 
  theme_dviz_vgrid(12, rel_small = 1) +
  theme(
    text = element_text(color = "gray30"),
    axis.text.x = element_text(color = "gray30"),
    axis.title.y = element_blank(),
    axis.line.y = element_blank(),
    axis.ticks.length = unit(0, "pt"),
    axis.text.y = element_text(
      size = 10, color = state_colors,
      face = state_fontface
    ),
    legend.position = c(.56, .68),
    legend.background = element_rect(fill = "#ffffffb0")
  )
```

When working with accent colors, it is critical that the baseline colors do not compete for attention. Notice how drab the baseline colors are in (Figure \@ref(fig:popgrowth-US-highlight)). Yet they work well to support the accent color. It is easy to make the mistake of using baseline colors that are too colorful, so that they end up competing for the reader's attention against the accent colors. There is an easy remedy, however. Just remove all color from all elements in the figure except the highlighted data categories or points. An example of this strategy is provided in Figure \@ref(fig:Aus-athletes-track).


(ref:Aus-athletes-track) Track athletes are among the shortest and leanest of male professional athletes participating in popular sports. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-track, fig.width = 6, fig.cap='(ref:Aus-athletes-track)'}
male_Aus <- filter(Aus_athletes, sex=="m") %>%
  filter(sport %in% c("basketball", "field", "swimming", "track (400m)",
                      "track (sprint)", "water polo")) %>%
  mutate(sport = case_when(sport == "track (400m)" ~ "track",
                           sport == "track (sprint)" ~ "track",
                           TRUE ~ sport))

male_Aus$sport <- factor(male_Aus$sport,
                         levels = c("track", "field", "water polo", "basketball", "swimming"))

colors <- c("#BD3828", rep("#808080", 4))
fills <- c("#BD3828D0", rep("#80808080", 4))

ggplot(male_Aus, aes(x=height, y=pcBfat, shape=sport, color = sport, fill = sport)) +
  geom_point(size = 3) +
  scale_shape_manual(values = 21:25) +
  scale_color_manual(values = colors) +
  scale_fill_manual(values = fills) +
  xlab("height (cm)") +
  ylab("% body fat") +
  theme_dviz_grid()
```



<!--chapter:end:color_basics.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(lubridate)
library(forcats)
library(tidyr)
library(ggrepel)
```

# Coordinate systems and axes {#coordinate-systems-axes}

To make any sort of data visualization, we need to define position scales, which determine where in a graphic different data values are located. We cannot visualize data without placing different data points at different locations, even if we just arrange them next to each other along a line. For regular 2d visualizations, two numbers are required to uniquely specify a point, and therefore we need two position scales. These two scales are usually but not necessarily the *x* and *y* axis of the plot. We also have to specify the relative geometric arrangement of these scales. Conventionally, the *x* axis runs horizontally and the *y* axis vertically, but we could choose other arrangements. For example, we could have the *y* axis run at an acute angle relative to the *x* axis, or we could have one axis run in a circle and the other run radially. The combination of a set of position scales and their relative geometric arrangement is called a *coordinate system.*


## Cartesian coordinates

The most widely used coordinate system for data visualization is the 2d *Cartesian coordinate system*, where each location is uniquely specified by an *x* and a *y* value. The *x* and *y* axes run orthogonally to each other, and data values are placed in an even spacing along both axes (Figure \@ref(fig:cartesian-coord)). The two axes are continuous position scales, and they can represent both positive and negative real numbers. To fully specify the coordinate system, we need to specify the range of numbers each axis covers. In Figure \@ref(fig:cartesian-coord), the *x* axis runs from -2.2 to 3.2 and the *y* axis runs from -2.2 to 2.2. Any data values between these axis limits are placed at the respective location in the plot. Any data values outside the axis limits are discarded. 

(ref:cartesian-coord) Standard cartesian coordinate system. The horizontal axis is conventionally called *x* and the vertical axis *y*. The two axes form a grid with equidistant spacing. Here, both the *x* and the *y* grid lines are separated by units of one. The point (2, 1) is located two *x* units to the right and one *y* unit above the origin (0, 0). The point (-1, -1) is located one *x* unit to the left and one *y* unit below the origin. 

```{r cartesian-coord, fig.asp = 0.8, fig.cap = '(ref:cartesian-coord)'}
df_points <- data.frame(x = c(-1, 0, 2),
                        y = c(-1, 0, 1),
                        label = c("(–1, –1)", "(0, 0)", "(2, 1)"),
                        vjust = c(1.4, -.8, -.8),
                        hjust = c(1.1, 1.1, -.1))

df_segments <- data.frame(x0 = c(0, 2, 0, -1),
                          x1 = c(2, 2, -1, -1),
                          y0 = c(1, 0, -1, 0),
                          y1 = c(1, 1, -1, -1))

df_labels <- data.frame(x = c(-1, -.5, 1, 2),
                        y = c(-.5, -1, 1, 0.5),
                        vjust = c(.5, 1.3, -.3, .5),
                        hjust = c(1.1, .5, .5, -.1),
                        label = c("y = –1", "x = –1", "x = 2", "y = 1"))

ggplot(df_points, aes(x, y)) +
  geom_hline(yintercept = 0, color = "gray50") +
  geom_vline(xintercept = 0, color = "gray50") +
  geom_segment(data = df_segments, aes(x = x0, xend = x1, y = y0, yend = y1),
               linetype = 2) +
  geom_point(size = 3, color = "#0072B2") +
  geom_text(aes(label = label, vjust = vjust, hjust = hjust),
            size = 12/.pt, family = dviz_font_family) +
  geom_text(data = df_labels, aes(label = label, hjust = hjust, vjust = vjust),
            size = 12/.pt, family = dviz_font_family) +
  coord_fixed(xlim = c(-2.2, 3.2), ylim = c(-2.2, 2.2), expand = FALSE) +
  xlab("x axis") +
  ylab("y axis") +
  theme_dviz_grid() +
  theme(axis.ticks.length = grid::unit(0, "pt"))
```

Data values usually aren't just numbers, however. They come with units. For example, if we're measuring temperature, the values may be measured in degrees Celsius or Fahrenheit. Similarly, if we're measuring distance, the values may be measured in kilometers or miles, and if we're measuring duration, the values may be measured in minutes, hours, or days. In a Cartesian coordinate system, the spacing between grid lines along an axis corresponds to discrete steps in these data units. In a temperature scale, for example, we may have a grid line every 10 degrees Fahrenheit, and in a distance scale, we may have a grid line every 5 kilometers.

A Cartesian coordinate system can have two axes representing two different units. This situation arises quite commonly whenever we're mapping two different types of variables to *x* and *y*. For example, in Figure \@ref(fig:temp-normals-vs-time), we plotted temperature vs. days of the year. The *y* axis of Figure \@ref(fig:temp-normals-vs-time) is measured in degrees Fahrenheit, with a grid line every at 20 degrees, and the *x* axis is measured in months, with a grid line at the first of every third month. Whenever the two axes are measured in different units, we can stretch or compress one relative to the other and maintain a valid visualization of the data (Figure \@ref(fig:temperature-normals-Houston)). Which version is preferable may depend on the story we want to convey. A tall and narrow figure emphasizes change along the *y* axis and a short and wide figure does the opposite. Ideally, we want to choose an aspect ratio that ensures that any important differences in position are noticeable.

(ref:temperature-normals-Houston) Daily temperature normals for Houston, TX. Temperature is mapped to the *y* axis and day of the year to the *x* axis. Parts (a), (b), and (c) show the same figure in different aspect ratios. All three parts are valid visualizations of the temperature data. Data source: NOAA.

```{r temperature-normals-Houston, fig.width = 5*6/4.2, fig.asp = 3/4, fig.cap = '(ref:temperature-normals-Houston)'}
temps_wide <- filter(ncdc_normals,
                station_id %in% c(
                  "USW00014819", # Chicago, IL 60638
                  "USC00516128", # Honolulu, HI 96813
                  "USW00027502", # Barrow, AK 99723, coldest point in the US
                  "USC00042319", # Death Valley, CA 92328 hottest point in the US
                  "USW00093107", # San Diego, CA 92145
                  "USW00012918", # Houston, TX 77061
                  "USC00427606"  # Salt Lake City, UT 84103
                )) %>%
  mutate(location = fct_recode(factor(station_id),
                               "Chicago" = "USW00014819",
                               "Honolulu" = "USC00516128",
                               "Barrow, AK" = "USW00027502",
                               "Death Valley" = "USC00042319",
                               "San Diego" = "USW00093107",
                               "Houston" = "USW00012918",
                               "Salt Lake City, UT" = "USC00427606")) %>%
  select(-station_id, -flag) %>%
  spread(location, temperature) %>%
  arrange(date)

temps_wide_label <- mutate(
  temps_wide,
  label = ifelse(
    date %in% c(ymd("0000-01-01"), ymd("0000-04-01"), ymd("0000-07-01"), ymd("0000-10-01")),
    format(date, "%b 1st"),
    ""
  ),
  nudge_x = ifelse(
    date %in% c(ymd("0000-01-01"), ymd("0000-04-01"), ymd("0000-07-01"), ymd("0000-10-01")),
    c(-1, -2, -2, 1)[round(month(date)/3)+1],
    0
  ),
  nudge_y = ifelse(
    date %in% c(ymd("0000-01-01"), ymd("0000-04-01"), ymd("0000-07-01"), ymd("0000-10-01")),
    c(-2, 1, 0.5, -2)[round(month(date)/3)+1],
    0
  )
)

temp_plot <- ggplot(temps_wide_label, aes(x = date, y = `Houston`)) +
  geom_line(size = 1, color = "#0072B2") +
  scale_x_date(name = "month", limits = c(ymd("0000-01-01"), ymd("0001-01-03")),
               breaks = c(ymd("0000-01-01"), ymd("0000-04-01"), ymd("0000-07-01"),
                          ymd("0000-10-01"), ymd("0001-01-01")),
               labels = c("Jan", "Apr", "Jul", "Oct", "Jan"), expand = c(2/366, 0)) + 
  scale_y_continuous(limits = c(50, 90),
                     name = "temperature (°F)") +
  theme_dviz_grid(12) +
  theme(plot.margin = margin(3, 5, 3, 1.5))

plot_grid(
  plot_grid(
    temp_plot, NULL, temp_plot, rel_widths = c(1, 0.06, 2), labels = c("a", "", "b"), nrow = 1
  ),
  NULL, temp_plot,
  rel_heights = c(1.5, 0.06, 1), labels = c("", "", "c"), label_y = c(1, 1, 1.03), ncol = 1
)
```

On the other hand, if the *x* and the *y* axes are measured in the same units, then the grid spacings for the two axes should be equal, such that the same distance along the *x* or *y* axis corresponds to the same number of data units. As an example, we can plot the temperature in Houston, TX against the temperature in San Diego, CA, for every day of the year (Figure \@ref(fig:temperature-normals-Houston-San-Diego)a). Since the same quantity is plotted along both axes, we need to make sure that the grid lines form perfect squares, as is the case in Figure \@ref(fig:temperature-normals-Houston-San-Diego).

(ref:temperature-normals-Houston-San-Diego) Daily temperature normals for Houston, TX, plotted versus the respective temperature normals of San Diego, CA. The first days of the months January, April, July, and October are highlighted to provide a temporal reference. (a) Temperatures are shown in degrees Fahrenheit. (b) Temperatures are shown in degrees Celsius. Data source: NOAA.

```{r temperature-normals-Houston-San-Diego, fig.width = 5.5*6/4.2, fig.asp = 0.5, fig.cap = '(ref:temperature-normals-Houston-San-Diego)'}
tempsplot_F <- ggplot(temps_wide_label, aes(x = `San Diego`, y = `Houston`)) +
  geom_path(size = 1, color = "#0072B2") +
  geom_text_repel(
    aes(label = label), point.padding = .4, color = "black",
    min.segment.length = 0, size = 12/.pt,
    family = dviz_font_family,
    nudge_x = (9/5)*temps_wide_label$nudge_x,
    nudge_y = (9/5)*temps_wide_label$nudge_y
  ) +
  coord_fixed(
    xlim = c(45, 85), ylim = c(48, 88),
    expand = FALSE
  ) +
  scale_color_continuous_qualitative(guide = "none") +
  scale_x_continuous(breaks = c(10*(5:8))) +
  xlab("temperature in San Diego (°F)") +
  ylab("temperature in Houston (°F)") +
  theme_dviz_grid() +
  theme(plot.margin = margin(3, 1.5, 3, 1.5))

# Fahrenheit to Celsius conversion

F2C <- function(t) {(t-32)*5/9}

tempsplot_C <- ggplot(temps_wide_label, aes(x = F2C(`San Diego`), y = F2C(`Houston`))) +
  geom_path(size = 1, color = "#0072B2") +
  geom_text_repel(
    aes(label = label), point.padding = .4, color = "black",
    min.segment.length = 0, size = 12/.pt,
    family = dviz_font_family,
    nudge_x = temps_wide_label$nudge_x,
    nudge_y = temps_wide_label$nudge_y
  ) +
  coord_fixed(
    xlim = F2C(c(45, 85)), ylim = F2C(c(48, 88)),
    expand = FALSE
  ) +
  scale_color_continuous_qualitative(guide = "none") +
  scale_x_continuous(breaks = c(5*(2:6))) +
  xlab("temperature in San Diego (°C)") +
  ylab("temperature in Houston (°C)") +
  theme_dviz_grid() +
  theme(plot.margin = margin(3, 1.5, 3, 1.5))


plot_grid(
  tempsplot_F, NULL, tempsplot_C,
  labels = c("a", "", "b"), nrow = 1, rel_widths = c(1, .04, 1)
)
```

You may wonder what happens if you change the units of your data. After all, units are arbitrary, and your preferences might be different from somebody else's. A change in units is a linear transformation, where we add or subtract a number to or from all data values and/or multiply all data values with another number. Fortunately, Cartesian coordinate systems are invariant under such linear transformations. Therefore, you can change the units of your data and the resulting figure will not change as long as you change the axes accordingly. As an example, compare Figures \@ref(fig:temperature-normals-Houston-San-Diego)a and \@ref(fig:temperature-normals-Houston-San-Diego)b. Both show the same data, but in part (a) the temperature units are degrees Fahrenheit and in part (b) they are degrees Celsius. Even though the grid lines are in different locations and the numbers along the axes are different, the two data visualizations look exactly the same.


## Nonlinear axes

In a Cartesian coordinate system, the grid lines along an axis are spaced evenly both in data units and in the resulting visualization. We refer to the position scales in these coordinate systems as *linear*. While linear scales generally provide an accurate representation of the data, there are scenarios where nonlinear scales are preferred. In a nonlinear scale, even spacing in data units corresponds to uneven spacing in the visualization, or conversely even spacing in the visualization corresponds to uneven spacing in data units.

The most commonly used nonlinear scale is the *logarithmic scale* or *log scale* for short. Log scales are linear in multiplication, such that a unit step on the scale corresponds to multiplication with a fixed value. To create a log scale, we need to log-transform the data values while exponentiating the numbers that are shown along the axis grid lines. This process is demonstrated in Figure \@ref(fig:linear-log-scales), which shows the numbers 1, 3.16, 10, 31.6, and 100 placed on linear and log scales. The numbers 3.16 and 31.6 may seem a strange choice, but they were chosen because they are exactly half-way between 1 and 10 and between 10 and 100 on a log scale. We can see this by observing that $10^{0.5} = \sqrt{10} \approx 3.16$ and equivalently $3.16 \times 3.16 \approx 10$. Similarly, $10^{1.5} = 10\times10^{0.5} \approx 31.6$.

(ref:linear-log-scales) Relationship between linear and logarithmic scales. The dots correspond to data values 1, 3.16, 10, 31.6, 100, which are evenly-spaced numbers on a logarithmic scale. We can display these data points on a linear scale, we can log-transform them and then show on a linear scale, or we can show them on a logarithmic scale. Importantly, the correct axis title for a logarithmic scale is the name of the variable shown, not the logarithm of that variable.

```{r linear-log-scales, fig.width = 6, fig.asp = 3/4, fig.cap = '(ref:linear-log-scales)'}
df <- data.frame(x = c(1, 3.16, 10, 31.6, 100))

xaxis_lin <- ggplot(df, aes(x, y = 1)) + 
  geom_point(size = 3, color = "#0072B2") + 
  scale_y_continuous(limits = c(0.8, 1.2), expand = c(0, 0), breaks = 1) +
  theme_dviz_grid(14, rel_large = 1) +
  theme(axis.ticks.length = grid::unit(0, "pt"),
        axis.text.y = element_blank(),
        axis.title.y = element_blank(),
        axis.ticks.y = element_blank(),
        plot.title = element_text(face = "plain"),
        plot.margin = margin(3, 14, 3, 1.5))

xaxis_log <- ggplot(df, aes(log10(x), y = 1)) + 
  geom_point(size = 3, color = "#0072B2") + 
  scale_y_continuous(limits = c(0.8, 1.2), expand = c(0, 0), breaks = 1) +
  theme_dviz_grid(14, rel_large = 1) +
  theme(axis.ticks.length = grid::unit(0, "pt"),
        axis.text.y = element_blank(),
        axis.title.y = element_blank(),
        axis.ticks.y = element_blank(),
        plot.title = element_text(face = "plain"),
        plot.margin = margin(3, 14, 3, 1.5))

plotlist <- 
  align_plots(xaxis_lin + scale_x_continuous(limits = c(0, 100)) + 
                ggtitle("original data, linear scale"),
              xaxis_log + scale_x_continuous(limits = c(0, 2)) +
                xlab(expression(paste("log"["10"], "(x)"))) + 
                ggtitle("log-transformed data, linear scale"),
              xaxis_lin + scale_x_log10(limits = c(1, 100), breaks = c(1, 3.16, 10, 31.6, 100),
                                        labels = c("1", "3.16", "10", "31.6", "100")) + 
                ggtitle("original data, logarithmic scale"),
              xaxis_lin + scale_x_log10(limits = c(1, 100), breaks = c(1, 3.16, 10, 31.6, 100),
                                        labels = c("1", "3.16", "10", "31.6", "100")) +
                xlab(expression(paste("log"["10"], "(x)"))) + 
                ggtitle("logarithmic scale with incorrect axis title"),
              align = 'vh')

plot_grid(plotlist[[1]], plotlist[[2]], plotlist[[3]], stamp_wrong(plotlist[[4]]), ncol = 1)

```

Mathematically, there is no difference between plotting the log-transformed data on a linear scale or plotting the original data on a logarithmic scale (Figure \@ref(fig:linear-log-scales)). The only difference lies in the labeling for the individual axis ticks and for the axis as a whole. In most cases, the labeling for a logarithmic scale is preferable, because it places less mental burden on the reader to interpret the numbers shown as the axis tick labels. There is also less of a risk of confusion about the base of the logarithm. When working with log-transformed data, we can get confused about whether the data were transformed using the natural logarithm or the logarithm to base 10. And it's not uncommon for labeling to be ambiguous, e.g. "log(x)", which doesn't specify a base at all. I recommend that you always verify the base when working with log-transformed data. When plotting log-transformed data, always specify the base in the labeling of the axis.

Because multiplication on a log scale looks like addition on a linear scale, log scales are the natural choice for any data that have been obtained by multiplication or division. In particular, ratios should generally be shown on a log scale. As an example, I have taken the number of inhabitants in each county in Texas and have divided it by the median number of inhabitants across all Texas counties. The resulting ratio is a number that can be larger or smaller than 1. A ratio of exactly 1 implies that the corresponding county has the median number of inhabitants. When visualizing these ratios on a log scale, we can see clearly that the population numbers in Texas counties are symmetrically distributed around the median, and that the most populous counties have over 100 times more inhabitants than the median while the least populous counties have over 100 times fewer inhabitants (Figure \@ref(fig:texas-counties-pop-ratio-log)). By contrast, for the same data, a linear scale obscures the differences between a county with median population number and a county with a much smaller population number than median (Figure \@ref(fig:texas-counties-pop-ratio-lin)).

(ref:texas-counties-pop-ratio-log) Population numbers of Texas counties relative to their median value. Select counties are highlighted by name. The dashed line indicates a ratio of 1, corresponding to a county with median population number. The most populous counties have approximately 100 times more inhabitants than the median county, and the least populous counties have approximately 100 times fewer inhabitants than the median county. Data source: 2010 Decennial U.S. Census.

```{r texas-counties-pop-ratio-log, fig.width = 5*6/4.2, fig.asp = 0.6, fig.cap = '(ref:texas-counties-pop-ratio-log)'}
set.seed(3878)
US_census %>% filter(state == "Texas") %>%
  select(name, pop2010) %>%
  extract(name, "county", regex = "(.+) County") %>%
  mutate(popratio = pop2010/median(pop2010)) %>%
  arrange(desc(popratio)) %>%
  mutate(index = 1:n(),
         label = ifelse(index <= 3 | index > n()-3 | runif(n()) < .04, county, ""),
         label_large = ifelse(index <= 6, county, "")) -> tx_counties

ggplot(tx_counties, aes(x = index, y = popratio)) +
  geom_hline(yintercept = 1, linetype = 2, color = "grey40") +
  geom_point(size = 0.5, color = "#0072B2") +
  geom_text_repel(aes(label = label), point.padding = .4, color = "black",
                  min.segment.length = 0, family = dviz_font_family) +
  scale_y_log10(breaks = c(.01, .1, 1, 10, 100),
                name = "population number / median",
                labels = label_log10) +
  scale_x_continuous(limits = c(.5, nrow(tx_counties) + .5), expand = c(0, 0),
                     breaks = NULL, #c(1, 50*(1:5)),
                     name = "Texas counties, from most to least populous") +
  theme_dviz_hgrid() +
  theme(axis.line = element_blank(),
        plot.margin = margin(3, 7, 3, 1.5))
```

(ref:texas-counties-pop-ratio-lin) Population sizes of Texas counties relative to their median value. By displaying a ratio on a linear scale, we have overemphasized ratios > 1 and have obscured ratios < 1. As a general rule, ratios should not be displayed on a linear scale. Data source: 2010 Decennial U.S. Census.

```{r texas-counties-pop-ratio-lin, fig.width = 5*6/4.2, fig.asp = 0.6, fig.cap = '(ref:texas-counties-pop-ratio-lin)'}
counties_lin <- ggplot(tx_counties, aes(x = index, y = popratio)) +
  geom_point(size = 0.5, color = "#0072B2") +
  geom_text_repel(aes(label = label_large), point.padding = .4, color = "black",
                  min.segment.length = 0, family = dviz_font_family) +
  scale_y_continuous(name = "population number / median") +
  scale_x_continuous(limits = c(.5, nrow(tx_counties) + .5), expand = c(0, 0),
                     breaks = NULL, #c(1, 50*(1:5)),
                     name = "Texas counties, from most to least populous") +
  theme_dviz_hgrid() +
  theme(axis.line = element_blank(),
        plot.margin = margin(3, 7, 3, 1.5))

stamp_bad(counties_lin)
```

On a log scale, the value 1 is the natural midpoint, similar to the value 0 on a linear scale. We can think of values greater than 1 as representing multiplications and values less than 1 divisions. For example, we can write $10 = 1\times 10$ and $0.1 = 1/10$. The value 0, on the other hand, can never appear on a log scale. It lies infinitely far from 1. One way to see this is to consider that $\log(0) = -\infty$. Or, alternatively, consider that to go from 1 to 0, it takes either an infinite number of divisions by a finite value (e.g., $1/10/10/10/10/10/10\dots = 0$) or alternatively one division by infinity (i.e., $1/\infty = 0$).

Log scales are frequently used when the data set contains numbers of very different magnitudes. For the Texas counties shown in Figures \@ref(fig:texas-counties-pop-ratio-log) and \@ref(fig:texas-counties-pop-ratio-lin), the most populous one (Harris) had 4,092,459 inhabitants in the 2010 U.S. Census while the least populous one (Loving) had 82. So a log scale would be appropriate even if we hadn't divided population numbers by their median to turn them into ratios. But what would we do if there was a county with 0 inhabitants? This county could not be shown on the logarithmic scale, because it would lie at minus infinity. In this situation, the recommendation is sometimes to use a square-root scale, which uses a square root transformation instead of a log transformation (Figure \@ref(fig:sqrt-scales)). Just like a log scale, a square-root scale compresses larger numbers into a smaller range, but unlike a log scale, it allows for the presence of 0.

(ref:sqrt-scales) Relationship between linear and square-root scales. The dots correspond to data values 0, 1, 4, 9, 16, 25, 36, 49, which are evenly-spaced numbers on a square-root scale, since they are the squares of the integers from 0 to 7. We can display these data points on a linear scale, we can square-root-transform them and then show on a linear scale, or we can show them on a square-root scale. 

```{r sqrt-scales, fig.width = 6, fig.asp = 3*(3/4)/4, fig.cap = '(ref:sqrt-scales)'}
df <- data.frame(x = c(0, 1, 4, 9, 16, 25, 36, 49))

xaxis_lin <- ggplot(df, aes(x, y = 1)) + 
  geom_point(size = 3, color = "#0072B2") + 
  scale_y_continuous(limits = c(0.8, 1.2), expand = c(0, 0), breaks = 1) +
  theme_dviz_grid(14, rel_large = 1) +
  theme(axis.ticks.length = grid::unit(0, "pt"),
        axis.text.y = element_blank(),
        axis.title.y = element_blank(),
        axis.ticks.y = element_blank(),
        plot.title = element_text(face = "plain"),
        plot.margin = margin(3, 14, 3, 1.5))

xaxis_sqrt <- ggplot(df, aes(sqrt(x), y = 1)) + 
  geom_point(size = 3, color = "#0072B2") + 
  scale_y_continuous(limits = c(0.8, 1.2), expand = c(0, 0), breaks = 1) +
  theme_dviz_grid(14, rel_large = 1) +
  theme(axis.ticks.length = grid::unit(0, "pt"),
        axis.text.y = element_blank(),
        axis.title.y = element_blank(),
        axis.ticks.y = element_blank(),
        plot.title = element_text(face = "plain"),
        plot.margin = margin(3, 14, 3, 1.5))

plotlist <- 
  align_plots(xaxis_lin + scale_x_continuous(limits = c(0, 50)) + 
                ggtitle("original data, linear scale"),
              xaxis_sqrt + scale_x_continuous(limits = c(0, 7.07)) +
                xlab(expression(sqrt(x))) + 
                ggtitle("square-root-transformed data, linear scale"),
              xaxis_sqrt + scale_x_continuous(limits = c(0, 7.07), breaks = c(0, 1, sqrt(5), sqrt(10*(1:5))),
                                              labels = c(0, 1, 5, 10*(1:5)), name = "x") + 
                expand_limits(expand = c(0, 1)) +
                ggtitle("original data, square-root scale"),
              align = 'vh')

plot_grid(plotlist[[1]], plotlist[[2]], plotlist[[3]], ncol = 1)
```

I see two problems with square-root scales. First, while on a linear scale one unit step corresponds to addition or subtraction of a constant value and on a log scale it corresponds to multiplication with or division by a constant value, no such rule exists for a square-root scale. The meaning of a unit step on a square-root scale depends on the scale value at which we're starting. Second, it is unclear how to best place axis ticks on a square-root scale. To obtain evenly spaced ticks, we would have to place them at squares, but axis ticks at, for example, positions 0, 4, 25, 49, 81 (every second square) would be highly unintuitive. Alternatively, we could place them at linear intervals (10, 20, 30, etc), but this would result in either too few axis ticks near the low end of the scale or too many near the high end. In Figure \@ref(fig:sqrt-scales), I have placed the axis ticks at positions 0, 1, 5, 10, 20, 30, 40, and 50 on the square-root scale. These values are arbitrary but provide a reasonable covering of the data range.

Despite these problems with square-root scales, they are valid position scales and I do not discount the possibility that they have appropriate applications. For example, just like a log scale is the natural scale for ratios, one could argue that the square-root scale is the natural scale for data that come in squares. One scenario in which data are naturally squares are in the context of geographic regions. If we show the areas of geographic regions on a square-root scale, we are highlighting the regions' linear extent from East to West or North to South. These extents could be relevant, for example, if we are wondering how long it might take to drive across a region. Figure \@ref(fig:northeast-state-areas) shows the areas of states in the U.S. Northeast on both a linear and a square-root scale. Even though the areas of these states are quite different (Figure \@ref(fig:northeast-state-areas)a), the time it will take to drive across each state will more closely resemble the figure on the square-root scale (Figure \@ref(fig:northeast-state-areas)b) than the figure on the linear scale (Figure \@ref(fig:northeast-state-areas)a).

(ref:northeast-state-areas) Areas of Northeastern U.S. states. (a) Areas shown on a linear scale. (b) Areas shown on a square-root scale. Data source: Google. 

```{r northeast-state-areas, fig.width = 5.5*6/4.2, fig.asp = 0.4, fig.cap = '(ref:northeast-state-areas)'}
# areas in square miles
# source: Google, 01/07/2018
northeast_areas <- read.csv(text = "state_abr,area
NY,54556
PA,46055
ME,35385
MA,10565
VT,9616
NH,9349
NJ,8723
CT,5543
RI,1212")

northeast_areas$state_abr <- factor(northeast_areas$state_abr, levels = northeast_areas$state_abr)

areas_base <- ggplot(northeast_areas, aes(x = state_abr, y = area)) +
  geom_col(fill = "#56B4E9") +
  ylab("area (square miles)") +
  xlab("state") +
  theme_dviz_hgrid() +
  theme(plot.margin = margin(3, 1.5, 3, 1.5))

p1 <- areas_base + scale_y_sqrt(limits = c(0, 55000), breaks = c(0, 1000, 5000, 10000*(1:5)), 
                                expand = c(0, 0))
  
p2 <- areas_base + scale_y_continuous(limits = c(0, 55000), breaks = 10000*(0:6), expand = c(0, 0))

plot_grid(
  p2, NULL, p1,
  labels = c("a", "", "b"), nrow = 1, rel_widths = c(1, .04, 1)
)
```

## Coordinate systems with curved axes

All coordinate systems we have encountered so far used two straight axes positioned at a right angle to each other, even if the axes themselves established a non-linear mapping from data values to positions. There are other coordinate systems, however, where the axes themselves are curved. In particular, in the *polar* coordinate system, we specify positions via an angle and a radial distance from the origin, and therefore the angle axis is circular (Figure \@ref(fig:polar-coord)).

(ref:polar-coord) Relationship between Cartesian and polar coordinates. (a) Three data points shown in a Cartesian coordinate system. (b) The same three data points shown in a polar coordinate system. We have taken the *x* coordinates from part (a) and used them as angular coordinates and the *y* coordinates from part (a) and used them as radial coordinates. The circular axis runs from 0 to 4 in this example, and therefore *x* = 0 and *x* = 4 are the same locations in this coordinate system. 

```{r polar-coord, fig.width = 5*6/4.2, fig.asp = 0.5, fig.cap = '(ref:polar-coord)'}
df_points <- data.frame(x = c(1, 3.5, 0),
                        y = c(3, 4, 0),
                        label = c("(1, 3)", "(3.5, 4)", "(0, 0)"),
                        vjust_polar = c(1.6, 1, 1.6),
                        hjust_polar = c(.5, -.1, 0.5),
                        vjust_cart = c(1.6, 1.6, -.6),
                        hjust_cart = c(0.5, 1.1, -.1))

df_segments <- data.frame(x0 = c(0, 1, 2, 3, 0, 0, 0, 0),
                          x1 = c(0, 1, 2, 3, 4, 4, 4, 4),
                          y0 = c(0, 0, 0, 0, 1, 2, 3, 4),
                          y1 = c(4, 4, 4, 4, 1, 2, 3, 4))


p_cart <- ggplot(df_points, aes(x, y)) +
  geom_point(size = 2, color = "#0072B2") +
  geom_text(aes(label = label, vjust = vjust_cart, hjust = hjust_cart),
            size = 12/.pt, family = dviz_font_family) +
  scale_x_continuous(limits = c(-0.5, 4.5), expand = c(0, 0)) +
  scale_y_continuous(limits = c(-0.5, 4.5), expand = c(0, 0)) +
  coord_fixed() +
  xlab("x axis") +
  ylab("y axis") +
  theme_dviz_grid(12) +
  theme(axis.ticks = element_blank(),
        axis.ticks.length = grid::unit(0, "pt"),
        plot.margin = margin(3, 1.5, 3, 1.5))


p_polar <- ggplot(df_points, aes(x, y)) +
  geom_segment(
    data = df_segments,
    aes(x = x0, xend = x1, y = y0, yend = y1),
    size = theme_dviz_grid()$panel.grid$size,
    color = theme_dviz_grid()$panel.grid$colour,
    inherit.aes = FALSE
  ) +
  geom_point(size = 2, color = "#0072B2") +
  geom_text(
    aes(label = label, vjust = vjust_polar, hjust = hjust_polar),
    size = 12/.pt, family = dviz_font_family
  ) +
  scale_x_continuous(limits = c(0, 4)) +
  scale_y_continuous(limits = c(0, 4)) +
  coord_polar() +
  xlab("x values (circular axis)") +
  ylab("y values (radial axis)") +
  theme_dviz_grid(12) +
  background_grid(major = "none") +
  theme(axis.line.x = element_blank(),
        axis.ticks = element_line(color = "black"),
        plot.margin = margin(3, 1.5, 3, 1.5))

plot_grid(
  p_cart, NULL, p_polar,
  labels = c("a", "", "b"), nrow = 1, rel_widths = c(1, .04, 1)
)
```

Polar coordinates can be useful for data of a periodic nature, such that data values at one end of the scale can be logically joined to data values at the other end. For example, consider the days in a year. December 31st is the last day of the year, but it is also one day before the first day of the year. If we want to show how some quantity varies over the year, it can be appropriate to use polar coordinates with the angle coordinate specifying each day. Let's apply this concept to the temperature normals of Figure \@ref(fig:temp-normals-vs-time). Because temperature normals are average temperatures that are not tied to any specific year, Dec. 31st can be thought of as 366 days later than Jan. 1st (temperature normals include Feb. 29) and also one day earlier. By plotting the temperature normals in a polar coordinate system, we emphasize this cyclical property they have (Figure \@ref(fig:temperature-normals-polar)). In comparison to Figure \@ref(fig:temp-normals-vs-time), the polar version highlights how similar the temperatures are in Death Valley, Houston, and San Diego from late fall to early spring. In the Cartesian coordinate system, this fact is obscured because the temperature values in late December and in early January are shown in opposite parts of the figure and therefore don't form a single visual unit.

(ref:temperature-normals-polar) Daily temperature normals for four selected locations in the U.S., shown in polar coordinates. The radial distance from the center point indicates the daily temperature in Fahrenheit, and the days of the year are arranged counter-clockwise starting with Jan. 1st at the 6:00 position.

```{r temperature-normals-polar, fig.width = 6, fig.cap = '(ref:temperature-normals-polar)'}
temps_long <- gather(temps_wide, location, temperature, -month, -day, -date) %>%
  filter(location %in% c("Chicago",
                         "Death Valley",
                         "Houston",
                         "San Diego")) %>%
  mutate(location = factor(location, levels = c("Death Valley",
                                                "Houston",
                                                "San Diego",
                                                "Chicago")))

ggplot(temps_long, aes(x = date, y = temperature, color = location)) +
  geom_line(size = 1) +
  scale_x_date(name = "date", expand = c(0, 0)) + 
  scale_y_continuous(limits = c(0, 105), expand = c(0, 0),
                     breaks = seq(-30, 90, by = 30),
                     name = "temperature (°F)") +
  scale_color_OkabeIto(order = c(1:3, 7), name = NULL) +
  coord_polar(theta = "x", start = pi, direction = -1) +
  theme_dviz_grid()
```


A second setting in which we encounter curved axes is in the context of geospatial data, i.e., maps. Locations on the globe are specified by their longitude and latitude. But because the earth is a sphere, drawing latitude and longitude as Cartesian axes is misleading and not recommended (Figure \@ref(fig:worldmap-four-projections)). Instead, we use various types of non-linear projections that attempt to minimize artifacts and that strike different balances between conserving areas or angles relative to the true shape lines on the globe (Figure \@ref(fig:worldmap-four-projections)).

(ref:worldmap-four-projections) Map of the world, shown in four different projections. The Cartesian longitude and latitude system maps the longitude and latitude of each location onto a regular Cartesian coordinate system. This mapping causes substantial distortions in both areas and angles relative to their true values on the 3D globe. The interrupted Goode homolosine projection perfectly represents true surface areas, at the cost of dividing some land masses into separate pieces, most notably Greenland and Antarctica. The Robinson projection and the Winkel tripel projection both strike a balance between angular and area distortions, and they are commonly used for maps of the entire globe.

```{r worldmap-four-projections, fig.width = 5.5*6/4.2, fig.cap = '(ref:worldmap-four-projections)'}
library(sf)

world_sf <- sf::st_as_sf(rworldmap::getMap(resolution = "low"))

## world in long-lat
crs_longlat <- "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"

p_longlat <- ggplot(world_sf) + 
  geom_sf(fill = "#E69F00B0", color = "black", size = 0.5/.pt) + 
  coord_sf(expand = FALSE, crs = crs_longlat) + 
  scale_x_continuous(
    name = "longitude",
    breaks = seq(-160, 160, by = 20),
    labels = parse(text = c("NA", "NA", "120*degree*W", "NA", "NA", "60*degree*W", "NA", "NA", "0*degree", "NA", "NA", "60*degree*E", "NA", "NA", "120*degree*E", "NA", "NA"))
  ) +
  scale_y_continuous(
    name = "latitude",
    breaks = seq(-80, 80, by = 20),
    labels = parse(text = c("80*degree*S", "NA", "40*degree*S", "NA", "0*degree", "NA", "40*degree*N", "NA", "80*degree*N"))
  ) +
  theme_dviz_grid(12) +
  theme(
    panel.background = element_rect(fill = "#56B4E950", color = "grey30", size = 0.5),
    panel.grid.major = element_line(color = "gray30", size = 0.25),
    axis.ticks = element_line(color = "gray30", size = 0.5/.pt),
    plot.margin = margin(5, 10, 1.5, 1.5)
  )


## Interrupted Goode homolosine
crs_goode <- "+proj=igh"

# projection outline in long-lat coordinates
lats <- c(
  90:-90, # right side down
  -90:0, 0:-90, # third cut bottom
  -90:0, 0:-90, # second cut bottom
  -90:0, 0:-90, # first cut bottom
  -90:90, # left side up
  90:0, 0:90, # cut top
  90 # close
)
longs <- c(
  rep(180, 181), # right side down
  rep(c(80.01, 79.99), each = 91), # third cut bottom
  rep(c(-19.99, -20.01), each = 91), # second cut bottom
  rep(c(-99.99, -100.01), each = 91), # first cut bottom
  rep(-180, 181), # left side up
  rep(c(-40.01, -39.99), each = 91), # cut top
  180 # close
)

goode_outline <- 
  list(cbind(longs, lats)) %>%
  st_polygon() %>%
  st_sfc(
    crs = "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"
  ) %>% 
  st_transform(crs = crs_goode)

# bounding box in transformed coordinates
xlim_goode <- c(-21945470, 21963330)
ylim_goode <- c(-9538022, 9266738)
goode_bbox <- 
  list(
    cbind(
      c(xlim_goode[1], xlim_goode[2], xlim_goode[2], xlim_goode[1], xlim_goode[1]), 
      c(ylim_goode[1], ylim_goode[1], ylim_goode[2], ylim_goode[2], ylim_goode[1])
    )
  ) %>%
  st_polygon() %>%
  st_sfc(crs = crs_goode)

# area outside the earth outline
goode_without <- st_difference(goode_bbox, goode_outline)

p_goode <- ggplot(world_sf) + 
  geom_sf(fill = "#E69F00B0", color = "black", size = 0.5/.pt) +
  geom_sf(data = goode_without, fill = "white", color = NA) +
  geom_sf(data = goode_outline, fill = NA, color = "grey30", size = 0.5/.pt) +
  scale_x_continuous(
    name = NULL,
    breaks = seq(-160, 160, by = 20)
  ) +
  scale_y_continuous(
    name = NULL,
    breaks = seq(-80, 80, by = 20)
  ) + 
  coord_sf(xlim = 0.95*xlim_goode, ylim = 0.95*ylim_goode, expand = FALSE, crs = crs_goode, ndiscr = 1000) + 
  theme_dviz_grid(12, rel_small = 1) +
  theme(
    panel.background = element_rect(fill = "#56B4E950", color = "white", size = 1),
    panel.grid.major = element_line(color = "gray30", size = 0.25),
    plot.margin = margin(1.5, 1.5, 24, 1.5)
  )


## Robinson projection
crs_robin <- "+proj=robin +lat_0=0 +lon_0=0 +x0=0 +y0=0"

# projection outline in long-lat coordinates
lats <- c(90:-90, -90:90, 90)
longs <- c(rep(c(180, -180), each = 181), 180)

robin_outline <- 
  list(cbind(longs, lats)) %>%
  st_polygon() %>%
  st_sfc(
    crs = "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"
  ) %>% 
  st_transform(crs = crs_robin)

# bounding box in transformed coordinates
xlim_robin <- c(-18494733, 18613795)
ylim_robin <- c(-9473396, 9188587)
robin_bbox <- 
  list(
    cbind(
      c(xlim_robin[1], xlim_robin[2], xlim_robin[2], xlim_robin[1], xlim_robin[1]), 
      c(ylim_robin[1], ylim_robin[1], ylim_robin[2], ylim_robin[2], ylim_robin[1])
    )
  ) %>%
  st_polygon() %>%
  st_sfc(crs = crs_robin)

# area outside the earth outline
robin_without <- st_difference(robin_bbox, robin_outline)

p_robin <- ggplot(world_sf) + 
  geom_sf(fill = "#E69F00B0", color = "black", size = 0.5/.pt) + 
  geom_sf(data = robin_without, fill = "white", color = NA) +
  geom_sf(data = robin_outline, fill = NA, color = "grey30", size = 0.5/.pt) +
  scale_x_continuous(
    name = NULL,
    breaks = seq(-160, 160, by = 20)
  ) +
  scale_y_continuous(
    name = NULL,
    breaks = seq(-80, 80, by = 20)
  ) +
  coord_sf(xlim = 0.95*xlim_robin, ylim = 0.95*ylim_robin, expand = FALSE, crs = crs_robin, ndiscr = 1000) + 
  theme_dviz_grid(12, rel_small = 1) +
  theme(
    panel.background = element_rect(fill = "#56B4E950", color = "white", size = 1),
    panel.grid.major = element_line(color = "gray30", size = 0.25),
    plot.margin = margin(6, 1.5, 1.5, 1.5)
  )


## Winkel tripel
# The Winkel tripel projection needs to be done manually, it is not supported by sf.
crs_wintri <- "+proj=wintri +datum=WGS84 +no_defs +over"

# world
world_wintri <- lwgeom::st_transform_proj(world_sf, crs = crs_wintri)

# graticule
grat_wintri <- sf::st_graticule(lat = c(-89.9, seq(-80, 80, 20), 89.9))
grat_wintri <- lwgeom::st_transform_proj(grat_wintri, crs = crs_wintri)

# earth outline
lats <- c(90:-90, -90:90, 90)
longs <- c(rep(c(180, -180), each = 181), 180)
wintri_outline <- 
  list(cbind(longs, lats)) %>%
  st_polygon() %>%
  st_sfc(
    crs = "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"
  ) %>% 
  lwgeom::st_transform_proj(crs = crs_wintri)

p_wintri <- ggplot() + 
  geom_sf(data = wintri_outline, fill = "#56B4E950", color = NA) +
  geom_sf(data = grat_wintri, color = "gray30", size = 0.25/.pt) + 
  geom_sf(data = world_wintri, fill = "#E69F00B0", color = "black", size = 0.5/.pt) + 
  geom_sf(data = wintri_outline, fill = NA, color = "grey30", size = 0.5/.pt) +
  coord_sf(datum = NA, expand = FALSE) +
  theme_dviz_grid(12, rel_small = 1) +
  theme(
    plot.margin = margin(6, 1.5, 3, 1.5)
  )

p <- plot_grid(
  p_longlat, p_goode, p_robin, p_wintri,
  labels = c(
    "Cartesian longitude and latitude", "Interrupted Goode homolosine", 
    "Robinson", "Winkel tripel"
  )
)

p + theme(plot.margin = margin(1.5, 0, 0, 0))
```

<!--chapter:end:coordinate_systems_axes.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(dplyr)
library(tidyr)
library(ggforce)
library(ggridges)
library(treemapify)
library(forcats)
library(statebins)
library(sf)
library(ungeviz)
```

# Directory of visualizations {#directory-of-visualizations}

This chapter provides a quick visual overview of the various plots and charts that are commonly used to visualize data. It is meant both to serve as a table of contents, in case you are looking for a particular visualization whose name you may not know, and as a source of inspiration, if you need to find alternatives to the figures you routinely make.

```{r}
## general setup code

# line_size = 0.6

# theme
theme_plot_icon <- function(bg_color = "#F5F8EA", line_color = "#243400",
                            line_size = .5, font_size = 14) {
  theme_dviz_open() %+replace% theme(
    axis.text.x       = element_blank(),
    axis.text.y       = element_blank(),
    axis.title.x      = element_blank(),
    axis.title.y      = element_blank(),
    #axis.line.x       = element_blank(),
    #axis.line.y       = element_blank(),
    #axis.ticks        = element_blank(),
    axis.line.x       = element_line(size = line_size, color = line_color),
    axis.line.y       = element_line(size = line_size, color = line_color),
    axis.ticks        = element_line(size = line_size, color = line_color),
    axis.ticks.length = grid::unit(4, "pt"),
    legend.position   = "none",
    plot.margin       = margin(
      font_size*8/14, font_size, font_size*10/14, font_size
    ),
    plot.title        = element_text(
      hjust = 0.5,
      #family = dviz_font_family_bold,
      family = dviz_font_family_condensed,
      color = line_color,
      size = font_size,
      margin = margin(0, 0, font_size*6/14, 0)
    ),
    plot.background   = element_rect(fill = bg_color)
  )
}

theme_plot_icon_hgrid <- function(bg_color = "#F5F8EA", line_color = "#243400",
                                  line_size = .5, font_size = 14) {
  theme_plot_icon(bg_color, line_color, line_size, font_size) %+replace% theme(
      # make grid lines
      #panel.grid.major.y   = element_line(colour = paste0(line_color, "30"),
      #                                    size = 0.5),

      # remove x axis
      axis.ticks.x        = element_blank(),
      axis.line.x         = element_blank()
  )
}

theme_plot_icon_vgrid <- function(bg_color = "#F5F8EA", line_color = "#243400",
                                  line_size = .5, font_size = 14) {
  theme_plot_icon(bg_color, line_color, line_size, font_size) %+replace% theme(
      # make grid lines
      #panel.grid.major.x   = element_line(colour = paste0(line_color, "30"),
      #                                    size = 0.5),

      # remove y axis
      axis.ticks.y        = element_blank(),
      axis.line.y         = element_blank()
  )
}

theme_plot_icon_blank <- function(bg_color = "#F5F8EA", line_color = "#243400",
                                  line_size = .5, font_size = 14) {
  theme_plot_icon(bg_color, line_color, line_size, font_size) %+replace% theme(
      axis.ticks          = element_blank(),
      axis.line.x         = element_blank(),
      axis.line.y         = element_blank(),
      axis.ticks.length    = grid::unit(0, "pt")
  )
}

# data sets
set.seed(5142)

n <- 15
x <- rnorm(n)
y <- .4*x + .6*rnorm(n)
df_scatter_xy <- data.frame(x, y)

df_one_dist <- data.frame(x = c(rnorm(1000, 1., 1.6), rnorm(300, 4, .4)))

df_one_normal <- data.frame(x = rnorm(20))

df_fractions <- data.frame(y = c(.3, .39, .48, .6, .25, .13, .22, .24, .45, .48, .3, .16),
                 x = factor(rep(1:4, 3)),
                 type = rep(c("A", "B", "C"), each = 4))


set.seed(2474)

n <- 8
x <- rnorm(n)
y <- .4*x + .6*rnorm(n)
z <- .5*x + .3*rnorm(n)
z <- (z - min(z) + 0.1)^2
df_scatter_xyz <- data.frame(x, y, z)


set.seed(5012)
df_multi_amounts <- mutate(df_fractions,
                           y = c(1.0, 1.1, 1.4, 1.2)[x]*y)

n <- 70
df_multi_dist <- data.frame(y = c(rnorm(n, 1, .8), rnorm(n, 2, .7), rnorm(n, 0, .5)),
                 type = rep(c("A", "B", "C"), each = n),
                 number = rep(c(2, 1, 3), each = n))


df_props = data.frame(value = c(55, 30, 15),
                      group = c("A", "B", "C"))

df_multi_props <- data.frame(
  var1 = rep(c("C", "B", "A"), 3),
  var2 = rep(c("A", "B", "C"), each = 3),
  count = c(4, 1, 2, 12, 9, 5, 4, 5, 4)
) %>% group_by(var2) %>%
  mutate(group_count = sum(count))

df_multi_props2 <- data.frame(
  var1 = rep(c("B", "A"), 9),
  var2 = rep(c("E", "E", "D", "D", "C", "C"), 3),
  var3 = rep(c("H", "G", "F"), each = 6),
  count = c(5, 8, 0, 0, 0, 0, 0, 3, 2, 7, 0, 0, 4, 0, 4, 2, 7, 4)
)

df_sets <- gather_set_data(df_multi_props2, 1:3)

df_one_line <- data.frame(
  x = 1:5,
  y = c(3.1, 3.3, 4.0, 3.8, 4.4)
)

set.seed(9681)
n1 <- 1500/5
n2 <- 800/5
x1 <- rnorm(n1, 0, .7)
y1 <- 2 * x1 + rnorm(n1, 0, .8)

x2 <- rnorm(n2, 0, 0.4)
y2 <- 1.5 * x2 + rnorm(n2, .5, .8)

df_dense_scatter <- na.omit(
  data.frame(
    x = scales::censor(c(x1, x2 + 2.2), c(-2, 4)),
    y = scales::censor(c(y1, y2 + 1.5), c(-3.5, 4.5))
  )
)

y1 <- 2 * x1 + rnorm(n1, 0, 1.6)
y2 <- 1.5 * x2 + rnorm(n2, .5, 1.6)
df_dense_scatter_sample <- na.omit(
  data.frame(
    x = scales::censor(c(x1, x2 + 2.2), c(-2, 4)),
    y = scales::censor(c(y1, y2 + 1.5), c(-3.5, 4.5))
  )
) %>% sample_n(50)

df_connected_scatter <- data.frame(
  x = c(1.9, 1.5, 2.2, 3, 3.3, 2.7, 1.7, 1),
  y = c(0.3, -1, -2.0, -0.9, .6, 1.8, 2, 0.7),
  t = 1:8
)

df_paired <- data.frame(
  y = c(6, 5.3, 3.8, 2.8, 2,
        4.3, 6.1, 5.1, 3.3, 2.4),
  x = rep(c("A", "B"), each = 5),
  group = rep(1:5, 2)
)

df_uncertain <- data.frame(
  type = c("A", "B", "C"),
  x = c(1.5, 2.2, 3.4),
  y = c(3.2, 5.1, 3.9),
  dx = c(.25, .3, .35),
  dy = c(.5, .4, .6)
)


# palettes

npal <- 5
# earth-brown (Amounts)
pal_earth_brown <- sequential_hcl(n = npal, h1 = 71, c1 = 80, c2 = 10, l1 = 18, l2 = 97, p1 = 1.5)

# brown-green (Proportions)
pal_brown_green <- sequential_hcl(n = npal, h1 = 86, c1 = 80, c2 = 10, l1 = 18, l2 = 97, p1 = 1.5)

# green-brown (Geospatial data)
pal_green_brown <- sequential_hcl(n = npal, h1 = -265, c1 = 80, c2 = 10, l1 = 18, l2 = 97, p1 = 1.5)

# burgundy-red 
pal_red_brown <- sequential_hcl(n = npal, h1 = 28, c1 = 80, c2 = 10, l1 = 18, l2 = 97, p1 = 1.5)

# brown-red (Uncertainty)
pal_brown_red <- sequential_hcl(n = npal, h1 = 41, c1 = 80, c2 = 10, l1 = 18, l2 = 97, p1 = 1.5)

# ocean-blue (Distributions)
pal_ocean_blue <- sequential_hcl(n = npal, h1 = 241, c1 = 80, c2 = 10, l1 = 18, l2 = 97, p1 = 1.5)

# steel-blue (x-y relationships)
pal_steel_blue <- sequential_hcl(n = npal, h1 = 257, c1 = 80, c2 = 10, l1 = 18, l2 = 97, p1 = 1.5)
pal_steel_blue_inv <- sequential_hcl(n = npal, h1 = 257-180, c1 = 80, c2 = 10, l1 = 18, l2 = 97, p1 = 1.5)
```

## Amounts

```{r amounts, fig.width = 5*6/4.2, fig.asp = 1/4}
palette <- pal_earth_brown

p1 <- ggplot(df_props, aes(x = group, y = value)) + 
  geom_col(
    position="identity", color = palette[npal],
    fill = palette[3], width = 0.8
  ) +
  scale_y_continuous(limits = c(0, 66), expand = c(0, 0)) +
  scale_fill_manual(values = palette[2:4]) +
  labs(title = "Bars") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p2 <- ggplot(df_props, aes(x = fct_rev(group), y = value)) + 
  geom_col(position="identity", color = palette[npal], fill = palette[3],
           width = .8) +
  scale_y_continuous(limits = c(0, 66), expand = c(0, 0)) +
  scale_fill_manual(values = palette[2:4]) +
  coord_flip() +
  labs(title = "Bars") +
  theme_plot_icon_vgrid(palette[npal], palette[1])

p3 <- ggplot(filter(df_multi_amounts, x!=4), aes(x, y,
                                   fill=factor(type, levels = c("A", "C", "B")))) + 
  geom_col(position="dodge", color = palette[npal],
           width = .7) +
  scale_y_continuous(expand = c(0, 0),
                     limits = c(0, .7)) +
  scale_fill_manual(values = palette[2:4]) +
  labs(title = "Grouped Bars") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p4 <- ggplot(filter(df_multi_amounts, x!=4), aes(x, y,
                                   fill=factor(type, levels = c("B", "C", "A")))) + 
  geom_col(position="dodge", color = palette[npal],
           width = .7) +
  scale_y_continuous(expand = c(0, 0),
                     limits = c(0, .7)) +
  scale_fill_manual(values = rev(palette[2:4])) +
  coord_flip() +
  labs(title = "Grouped Bars") +
  theme_plot_icon_vgrid(palette[npal], palette[1])

p5 <- ggplot(df_multi_amounts, aes(x, y, fill=factor(type, levels = c("B", "C", "A")))) + 
  geom_col(position="stack", color = palette[npal]) +
  scale_y_continuous(limits = c(0, 1.55),
                     expand = c(0, 0)) +
  scale_fill_manual(values = rev(palette[2:4])) +
  labs(title = "Stacked Bars") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p6 <- p5 + coord_flip() + theme_plot_icon_vgrid(palette[npal], palette[1])

p7 <- ggplot(df_props, aes(x = fct_rev(group), y = value)) + 
  geom_point(color = palette[2], size = 2) +
  scale_y_continuous(limits = c(0, 66), expand = c(0, 0)) +
  coord_flip() + 
  labs(title = "Dots") +
  theme_plot_icon_vgrid(palette[npal], palette[1])

p8 <- ggplot(filter(df_multi_amounts, x != 1), aes(x, y = factor(type, levels = c("A", "C", "B")), fill = y)) + 
  geom_tile(color = palette[5], size = 1.5) +
  scale_fill_continuous_sequential(
    h1 = 71, c1 = 80, c2 = 10, l1 = 18, l2 = 97, p1 = 1.5,
    begin = 0.2, end = 0.75,
    rev = FALSE
  ) +
  labs(title = "Heatmap") +
  theme_plot_icon_blank(palette[npal], palette[1])

plot_grid(p1, p2, p7, ncol = 4, scale = .9)
```

The most common approach to visualizing amounts (i.e., numerical values shown for some set of categories) is using bars, either vertically or horizontally arranged (Chapter \@ref(visualizing-amounts)). However, instead of using bars, we can also place dots at the location where the corresponding bar would end (Chapter \@ref(visualizing-amounts)).

```{r amounts_multi, fig.width = 5*6/4.2, fig.asp = 1/2}
plot_grid(p3, p4, p5, p6, 
          p8, ncol = 4, scale = .9)
```

If there are two or more sets of categories for which we want to show amounts, we can group or stack the bars (Chapter \@ref(visualizing-amounts)). We can also map the categories onto the *x* and *y* axis and show amounts by color, via a heatmap (Chapter \@ref(visualizing-amounts)). 


## Distributions

```{r single-distributions, fig.width = 5*6/4.2, fig.asp = 1/4}

palette <- pal_ocean_blue

p1 <- ggplot(df_one_dist, aes(x)) +
  geom_histogram(fill = palette[3], color = palette[npal], binwidth = 1, center = 0) +
  scale_x_continuous(limits = c(-4.8, 6.8), expand = c(0, 0)) +
  scale_y_continuous(limits = c(0, 350), 
                     expand = c(0, 0)) +
  labs(title = "Histogram") +
  theme_plot_icon(palette[npal], palette[1])


p2 <- ggplot(df_one_dist, aes(x)) +
  geom_density(fill = palette[3], color = palette[npal], bw = .35) +
  scale_x_continuous(limits = c(-4.8, 6.8), expand = c(0, 0)) +
  scale_y_continuous(limits = c(0, .27), expand = c(0, 0)) +
  labs(title = "Density Plot") +
  theme_plot_icon(palette[npal], palette[1])

p3 <- ggplot(df_one_normal, aes(x)) +
  stat_ecdf(color = palette[2], size = .7) +
  scale_x_continuous(expand = c(0.05, 0)) +
  scale_y_continuous(limits = c(0, 1.08), expand = c(0, 0)) +
  labs(title = "Cumulative Density") +
  theme_plot_icon(palette[npal], palette[1])

p4 <- ggplot(df_one_normal, aes(sample = x)) +
  geom_abline(intercept = 0, slope = 1, color = palette[3]) +
  geom_qq(color = palette[1], size = 0.8) +
  labs(title = "Quantile-Quantile Plot") +
  theme_plot_icon(palette[npal], palette[1])

plot_grid(p1, p2, p3, p4, ncol = 4, scale = .9)
```

Histograms and density plots (Chapter \@ref(histograms-density-plots)) provide the most intuitive visualizations of a distribution, but both require arbitrary parameter choices and can be misleading. Cumulative densities and quantile-quantile (q-q) plots (Chapter \@ref(ecdf-qq)) always represent the data faithfully but can be more difficult to interpret.


```{r multiple-distributions, fig.width = 5*6/4.2, fig.asp = 1/2}
palette <- pal_ocean_blue

p1 <- ggplot(df_multi_dist, aes(x = type, y = y)) + 
  geom_boxplot(color = palette[1], fill = palette[4]) +
  labs(title = "Boxplots") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p2 <- ggplot(df_multi_dist, aes(x = type, y = y)) + 
  geom_violin(color = palette[npal], fill = palette[2], size = 0) +
  labs(title = "Violins") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

df_multi_dist_small <- group_by(df_multi_dist, type) %>%
  sample_n(50)

p3 <- ggplot(df_multi_dist_small, aes(x = type, y = y)) + 
  geom_jitter(color = palette[1], width = 0.15, height = 0, size = .3) +
  labs(title = "Strip Charts") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p4 <- ggplot(df_multi_dist_small, aes(x = type, y = y)) + 
  dviz.supp::stat_sina(color = palette[1], size = 0.3) +
  labs(title = "Sina Plots") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p5 <- ggplot(df_multi_dist, aes(x = y, fill = factor(type, levels = c("C", "A", "B")))) + 
  geom_histogram(color = palette[npal], binwidth = 0.5, center = 0) +
  scale_fill_manual(values = palette[2:4]) +
  labs(title = "Stacked Histograms") +
  scale_x_continuous() +
  scale_y_continuous(limits = c(0, 49), expand = c(0, 0)) +
  theme_plot_icon(palette[npal], palette[1])

p6 <- ggplot(df_multi_dist, aes(x = y, fill = factor(type, levels = c("C", "A", "B")))) + 
  geom_density(alpha = 0.7, color = palette[npal]) +
  scale_fill_manual(values = palette[1:3]) +
  labs(title = "Overlapping Densities") +
  scale_x_continuous() +
  scale_y_continuous(limits = c(0, 1.1), expand = c(0, 0)) +
  theme_plot_icon(palette[npal], palette[1])

p7 <- ggplot(df_multi_dist, aes(x = y, y = number, group = number)) + 
  geom_density_ridges(alpha = 0.7, color = palette[npal], fill = palette[2], scale = 2.5) +
  labs(title = "Ridgeline Plot") +
  scale_x_continuous(expand = c(0, 0)) +
  scale_y_continuous(limits = c(1, 6.5), expand = c(0, 0)) +
  theme_plot_icon(palette[npal], palette[1])


plot_grid(p1, p2, p3, p4, 
          p5, p6, p7, ncol = 4, scale = .9)
```

Boxplots, violins, strip charts, and sina plots are useful when we want to visualize many distributions at once and/or if we are primarily interested in overall shifts among the distributions (Chapter \@ref(boxplots-violins-vertical)). Stacked histograms and overlapping densities allow a more in-depth comparison of a smaller number of distributions, though stacked histograms can be difficult to interpret and are best avoided (Chapter \@ref(multiple-histograms-densities)). Ridgeline plots can be a useful alternative to violin plots and are often useful when visualizing very large numbers of distributions or changes in distributions over time (Chapter \@ref(boxplots-violins-horizontal)).

## Proportions

```{r proportions, fig.width = 5*6/4.2, fig.asp = 1/4}
palette <- pal_brown_green

p1_main <- ggplot(df_props, aes(x = 1, y = value, fill = group)) + 
  geom_col(position = "stack", color = palette[npal]) + 
  coord_polar(theta = "y") +
  scale_y_continuous(breaks = NULL, name = "") +
  scale_x_continuous(breaks = NULL, name = "") +
  scale_fill_manual(values = palette[2:4]) +
  theme_plot_icon_blank(palette[npal], palette[1]) +
  theme(plot.margin = margin(0, 0, 0, 0))

# make sure plot background is fully filled, as in the other plots
p1 <- ggdraw(p1_main) +
  labs(title = "Pie Chart") +
  theme_plot_icon_blank(palette[npal], palette[1])

p2 <- ggplot(df_props, aes(x = factor(1), y = value, fill = group)) + 
  geom_col(position = position_stack(reverse = TRUE), width = .45, color = palette[npal]) + 
  scale_y_continuous(limits = c(0, 108), expand = c(0, 0)) +
  scale_fill_manual(values = palette[2:4]) +
  labs(title = "Stacked Bars") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p3 <- ggplot(df_props, aes(x = factor(1), y = value, fill = group)) + 
  geom_col(position = position_stack(reverse = TRUE), width = .45, color = palette[npal]) + 
  #scale_y_continuous(limits = c(0, 110), expand = c(0, 0), position = "right") +
  scale_y_continuous(limits = c(0, 110), expand = c(0, 0)) +
  coord_flip() +
  scale_fill_manual(values = palette[2:4]) +
  labs(title = "Stacked Bars") +
  theme_plot_icon_vgrid(palette[npal], palette[1])

p4 <- ggplot(df_props, aes(x = group, y = value, fill = group)) + 
  geom_col(position="identity", color = palette[npal],
           width = .8) +
  scale_y_continuous(limits = c(0, 66), expand = c(0, 0)) +
  scale_fill_manual(values = palette[2:4]) +
  labs(title = "Bars") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p5 <- ggplot(df_props, aes(x = fct_rev(group), y = value, fill = group)) + 
  geom_col(position="identity", color = palette[npal],
           width = .8) +
  scale_y_continuous(limits = c(0, 66), expand = c(0, 0)) +
  scale_fill_manual(values = palette[2:4]) +
  coord_flip() +
  labs(title = "Bars") +
  theme_plot_icon_vgrid(palette[npal], palette[1])


plot_grid(p1, p4, p5, p2, ncol = 4, scale = .9)
```

Proportions can be visualized as pie charts, side-by-side bars, or stacked bars (Chapter \@ref(visualizing-proportions)), and as in the case for amounts, bars can be arranged either vertically or horizontally. Pie charts emphasize that the individual parts add up to a whole and highlight simple fractions. However, the individual pieces are more easily compared in side-by-side bars. Stacked bars look awkward for a single set of proportions, but can be useful when comparing multiple sets of proportions (see below).


```{r proportions-comp, fig.width = 5*6/4.2, fig.asp = 1/4}
p5 <- ggplot(filter(df_fractions, x!=4), aes(x, y,
                                   fill=factor(type, levels = c("A", "C", "B")))) + 
  geom_col(position="dodge", color = palette[npal],
           width = .7) +
  scale_y_continuous(expand = c(0, 0),
                     limits = c(0, .58)) +
  scale_fill_manual(values = palette[2:4]) +
  labs(title = "Grouped Bars") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p6 <- ggplot(df_fractions, aes(x, y, fill=type)) + 
  geom_col(position="stack", color = palette[npal]) +
  scale_y_continuous(limits = c(0, 1.08), expand = c(0, 0)) +
  scale_fill_manual(values = palette[2:4]) +
  labs(title = "Stacked Bars") +
  theme_plot_icon_hgrid(palette[npal], palette[1])

p7 <- ggplot(df_multi_dist, aes(x = y, fill = factor(type, levels = c("C", "A", "B")))) + 
  geom_density(color = palette[npal], position = "fill") +
  scale_fill_manual(values = palette[2:4]) +
  scale_x_continuous(expand = c(0.04, 0)) +
  scale_y_continuous(limits = c(0, 1.08), expand = c(0, 0)) +
  labs(title = "Stacked Densities") +
  theme_plot_icon(palette[npal], palette[1])

p8_a <- ggplot(filter(df_fractions, x==1), aes(x = 1, y = y, fill = type)) + 
  geom_col(position = "stack", color = palette[npal]) + 
  coord_polar(theta = "y") +
  scale_y_continuous(breaks = NULL, name = "") +
  scale_x_continuous(breaks = NULL, name = "") +
  scale_fill_manual(values = palette[c(2, 1, 3)]) +
  theme_plot_icon_blank(palette[npal], palette[1], font_size = 5) +
  theme(
    plot.background = element_blank(),
    plot.margin = margin(0, 0, 0, 0)
  )

p8_b <- ggplot(filter(df_fractions, x==2), aes(x = 1, y = y, fill = type)) + 
  geom_col(position = "stack", color = palette[npal]) + 
  coord_polar(theta = "y") +
  scale_y_continuous(breaks = NULL, name = "") +
  scale_x_continuous(breaks = NULL, name = "") +
  scale_fill_manual(values = palette[c(2, 1, 3)]) +
  theme_plot_icon_blank(palette[npal], palette[1], font_size = 5) +
  theme(
    plot.background = element_blank(),
    plot.margin = margin(0, 0, 0, 0)
  )

p8_c <- ggplot(filter(df_fractions, x==3), aes(x = 1, y = y, fill = type)) + 
  geom_col(position = "stack", color = palette[npal]) + 
  coord_polar(theta = "y") +
  scale_y_continuous(breaks = NULL, name = "") +
  scale_x_continuous(breaks = NULL, name = "") +
  scale_fill_manual(values = palette[c(2, 1, 3)]) +
  theme_plot_icon_blank(palette[npal], palette[1], font_size = 5) +
  theme(
    plot.background = element_blank(),
    plot.margin = margin(0, 0, 0, 0)
  )


# combine
p8 <- plot_grid(p8_a, p8_b, p8_c, ncol = 3, scale = 1.1) +
      labs(title = "Multiple Pie Charts") +
      theme_plot_icon_blank(palette[npal], palette[1])

plot_grid(p8, p5, p6, p7, ncol = 4, scale = .9)
```

When visualizing multiple sets of proportions or changes in proportions across conditions, pie charts tend to be space-inefficient and often obscure relationships. Grouped bars work well as long as the number of conditions compared is moderate, and stacked bars can work for large numbers of conditions. Stacked densities (Chapter \@ref(visualizing-proportions)) are appropriate when the proportions change along a continuous variable.

```{r proportions-multi, fig.width = 5*6/4.2, fig.asp = 1/4}
p1 <- ggplot(df_multi_props, aes(x = var2, y = count, fill = var1, width = group_count)) +
  geom_bar(stat = "identity", position = "fill", colour = palette[npal], size = 0.5) +
  facet_grid(~var2, scales = "free_x", space = "free_x") +
  scale_x_discrete(name = NULL, breaks = NULL) +
  scale_y_continuous(name = NULL, breaks = NULL, expand = c(0, 0)) +
  scale_fill_manual(values = palette[4:2], guide = "none") +
  coord_cartesian(clip = "off") +
  labs(title = "Mosaic Plot") +
  theme_plot_icon_blank(palette[npal], palette[1]) +
  theme(
    strip.text = element_blank(),
    panel.spacing.x = unit(0, "pt")
  )
  
p2 <- ggplot(df_multi_props, aes(area = count, subgroup = var2, fill = var2)) +
  geom_treemap(color = palette[npal], size = 0.5*.pt, alpha = NA) + 
  geom_treemap_subgroup_border(color = palette[npal], size = 1.5*.pt) +
  scale_fill_manual(values = palette[4:2], guide = "none") +
  coord_cartesian(clip = "off") +
  labs(title = "Treemap") +
  theme_plot_icon_blank(palette[npal], palette[1]) 

p3 <- ggplot(df_sets, aes(x, id = id, split = y, value = count)) +
  geom_parallel_sets(aes(fill = var1), alpha = 0.7, axis.width = 0.15) +
  geom_parallel_sets_axes(axis.width = 0.06, fill = palette[2], color = palette[2]) +
  scale_x_discrete(
    name = NULL,
    breaks = NULL,
    expand = c(0, 0.15/2)
  ) +
  scale_y_continuous(breaks = NULL, expand = c(0, 0)) +
  scale_fill_manual(values = c(palette[3], palette[2]), guide = "none") +
  labs(title = "Parallel Sets") +
  theme_plot_icon_blank(palette[npal], palette[1])

plot_grid(p1, p2, p3, ncol = 4, scale = .9)
```

When proportions are specified according to multiple grouping variables, then mosaic plots, treemaps, or parallel sets are useful visualization approaches
(Chapter \@ref(nested-proportions)). Mosaic plots assume that every level of one grouping variable can be combined with every level of another grouping variable, whereas treemaps do not make such an assumption. Treemaps work well even if the subdivisions of one group are entirely distinct from the subdivisions of another. Parallel sets work better than either mosaic plots or treemaps when there are more than two grouping variables.


## *x*--*y* relationships

```{r basic-scatter, fig.width = 5*6/4.2, fig.asp = 1/4}
palette <- pal_steel_blue

p1 <- ggplot(df_scatter_xy, aes(x, y)) + 
  geom_point(fill = palette[2], color = palette[npal], pch = 21, size = 2.4) + 
  scale_x_continuous(expand = c(.2, 0)) +
  scale_y_continuous(expand = c(.2, 0)) +
  labs(title = "Scatterplot") +
  theme_plot_icon(palette[npal], palette[1])

p2 <- ggplot(df_scatter_xyz, aes(x, y, size = z)) + 
  geom_point(fill = palette[2], color = palette[npal], pch = 21, alpha = 0.7) + 
  scale_x_continuous(expand = c(.2, 0)) +
  scale_y_continuous(expand = c(.2, 0)) +
  scale_radius(range = c(2, 8)) +
  labs(title = "Bubble Chart") +
  theme_plot_icon(palette[npal], palette[1])

p3 <- ggplot(spread(df_paired, x, y), aes(A, B)) + 
  geom_abline(slope = 1, intercept = 0, color = palette[3], size = 0.3) + 
  geom_point(
    shape = 21, size = 2.4, stroke = 1,
    fill = palette[2], color = palette[npal]
  ) +
  scale_x_continuous(limits = c(1.5, 6.5)) +
  scale_y_continuous(limits = c(1.5, 6.5)) +
  labs(title = "Paired Scatterplot") +
  theme_plot_icon(palette[npal], palette[1])

p4 <- ggplot(df_paired, aes(x, y, group = group)) + 
  geom_line(color = palette[1]) + 
  geom_point(
    shape = 21, size = 2.4, stroke = 1,
    fill = palette[2], color = palette[npal]
  ) +
  scale_x_discrete(expand = c(0, 0.4)) +
  scale_y_continuous(limits = c(1.5, 6.5)) +
  labs(title = "Slopegraph") +
  theme_plot_icon(palette[npal], palette[1]) +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank()
  )

plot_grid(p1, p2, p3, p4, ncol = 4, scale = .9)
```


Scatterplots represent the archetypical visualization when we want to show one quantitative variable relative to another (Chapter \@ref(associations-scatterplots)). If we have three quantitative variables, we can map one onto the dot size, creating a variant of the scatterplot called bubble chart. For paired data, where the variables along the *x* and the *y* axes are measured in the same units, it is generally helpful to add a line indicating *x* = *y* (Chapter \@ref(associations-paired-data)). Paired data can also be shown as a slope graph of paired points connected by straight lines (Chapter \@ref(associations-paired-data)).

```{r xy-binning, fig.width = 5*6/4.2, fig.asp = 1/4}
p5 <- ggplot(df_dense_scatter, aes(x, y)) + 
  geom_density2d(binwidth = 0.02, color = palette[1]) +
  scale_x_continuous(limits = c(-2, 3.6), expand = c(0, 0)) +
  scale_y_continuous(limits = c(-4, 5), expand = c(0, 0)) +
  labs(title = "Density Contours") +
  theme_plot_icon(palette[npal], palette[1])

p6 <- ggplot(df_dense_scatter, aes(x, y)) + 
  geom_bin2d(bins = 12, color = palette[npal], size = 0.5) +
  scale_x_continuous(limits = c(-2, 3.6), expand = c(0, 0)) +
  scale_y_continuous(limits = c(-4, 5), expand = c(0, 0)) +
  scale_fill_gradientn(colors = palette[1:(npal-1)]) +
  labs(title = "2D Bins") +
  theme_plot_icon(palette[npal], palette[1])

p7 <- ggplot(df_dense_scatter, aes(x, y)) + 
  geom_hex(bins = 12, color = palette[npal], size = 0.5) +
  scale_x_continuous(limits = c(-2, 3.6), expand = c(0, 0)) +
  scale_y_continuous(limits = c(-4, 5), expand = c(0, 0)) +
  scale_fill_gradientn(colors = palette[1:(npal-1)]) +
  labs(title = "Hex Bins") +
  theme_plot_icon(palette[npal], palette[1])

cm <- cor(select(mtcars, mpg, hp, drat, wt, qsec))
df_wide <- as.data.frame(cm)
df_long <- stack(df_wide)
names(df_long) <- c("cor", "var1")
df_long <- cbind(df_long, var2 = rep(rownames(cm), length(rownames(cm))))
clust <- hclust(as.dist(1-cm), method="average") 
levels <- clust$labels[clust$order]
df_long$var1 <- factor(df_long$var1, levels = levels)
df_long$var2 <- factor(df_long$var2, levels = levels)
p8 <- ggplot(filter(df_long, as.integer(var1) < as.integer(var2)),
       aes(var1, var2, fill=cor, size = abs(cor))) + 
  geom_point(shape = 21, stroke = 0) + 
  scale_x_discrete(position = "top", name = NULL, expand = c(0, 0.5)) +
  scale_y_discrete(name = NULL, expand = c(0, 0.5)) +
  scale_size_area(max_size = 8, limits = c(0, 0.9), guide = "none") +
  scale_fill_gradient2(high = palette[2], mid = palette[npal], low = pal_steel_blue_inv[2], guide = "none") +
  labs(title = "Correlogram") +
  theme_plot_icon(palette[npal], palette[1])


plot_grid(p5, p6, p7, p8, ncol = 4, scale = .9)
```

For large numbers of points, regular scatterplots can become uninformative due to overplotting. In this case, contour lines, 2D bins, or hex bins may provide an alternative (Chapter \@ref(overlapping-points)). When we want to visualize more than two quantities, on the other hand, we may choose to plot correlation coefficients in the form of a correlogram instead of the underlying raw data (Chapter \@ref(associations-correlograms)).

```{r xy-lines, fig.width = 5*6/4.2, fig.asp = 1/4}
p1 <- ggplot(df_one_line, aes(x, y)) +
  geom_line(color = palette[1]) + 
  geom_point(
    shape = 21, size = 2.4, stroke = 1,
    fill = palette[2], color = palette[npal]
  ) +
  scale_x_continuous(limits = c(0.5, 5.5), breaks = c(1, 3, 5)) +
  scale_y_continuous(limits = c(2.8, 4.8)) +
  labs(title = "Line Graph") +
  theme_plot_icon(palette[npal], palette[1])

p2 <- ggplot(df_connected_scatter, aes(x, y, color = t, fill = t)) +
  geom_path() +
  geom_point(
    shape = 21, size = 2.4, stroke = 1,
    color = palette[npal]
  ) +
  scale_color_gradientn(
    aesthetics = c("colour", "fill"),
    colors = palette[(npal-2):1]
  ) +
  scale_x_continuous(limits = c(0.3, 3.7)) +
  scale_y_continuous(limits = c(-2.5, 2.5)) +
  labs(title = "Connected Scatterplot") +
  theme_plot_icon(palette[npal], palette[1])

p3 <- ggplot(df_dense_scatter_sample, aes(x, y)) +
  geom_point(color = palette[2], size = 0.3, alpha = 1/2) +
  geom_smooth(
    color = palette[1],
    fill = palette[npal-2],
    size = 0.5,
    se = FALSE
  ) +
  scale_y_continuous(limits = c(-5, 5)) +
  labs(title = "Smooth Line Graph") +
  theme_plot_icon(palette[npal], palette[1])

plot_grid(p1, p2, p3, ncol = 4, scale = .9)
```

When the *x* axis represents time or a strictly increasing quantity such as a treatment dose, we commonly draw line graphs (Chapter \@ref(time-series)). If we have a temporal sequence of two response variables, we can draw a connected scatterplot where we first plot the two response variables in a scatterplot and then connect dots corresponding to adjacent time points (Chapter \@ref(time-series-connected-scatter)). We can use smooth lines to represent trends in a larger dataset (Chapter \@ref(visualizing-trends)). 


## Geospatial data {#directory-geospatial-data}

```{r geospatial, fig.width = 5*6/4.2, fig.asp = 1/4}
palette <- pal_green_brown

lower48 <- mutate(
  US_income,
  income_bins = cut(
    ifelse(is.na(median_income), 25000, median_income), # hide missing value
    breaks = c(0, 40000, 50000, 60000, 70000, 80000)
  )
) %>% filter(!name %in% c("Alaska", "Hawaii", "District of Columbia"))


p1_main <- ggplot(lower48) +
  geom_sf(color = palette[1], fill = palette[4], size = 0.3) +
  coord_sf(datum = NA, expand = FALSE) +
  scale_x_continuous(limits = c(-2500000, 100000)) +
  scale_y_continuous(limits = c(-900000, 1558935)) +
  theme_plot_icon_blank(palette[npal], palette[1]) +
  theme(
    plot.margin = margin(2, 5, 3, 5)
  )

# make sure plot background is fully filled, as in the other plots
p1 <- ggdraw(p1_main) +
  labs(title = "Map") +
  theme_plot_icon_blank(palette[npal], palette[1])


p2_main <- ggplot(lower48, aes(fill = income_bins)) +
  geom_sf(color = palette[1], size = 0.2) +
  coord_sf(datum = NA, expand = FALSE) +
  scale_x_continuous(limits = c(-2500000, 100000)) +
  scale_y_continuous(limits = c(-900000, 1558935)) +
  scale_fill_manual(values = palette) +
  theme_plot_icon_blank(palette[npal], palette[1]) +
  theme(
    plot.margin = margin(2, 5, 3, 5)
  )

p2 <- ggdraw(p2_main) +
  labs(title = "Choropleth") +
  theme_plot_icon_blank(palette[npal], palette[1])

lower48_carto <- mutate(
  US_income_cartogram,
  income_bins = cut(
    ifelse(is.na(median_income), 25000, median_income), # hide missing value
    breaks = c(0, 40000, 50000, 60000, 70000, 80000)
  )
) %>% filter(!name %in% c("Alaska", "Hawaii", "District of Columbia"))

p3_main <- ggplot(lower48_carto, aes(fill = income_bins)) +
  geom_sf(color = palette[1], size = 0.2) +
  coord_sf(datum = NA, expand = FALSE) +
  scale_x_continuous(limits = c(-2500000, 100000)) +
  scale_y_continuous(limits = c(-1000000, 1458935)) +
  scale_fill_manual(values = palette) +
  theme_plot_icon_blank(palette[npal], palette[1]) +
  theme(
    plot.margin = margin(2, 5, 3, 5)
  )

p3 <- ggdraw(p3_main) +
  labs(title = "Cartogram") +
  theme_plot_icon_blank(palette[npal], palette[1])

lower48_small <- filter(lower48, GEOID %in% c(
  "04", "06", "08", "16", "20", "30", "31", "32", "35", "38", "41", "46", "49", "53", "56"))

p4_main <- ggplot(lower48_small, aes(state = name, fill = income_bins)) +
  geom_statebins(
    family = dviz.supp::dviz_font_family,
    lbl_size = 8/.pt,
    border_size = 1.,
    border_col = palette[npal]
  ) +
  coord_equal(xlim = c(1.5, 5.5), ylim = c(-2.5, -6.5), expand = FALSE, clip = "off") +
  scale_fill_manual(values = palette[2:5]) +
  theme_plot_icon_blank(palette[npal], palette[1]) +
  theme(
    plot.margin = margin(2, 0, 0, 7)
  )

p4 <- ggdraw(p4_main) + labs(title = "Cartogram Heatmap") +
  theme_plot_icon_blank(palette[npal], palette[1])

plot_grid(p1, p2, p3, p4, scale = 0.9, nrow = 1)
```

The primary mode of showing geospatial data is in the form of a map (Chapter \@ref(geospatial-data)). A map takes coordinates on the globe and projects them onto a flat surface, such that shapes and distances on the globe are approximately represented by shapes and distances in the 2D representation. In addition, we can show data values in different regions by coloring those regions in the map according to the data. Such a map is called a choropleth (Chapter \@ref(choropleth-mapping)). In some cases, it may be helpful to distort the different regions according to some other quantity (e.g., population number) or simplify each region into a square. Such visualizations are called cartograms.

## Uncertainty {#directory-uncertainty}

```{r errorbars, fig.width = 5*6/4.2, fig.asp = 1/4}
palette <- pal_brown_red

p1 <- ggplot(df_uncertain, aes(y, type)) +
  geom_errorbarh(
    aes(xmin = y-dy, xmax = y+dy),
    color = palette[1], height = 0.2, size = 0.5
  ) +
  geom_point(
    color = palette[1],
    size = 2
  ) +
  labs(title = "Error Bars") +
  theme_plot_icon(palette[npal], palette[1]) +
  theme(
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank()
  )

p2 <- ggplot(df_uncertain, aes(type, y)) +
  geom_col(fill = palette[3], width = 0.8) +
  geom_segment(
    aes(xend = type, y = y-dy, yend = y+dy),
    color = palette[1],
    size = 0.7
  ) +
  scale_y_continuous(limits = c(0, 6), expand = c(0, 0)) +
  labs(title = "Error Bars") +
  theme_plot_icon(palette[npal], palette[1]) +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank()
  )

p3 <- ggplot(df_uncertain, aes(y, type)) +
  geom_errorbarh(
    aes(xmin = y-2.58*dy, xmax = y+2.58*dy), # 99% CI
    color = palette[3], height = 0, size = 0.5
  ) +
  geom_errorbarh(
    aes(xmin = y-1.96*dy, xmax = y+1.96*dy), # 95% CI
    color = palette[2], height = 0, size = 1
  ) +
  geom_errorbarh(
    aes(xmin = y-1.28*dy, xmax = y+1.28*dy), # 80% CI
    color = palette[1], height = 0, size = 1.5
  ) +
  #geom_errorbarh(
  #  aes(xmin = y-dy, xmax = y+dy),
  #  color = palette[1], height = 0.1, size = 0.5
  #) +
  geom_point(
    color = palette[1],
    size = 2
  ) +
  labs(title = "Graded Error Bars") +
  theme_plot_icon(palette[npal], palette[1]) +
  theme(
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank()
  )

p4 <- ggplot(df_uncertain, aes(x, y)) +
  geom_point(color = palette[1], size = 2) +
  geom_segment(
    aes(xend = x, y = y-dy, yend = y+dy),
    color = palette[1],
    size = 0.7
  ) +
  geom_segment(
    aes(yend = y, x = x-dx, xend = x+dx),
    color = palette[1],
    size = 0.7
  ) +
  scale_x_continuous(limits = c(1, 4)) +
  scale_y_continuous(limits = c(2, 6)) +
  labs(title = "2D Error Bars") +
  theme_plot_icon(palette[npal], palette[1])
  


plot_grid(p1, p2, p4, p3, ncol = 4, scale = .9)
  
```

Error bars are meant to indicate the range of likely values for some estimate or measurement. They extend horizontally and/or vertically from some reference point representing the estimate or measurement (Chapter \@ref(visualizing-uncertainty)). Reference points can be shown in various ways, such as by dots or by bars. Graded 
error bars show multiple ranges at the same time, where each range corresponds to a different degree of confidence. They are in effect multiple error bars with different line thicknesses plotted on top of each other.

```{r confidence-dists, fig.width = 5*6/4.2, fig.asp = 1/4}

p1 <- ggplot(df_uncertain, aes(y, type)) +
  stat_confidence_density(aes(moe = dy), fill = palette[3], height = 0.6, confidence = 0.68) +
  scale_x_continuous(limits = c(1.6, 6.4), expand = c(0, 0)) +
  scale_y_discrete(expand = c(0, 1)) +
  labs(title = "Confidence Strips") +
  theme_plot_icon(palette[npal], palette[1]) +
  theme(
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank()
  )


p2 <- ggplot(df_uncertain, aes(y, type)) +
  geom_ribbon(
    data = filter(df_uncertain, type == "A"),
    aes(moe = dy, ymin = 1 - .5*stat(density), ymax = 1 + .5*stat(density)),
    stat = "confidence_density",
    fill = palette[3], color = NA, alpha = NA, confidence = 0.68
  ) +
  geom_ribbon(
    data = filter(df_uncertain, type == "B"),
    aes(moe = dy, ymin = 2 - .5*stat(density), ymax = 2 + .5*stat(density)),
    stat = "confidence_density",
    fill = palette[3], color = NA, alpha = NA, confidence = 0.68
  ) +
  geom_ribbon(
    data = filter(df_uncertain, type == "C"),
    aes(moe = dy, ymin = 3 - .5*stat(density), ymax = 3 + .5*stat(density)),
    stat = "confidence_density",
    fill = palette[3], color = NA, alpha = NA, confidence = 0.68
  ) +
  geom_errorbarh(
    aes(xmin = y-1.28*dy, xmax = y+1.28*dy), 
    color = palette[1], height = 0, size = 0.5
  ) +
  geom_point(
    color = palette[1],
    size = 2
  ) +
  scale_x_continuous(limits = c(1.6, 6.4), expand = c(0, 0)) +
  scale_y_discrete(expand = expand_scale(add = c(0.8, 0.8))) +
  labs(title = "Eyes") +
  theme_plot_icon(palette[npal], palette[1]) +
  theme(
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank()
  )

p3 <- ggplot(df_uncertain, aes(y, type)) +
  stat_confidence_density(
    aes(moe = dy, height = .9*stat(density)),
    geom = "ridgeline",
    fill = palette[3], color = NA, alpha = NA, confidence = 0.68
  ) +
  geom_errorbarh(
    aes(xmin = y-1.28*dy, xmax = y+1.28*dy),
    color = palette[1], height = 0, size = 0.5
  ) +
  geom_point(
    color = palette[1],
    size = 2
  ) +
  scale_x_continuous(limits = c(1.6, 6.4), expand = c(0, 0)) +
  scale_y_discrete(expand = expand_scale(add = c(0.2, 0.8))) +
  labs(title = "Half-Eyes") +
  theme_plot_icon(palette[npal], palette[1]) +
  theme(
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank()
  )

df_norm <- data.frame(
  x = seq(-3, 3, length.out = 100),
  y = dnorm(seq(-3, 3, length.out = 100))
)
df_q <- data.frame(x = qnorm(ppoints(20)))

p4 <- ggplot(df_q, aes(x)) +
  geom_line(data = df_norm, aes(x, .36*y), color = palette[1], na.rm = FALSE, size = 0.25) + # factor .36 manually determined
  geom_dotplot(binwidth = .4, fill = palette[3], color = palette[1]) +
  scale_x_continuous(
    limits = c(-2.8, 2.8),
    expand = c(0, 0)
  ) +
  scale_y_continuous(
    expand = c(0.02, 0),
    limits = c(0, 0.4)
  ) +
  labs(title = "Quantile Dot Plot") +
  theme_plot_icon(palette[npal], palette[1]) +
  theme(
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank()
  )

plot_grid(p1, p2, p3, p4, ncol = 4, scale = .9)
  
```

To achieve a more detailed visualization than is possible with error bars or graded error bars, we can visualize the actual confidence or posterior distributions (Chapter \@ref(visualizing-uncertainty)). Confidence strips provide a clear visual sense of uncertainty but are difficult to read accurately. Eyes and half-eyes combine error bars with approaches to visualize distributions (violins and ridgelines, respectively), and thus show both precise ranges for some confidence levels and the overall uncertainty distribution. A quantile dot plot can serve as an alternative visualization of an uncertainty distribution (Chapter \@ref(frequency-framing)). By showing the distribution in discrete units, the quantile dot plot is not as precise but can be easier to read than the continuous distribution shown by a violin or ridgeline plot.

```{r confidence-bands, fig.width = 5*6/4.2, fig.asp = 1/4}
p1 <- ggplot(df_dense_scatter_sample, aes(x, y)) +
  geom_smooth(
    color = palette[1],
    fill = palette[npal-2],
    size = 0.5,
    level = 0.95
  ) +
  scale_y_continuous(limits = c(-5, 5)) +
  labs(title = "Confidence Band") +
  theme_plot_icon(palette[npal], palette[1])

p2 <- ggplot(df_dense_scatter_sample, aes(x, y)) +
  geom_smooth(color = NA, fill = palette[npal-1], level = 0.99) +
  geom_smooth(color = NA, fill = palette[npal-2], level = 0.95) +
  geom_smooth(
    color = palette[1],
    fill = palette[npal-3],
    size = 0.5,
    level = 0.8
  ) +
  scale_y_continuous(limits = c(-5, 5)) +
  labs(title = "Graded Confidence Band") +
  theme_plot_icon(palette[npal], palette[1])

p3 <- ggplot(df_dense_scatter_sample, aes(x, y)) +
  stat_smooth_draws(
    times = 8,
    aes(group = stat(.draw)),
    color = palette[1],
    size = 0.15
  ) +
  scale_y_continuous(limits = c(-5, 5)) +
  labs(title = "Fitted Draws") +
  theme_plot_icon(palette[npal], palette[1])

plot_grid(p1, p2, p3, ncol = 4, scale = .9)
```

For smooth line graphs, the equivalent of an error bar is a confidence band (Chapter \@ref(uncertainty-curve-fits)). It shows a range of values the line might pass through at a given confidence level. As in the case of error bars, we can draw graded confidence bands that show multiple confidence levels at once. We can also show individual fitted draws in lieu of or in addition to the confidence bands.


<!--chapter:end:directory_of_visualizations.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")
library(ggrepel)
library(grid)
library(gridExtra)
library(gtable)
library(tibble)
library(lubridate)
```

# Titles, captions, and tables {#figure-titles-captions}

A data visualization is not a piece of art meant to be looked at only for its aesthetically pleasing features. Instead, its purpose is to convey information and make a point. To reliably achieve this goal when preparing visualizations, we have to place the data into context and provide accompanying titles, captions, and other annotations. In this chapter, I will discuss how to properly title and label figures. I will also discuss how to present data in table form.

## Figure titles and captions

One critical component of every figure is the figure title. Every figure needs the title. The job of the title is to accurately convey to the reader what the figure is about, what point it makes. However, the figure title may not necessarily appear where you were expecting to see it. Consider Figure \@ref(fig:corruption-development). It's title is "Corruption and human development: The most developed countries experience the least corruption." This title is not shown above the figure, however. Instead, the title is provided as the first part of the caption block, underneath the figure display. This is the style I am using throughout this book. I consistently show figures without integrated titles and with separate captions. (One exception are the stylized plot examples in Chapter \@ref(directory-of-visualizations), which instead have titles and no captions.)

(ref:corruption-development) Corruption and human development: The most developed countries experience the least corruption. This figure was inspired by a posting in @Economist-corruption. Data sources: Transparency International & UN Human Development Report

```{r corruption-development, fig.width = 5*6/4.2, fig.asp = 0.7, fig.cap = '(ref:corruption-development)'}
country_highlight <- c("Germany", "Norway", "United States", "Greece", "Singapore", "Rwanda", "Russia", "Venezuela", "Sudan", "Iraq", "Ghana", "Niger", "Chad", "Kuwait", "Qatar", "Myanmar", "Nepal", "Chile", "Argentina", "Japan", "China")


corruption %>% filter(year == 2015) %>% na.omit() %>%
  mutate(region = case_when(
    region == "Middle East and North Africa" ~ "Middle East\nand North Africa",
    region == "Europe and Central Asia" ~ "Europe and\nCentral Asia",
    region == "Sub Saharan Africa" ~ "Sub-Saharan\nAfrica",
    TRUE ~ region),
    label = ifelse(country %in% country_highlight, country, "")
    ) %>%
  ggplot(aes(cpi, hdi)) + 
    geom_smooth(
      aes(color = "y ~ log(x)", fill = "y ~ log(x)"),
      method = 'lm', formula = y~log(x), se = FALSE, fullrange = TRUE
    ) +
    geom_point(
      aes(color = region, fill = region),
      size = 2.5, alpha = 0.5, shape = 21
    ) + 
    geom_text_repel(
      aes(label = label), color = "black", size = 10/.pt,
      point.padding = 0.1, box.padding = .6, force = 1.,
      min.segment.length = 0, seed = 7654,
      family = dviz_font_family
    ) +
    scale_color_OkabeIto(name = NULL, order = c(1:5, 8), darken = 0.3) +
    scale_fill_OkabeIto(name = NULL, order = c(1:5, 8)) +
    scale_y_continuous(
      limits = c(0.3, 1.05), breaks = c(0.2, 0.4, 0.6, 0.8, 1.0),
      expand = c(0, 0),
      name = "Human Development Index, 2015\n(1.0 = most developed)"
    ) +
    scale_x_continuous(
      limits = c(10, 95),
      breaks = c(20, 40, 60, 80, 100),
      expand = c(0, 0),
      name = "Corruption Perceptions Index, 2015 (100 = least corrupt)"
    ) +
    guides(
      color = guide_legend(
        nrow = 1,
        override.aes = list(
          linetype = c(rep("blank", 5), "solid"),
          shape = c(rep(21, 5), NA)
        )
      )
    ) +
    theme_dviz_hgrid(12, rel_small = 1) +
    theme(
      legend.position = "top",
      legend.justification = "right",
      legend.text = element_text(size = 10),
      legend.box.spacing = unit(0, "pt")
    ) -> plot_corrupt_base

## for some reason grid::forceGrob creates an empty plot, not sure why
#cur_dev <- grDevices::dev.cur()
#cowplot::png_null_device(width = 7, height = 4.9)
#null_dev <- dev.cur()
#grob_corrupt_base <- grid::forceGrob(ggplotGrob(plot_corrupt_base))
#null <- grDevices::dev.off(null_dev)
#if (cur_dev > 1 ) null <- grDevices::dev.set(cur_dev)
#ggdraw(grob_corrupt_base)


ggsave("figures/corruption_plot_base.png", plot_corrupt_base, width = 7, height = 4.9,
       dpi = 600)

ggdraw() + draw_image("figures/corruption_plot_base.png")
```

Alternatively, I could incorporate the figure title---as well as other elements of the caption, such as the data source statement---into the main display (Figure \@ref(fig:corruption-development-infographic)). In a direct comparison, you may find Figure \@ref(fig:corruption-development-infographic) more attractive than Figure \@ref(fig:corruption-development), and you may wonder why I am choosing the latter style throughout this book. I do so because the two styles have different application areas, and figures with integrated titles are not appropriate for conventional book layouts. The underlying principle is that a figure can have only one title. Either the title is integrated into the actual figure display or it is provided as the first element of the caption underneath the figure. And, if a publication is laid out such that each figure has a regular caption block underneath the display item, then the title *must* be provided in that block of text. For this reason, in the context of conventional book or article publishing, we do not normally integrate titles into figures. Figures with integrated titles, subtitles, and data source statements are appropriate, however, if they are meant to be used as stand-alone infographics or to be posted on social media or on a web page without accompanying caption text.

(ref:corruption-development-infographic) Infographic version of Figure \@ref(fig:corruption-development). The title, subtitle, and data source statements have been incorporated into the figure. This figure could be posted on the web as is or otherwise used without separate caption block.

```{r corruption-development-infographic, fig.width = 5*6/4.2, fig.asp = 4.9*(0.12+1+.07)/7, fig.cap = '(ref:corruption-development-infographic)'}
plot_corrupt_title <- ggdraw() +
  labs(title = "Corruption and human development",
       subtitle = "The most developed countries experience the least corruption") +
  theme_dviz_map(12, rel_small = 1) +
  theme(
    plot.title = element_text(size = 14, face = "bold"),
    plot.subtitle = element_text(size = 12),
    plot.margin = margin(6, 1.5, 0, 1.5)
  )

plot_corrupt_caption <- ggplot() +
  labs(caption = "Data sources: Transparency International & UN Human Development Report") +
  theme_dviz_map(12) +
  theme(plot.margin = margin(0, 1.5, 6, 1.5))


plot_grid(
  plot_corrupt_title,
  ggdraw() + draw_image("figures/corruption_plot_base.png"),
  plot_corrupt_caption,
  ncol = 1, rel_heights = c(.12, 1, .07)
)
```

```{block type='rmdtip', echo=TRUE}
If your document layout uses caption blocks underneath each figure, then place the figure titles as the first element of each caption block, not on top of the figures.
```

One of the most common mistakes I see in figure captions is the omission of a proper figure title as the first element of the caption. Take a look back at the caption to Figure \@ref(fig:corruption-development). It begins with "Corruption and human development." It *does not* begin with "This figure shows how corruption is related to human development." The first part of the caption is always the title, not a description of the contents of the figure. A title does not have to be a complete sentence, though short sentences making a clear assertion can serve as titles. For example, for Figure \@ref(fig:corruption-development), a title such as "The most developed countries are the least corrupt" would have worked fine.

## Axis and legend titles

Just like every plot needs a title, axes and legends need titles as well. (Axis titles are often colloquially referred to as *axis labels*.) Axis and legend titles and labels explain what the displayed data values are and how they map to plot aesthetics.

To present an example of a plot where all axes and legends are appropriately labeled and titled, I have taken the blue jay dataset discussed at length in Chapter \@ref(visualizing-associations) and visualized it as a bubble plot (Figure \@ref(fig:blue-jays-scatter-bubbles2)). In this plot, the axis titles clearly indicate that the *x* axis shows body mass in grams and the *y* axis shows head length in millimeters. Similarly, the legend titles show that point coloring indicates the birds' sex and point size indicates the birds' skull size in millimeters. I emphasize that for all numerical variables (body mass, head length, and skull size) the relevant titles not only state the variables shown but also the units in which the variables are measured. This is good practice and should be done whenever possible. Categorical variables (such as sex) do not require units.

(ref:blue-jays-scatter-bubbles2) Head length versus body mass for 123 blue jays. The birds' sex is indicated by color, and the birds' skull size by symbol size. Head-length measurements include the length of the bill while skull-size measurements do not. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-scatter-bubbles2, fig.asp = 3/4, fig.cap='(ref:blue-jays-scatter-bubbles2)'}
blue_jays$sex <- ifelse(blue_jays$KnownSex == "F", "female birds", "male birds")
blue_jays$sex <- factor(blue_jays$sex, levels = c("male birds", "female birds"))
ggplot(blue_jays, aes(Mass, Head, size = Skull, fill = KnownSex)) + 
  geom_point(pch = 21, color = "white") +
  scale_x_continuous(name = "body mass (g)") +
  scale_y_continuous(name = "head length (mm)", breaks = c(52, 54, 56, 58, 60)) +
  scale_fill_manual(
    values = c(F = "#D55E00", M = "#0072B2"),
    labels = c("female   ", "male"),
    name = "sex",
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      title.hjust = 0.5,
      label.position = "right",
      keyheight = grid::unit(19, "pt"),
      order = 1,
      override.aes = list(size = 4)
    )
  ) +
  scale_radius(
    name = "skull size (mm)",
    range = c(2, 7),
    limits = c(28, 34),
    breaks = c(28, 30, 32, 34),
    labels = c("28   ", "30   ", "32   ", "34"),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      title.hjust = 0.5,
      label.position = "right",
      order = 2,
      override.aes = list(fill = "gray40")
    )
  ) +
  theme_dviz_grid() +
  theme(
    legend.margin = margin(0, 0, 0, 20),
    legend.position = "top",
    legend.box = "horizontal",
    legend.box.spacing = grid::unit(0, "pt"),
    legend.justification = c(1, 0),
    legend.spacing.x = unit(2, "pt"),
    legend.spacing.y = unit(2, "pt"),
    legend.background = element_rect(fill = "white", color = NA),
    legend.key.width = unit(10, "pt")
  )
```

There are cases, however, when axis or legend titles can be omitted, namely when the labels themselves are fully explanatory. For example, a legend showing two differently colored dots labeled "female" and "male" clearly indicates that color encodes sex. The title "sex" is not required to clarify this fact, and indeed throughout this book I have often omitted the legend title for legends indicating sex or gender (see e.g. Figures \@ref(fig:titanic-passengers-by-class-sex), \@ref(fig:blue-jays-scatter-sex), or \@ref(fig:titanic-passenger-breakdown)). Similarly, country names will generally not require a title stating what they are (Figure \@ref(fig:Americas-life-expect)), nor will movie titles (Figure \@ref(fig:boxoffice-vertical)) or years (Figure \@ref(fig:tech-stocks-minimal-labeling)).

(ref:tech-stocks-minimal-labeling) Stock price over time for four major tech companies. The stock price for each company has been normalized to equal 100 in June 2012. This figure is a slightly modified version of Figure \@ref(fig:tech-stocks-good-legend) in Chapter \@ref(redundant-coding). Here, the *x* axis representing time does not have a title. It is clear from the context that the numbers 2013, 2014, etc. refer to years.

```{r tech-stocks-minimal-labeling, fig.cap = '(ref:tech-stocks-minimal-labeling)'}
price_plot_base <- ggplot(tech_stocks, aes(x = date, y = price_indexed, color = ticker)) +
  geom_line(na.rm = TRUE) +
  scale_color_manual(
    values = c("#000000", "#E69F00", "#56B4E9", "#009E73"),
    name = "",
    breaks = c("FB", "GOOG", "MSFT", "AAPL"),
    labels = c("Facebook", "Alphabet", "Microsoft", "Apple")
  ) +
  scale_x_date(
    limits = c(ymd("2012-06-01"), ymd("2017-05-31")),
    expand = c(0,0)
  ) + 
  scale_y_continuous(
    limits = c(0, 560),
    expand = c(0,0)
  ) +
  theme_dviz_hgrid() + 
  theme(plot.margin = margin(3, 7, 3, 1.5))

price_plot_base + xlab(NULL) + ylab("stock price, indexed")
```

However, we have to be careful when omitting axis or legend titles, because it is easy to misjudge what is and isn't obvious from the context. I frequently see graphs in the popular press that push omitting axis titles to a point that would make me uncomfortable. For example, some publications might produce a figure such as Figure \@ref(fig:tech-stocks-minimal-labeling-bad), assuming that the meaning of the axes is clear from the plot title and subtitle (here: "stock price over time for four major tech companies" and "the stock price for each company has been normalized to equal 100 in June 2012"). I disagree with the perspective that context clearly defines the axes. Because the caption typically doesn't include words such as "the *x*/*y* axis shows", some amount of guesswork is always required to interpret the figure. In my own experience, figures without properly labeled axes tend to leave me with a nagging feeling of uncertainty---even if I'm 95% certain I understand what is shown, I don't feel 100% certain. As a general principle, I think it is a bad practice to make your readers guess what you mean. Why would you want to create a feeling of uncertainty in your readers?

(ref:tech-stocks-minimal-labeling-bad) Stock price over time for four major tech companies. The stock price for each company has been normalized to equal 100 in June 2012. This variant of Figure \@ref(fig:tech-stocks-minimal-labeling) has been labeled as "bad" because the *y* axis now does not have a title either, and what the values shown along the *y* axis represent is not immediately obvious from the context.

```{r tech-stocks-minimal-labeling-bad, fig.cap = '(ref:tech-stocks-minimal-labeling-bad)'}
stamp_bad(
  price_plot_base + xlab(NULL) + ylab(NULL)
)
```

On the flip side, we can overdo the labeling. If the legend lists the names of four well-known companies, the legend title "company" is redundant and doesn't add anything useful (Figure \@ref(fig:tech-stocks-labeling-ugly)). Similarly, even though we generally should report units for all quantitative variables, if the *x* axis shows a few recent years titling it as "time (years AD)" is awkward (Figure \@ref(fig:tech-stocks-labeling-ugly)).

(ref:tech-stocks-labeling-ugly) Stock price over time for four major tech companies. The stock price for each company has been normalized to equal 100 in June 2012. This variant of Figure \@ref(fig:tech-stocks-minimal-labeling) has been labeled as "ugly" because it is labeled excessively. In particular, providing a unit ("years AD") for the values along the *x* axis is awkward.

```{r tech-stocks-labeling-ugly, fig.cap = '(ref:tech-stocks-labeling-ugly)'}
stamp_ugly(
  price_plot_base + xlab("time (years AD)") + ylab("stock price, indexed\n(100/share price on Jun 1, 2012)") +
    scale_color_manual(
      values = c("#000000", "#E69F00", "#56B4E9", "#009E73"),
      name = "company",
      breaks = c("FB", "GOOG", "MSFT", "AAPL"),
      labels = c("Facebook", "Alphabet", "Microsoft", "Apple")
    ) +
    theme(legend.title.align = 0.5)

)
```

Finally, in some cases it is acceptable to omit not only the axis title but the entire axis.
Pie charts typically don't have explicit axes (e.g., Figure \@ref(fig:bundestag-pie)), and neither do treemaps (Figure \@ref(fig:bridges-treemap)). Mosaic plots or bar charts can be shown without one or both axes if the meaning of the plot is otherwise clear (Figures \@ref(fig:bridges-mosaic) and \@ref(fig:titanic-passengers-by-class-sex)). Omitting explicit axes with axis ticks and tick labels signals to the reader that the qualitative features of the graph are more important than the specific data values.

## Tables

Tables are an important tool for visualizing data. Yet because of their apparent simplicity, they may not always receive the attention they need. I have shown a handful of tables throughout this book, for example Tables \@ref(tab:boxoffice-gross), \@ref(tab:titanic-ages), and \@ref(tab:color-codes). Take a moment and locate these tables, look how they are formatted, and compare them to a table you or a colleague has recently made. In all likelihood, there are important differences. In my experience, absent proper training in table formatting, few people will instinctively make the right formatting choices. In self-published documents, poorly formatted tables are even more prevalent than poorly designed figures. Also, most software commonly used to create tables provides defaults that are not recommended. For example, my version of Microsoft Word provides 105 pre-defined table styles, and of these at least 70--80 violate some of the table rules I'm going to discuss here. So if you pick a Microsoft Word table layout at random, you have an 80% chance of picking one that has issues. And if you pick the default, you will end up with a poorly formatted table every time.

Some key rules for table layout are the following:

1. Do not use vertical lines.
2. Do not use horizontal lines between data rows. (Horizontal lines as separator between the title row and the first data row or as frame for the entire table are fine.) 
3. Text columns should be left aligned.
4. Number columns should be right aligned and should use the same number of decimal digits throughout.
5. Columns containing single characters are centered.
6. The header fields are aligned with their data, i.e., the heading for a text column will be left aligned and the heading for a number column will be right aligned. 

Figure \@ref(fig:table-examples) reproduces Table \@ref(tab:boxoffice-gross) from Chapter \@ref(visualizing-amounts) in four diferent ways, two of which (a, b) violate several of these rules and two of which (c, d) do not.

(ref:table-examples) Examples of poorly and appropriately formatted tables, using the data from Table \@ref(tab:boxoffice-gross) in Chapter \@ref(visualizing-amounts). (a) This table violates numerous conventions of proper table formatting, including using vertical lines, using horizontal lines between data rows, and using centered data columns. (b) This table suffers from all problems of Table (a), and in addition it creates additional visual noise by alternating between very dark and very light rows. Also, the table header is not strongly visually separated from the table body. (c) This is an appropriately formatted table with a minimal design. (d) Colors can be used effectively to group data into rows, but the color differences should be subtle. The table header can be set off by using a stronger color. Data source: Box Office Mojo (http://www.boxofficemojo.com/). Used with permission

```{r table-examples, fig.width = 5*6/4.2, fig.asp = 0.51, fig.cap = '(ref:table-examples)'}
boxoffice <- tibble(
  Rank = 1:5,
  Title = c("Star Wars: The Last Jedi", "Jumanji: Welcome to the Jungle    ", "Pitch Perfect 3", "The Greatest Showman", "Ferdinand"),
  Amount = c("$71,565,498", "$36,169,328", "$19,928,525", "$8,805,843", "$7,316,746")
)

boxoffice_ctr <- tibble(
  Rank = 1:5,
  Title = c("Star Wars: The Last Jedi", "  Jumanji: Welcome to the Jungle  ", "Pitch Perfect 3", "The Greatest Showman", "Ferdinand"),
  Amount = c("$71,565,498", "$36,169,328", "$19,928,525", "$8,805,843", "$7,316,746")
)

table_base_size = 10
zgrob <- function(...) ggplot2::zeroGrob()

tt1 <- ttheme_minimal(
  base_size = table_base_size,
  base_family = dviz_font_family,
  core = list(
    fg_params = list(
      fontface = rep(c(1L, 3L, 1L), each = 5)
    ),
    bg_params = list(
      col = "black",
      lwd = 1
    )
  ),
  colhead = list(
    fg_params = list(
      fontface = 1L,
      fontfamily = dviz_font_family_bold
    ),
    bg_params = list(
      col = "black",
      lwd = 1
    )
  ),
  rowhead = list(fg_fun = zgrob, bg_fun = zgrob)
)

tt2 <- ttheme_default(
  base_size = table_base_size,
  base_family = dviz_font_family,
  core = list(
    fg_params = list(
      fontface = rep(c(1L, 3L, 1L), each = 5),
      col = c("white", "black")
    ),
    bg_params = list(
      col = "black",
      lwd = 0.5,
      fill = c("grey45", "grey85")
    )
  ),
  colhead = list(
    fg_params = list(
      fontface = 1L,
      fontfamily = dviz_font_family_bold
    ),
    bg_params = list(
      col = "black",
      lwd = 0.5,
      fill = "grey85"
    )
  ),
  rowhead = list(fg_fun = zgrob, bg_fun = zgrob)
)

tt3 <- ttheme_minimal(
  base_size = table_base_size,
  base_family = dviz_font_family,
  padding = unit(c(4, 3.2), "mm"),
  core = list(
    fg_params = list(
      fontface = rep(c(1L, 3L, 1L), each = 5),
      hjust = rep(c(0.5, 0, 1), each = 5),
      x = rep(c(0.5, 0.1, 0.9), each = 5)
    ),
    bg_params = list(
      col = NA
    )
  ),
  colhead = list(
    fg_params = list(
      hjust = c(0.5, 0, 1),
      x = c(0.5, 0.1, 0.9),
      fontface = 1L,
      fontfamily = dviz_font_family_bold
    ),
    bg_params = list(
      col = NA
    )
  ),
  rowhead = list(fg_fun = zgrob, bg_fun = zgrob)
)

tt4 <- ttheme_default(
  base_size = table_base_size,
  base_family = dviz_font_family,
  core = list(
    fg_params = list(
      fontface = rep(c(1L, 3L, 1L), each = 5),
      col = "black",
      hjust = rep(c(0.5, 0, 1), each = 5),
      x = rep(c(0.5, 0.1, 0.9), each = 5)
    ),
    bg_params = list(
      col = NA,
      fill = c('#D9E0EF', '#C2CCE3') #c("grey95", "grey85")
    )
  ),
  colhead = list(
    fg_params = list(
      col = "white",
      hjust = c(0.5, 0, 1),
      x = c(0.5, 0.1, 0.9),
      fontface = 1L,
      fontfamily = dviz_font_family_bold
    ),
    bg_params = list(
      col = NA,
      fill = "#4069A6"#"grey65"
    )
  ),
  rowhead = list(fg_fun = zgrob, bg_fun = zgrob)
)

# horizontal line to be used as separator
hline_top <- segmentsGrob(
  x0 = unit(0,"npc"),
  y0 = unit(1,"npc"),
  x1 = unit(1,"npc"),
  y1 = unit(1,"npc"),
  gp = gpar(lwd = 0.75, col = "black")
)
hline_bottom <- segmentsGrob(
  x0 = unit(0,"npc"),
  y0 = unit(0,"npc"),
  x1 = unit(1,"npc"),
  y1 = unit(0,"npc"),
  gp = gpar(lwd = 0.75, col = "black")
)

t1 <- tableGrob(boxoffice_ctr, rows = rep("", nrow(boxoffice)), theme = tt1)
t1$layout$clip <- "off"
t1 <- gtable_add_padding(t1, margin(14, 16, 0, -2))

t2 <- tableGrob(boxoffice_ctr, rows = rep("", nrow(boxoffice)), theme = tt2)
t2$layout$clip <- "off"
t2 <- gtable_add_padding(t2, margin(14, 16, 0, -2))

t3 <- tableGrob(boxoffice, rows = rep("", nrow(boxoffice)), theme = tt3)
t3 <- gtable_add_grob(t3,
  grobs = grobTree(hline_top, hline_bottom),
  t = 1, b = 1, l = 2, r = 4)
t3 <- gtable_add_grob(t3,
  grobs = hline_bottom,
  t = 6, b = 6, l = 2, r = 4)
t3$layout$clip <- "off"
t3 <- gtable_add_padding(t3, margin(14, 14, 0, -7))

t4 <- tableGrob(boxoffice, rows = rep("", nrow(boxoffice)), theme = tt4)
t4$layout$clip <- "off"
t4 <- gtable_add_padding(t4, margin(14, 16, 0, -2))

plot_grid(
  stamp_ugly(t1), NULL, stamp_ugly(t2),
  NULL, NULL, NULL,
  t3, NULL, t4,
  rel_widths = c(1, 0.06, 1),
  rel_heights = c(1, 0.08, 1),
  labels = c("a", "", "b", "", "", "", "c", "", "d")
)

```

When authors draw tables with horizontal lines between data rows, the intent is usually to help the eye follow individual lines. However, unless the table is very wide and sparse, this visual aid is not normally needed. We don't draw horizontal lines between rows in a piece of regular text either. The cost of horizontal (or vertical) lines is visual clutter. Compare parts (a) and (c) of Figure \@ref(fig:table-examples). Part (c) is much easier to read than part (a). If we feel that a visual aid separating table rows is necessary, then alternating lighter and darker shading of rows tends to work well without creating much clutter (Figure \@ref(fig:table-examples)d).

Finally, there is a key distinction between figures and tables in where the caption is located relative to the display item. For figures, it is customary to place the caption underneath, whereas for tables it is customary to place it above. This caption placement is guided by the way in which readers process figures and tables. For figures, readers tend to first look at the graphical display and then read the caption for context, hence the caption makes sense below the figure. By contrast, tables tend to be processed like text, from top to bottom, and reading the table contents before reading the caption will frequently not be useful. Hence, captions are placed above the table.

<!--chapter:end:figure_titles_captions.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")

library(forcats)
library(ggmap)
library(statebins)
library(sf)
library(lubridate)
library(geofacet)
```

# Visualizing geospatial data {#geospatial-data}

Many datasets contain information linked to locations in the physical world. For example, in an ecological study, a dataset may list where specific plants or animals have been found. Similarly, in a socioeconomic or political context, a dataset may contain information about where people with specific attributes (such as income, age, or educational attainment) live, or where man-made objects (e.g., bridges, roads, buildings) have been constructed. In all these cases, it can be helpful to visualize the data in their proper geospatial context, i.e., to show the data on a realistic map or alternatively as a map-like diagram.

Maps tend to be intuitive to readers but they can be challenging to design. We need to think about concepts such as map projections and whether for our specific application the accurate representation of angles or areas is more critical. A common mapping technique, the *choropleth map,* consists of representing data values as differently colored spatial areas. Choropleth maps can at times be very useful and at other times highly misleading. As an alternative, we can construct map-like diagrams called *cartograms,* which may purposefully distort map areas or represent them in stylized form, for example as equal-sized squares.


## Projections

The earth is approximately a sphere (Figure \@ref(fig:world-orthographic)), and more precisely an oblate spheroid that is slightly flattened along its axis of rotation. The two locations where the axis of rotation intersects with the spheroid are called the *poles* (north pole and south pole). We separate the spheroid into two hemispheres, the northern and the southern hemisphere, by drawing a line equidistant to both poles around the spheroid. This line is called the *equator.* To uniquely specify a location on the earth, we need three pieces of information: where we are located along the direction of the equator (the longitude), how close we are to either pole when moving perpendicular to the equator (the latitude), and how far we are from the earth's center (the altitude). Longitude, latitude, and altitude are specified relative to a reference system called the *datum.* The datum specifies properties such as the shape and size of the earth, as well as the location of zero longitude, latitude, and altitude. One widely used datum is the World Geodetic System (WGS) 84, which is used by the Global Positioning System (GPS).

(ref:world-orthographic) Orthographic projection of the world, showing Europe and Northern Africa as they would be visible from space. The lines emanating from the north pole and running south are called *meridians*, and the lines running orthogonal to the meridians are called *parallels.* All meridians have the same length but parallels become shorter the closer we are to either pole.

```{r world-orthographic, fig.width = 5.5, fig.asp = 1, fig.cap = '(ref:world-orthographic)'}
intersperse <- function(x1, x2) {
  x <- numeric(0)
  for (i in seq_along(x1)) {
    x <- c(x, x1[i], x2[i], NA)
  }
  x
}

cenlat <- 40
cenlong <- 15

ocean_col <- "#56B4E950"
land_col <- "#E69F00B0"
graticule_col <- "grey30"
line_col <- "black"

draw_ocean(cenlat, cenlong, col = ocean_col, line_col = graticule_col, lwd = 0.25)
draw_land(map_polys$world, cenlat, cenlong, col = land_col, line_col = line_col, lwd = 0.5)

p1 <- dviz.supp::orthproj(
  lat = c(0.5, 66, 63, 54, 30.5, 44.5, 59.5),
  long = c(-25, -29, 1, 31, -22, -22, -22),
  cenlat = cenlat,
  cenlong = cenlong
)

p2 <- dviz.supp::orthproj(
  lat = c(6, 65, 65, 65, 35, 35, 35),
  long = c(-28, 52, 52, 52, -35, -35, -35),
  cenlat = cenlat,
  cenlong = cenlong
)

lines(x = intersperse(p1$x, p2$x), y = intersperse(p1$y, p2$y))
par(family = dviz_font_family, ps = 12)
text(
  x = c(0.03, p2$x[1], p2$x[7], p2$x[2]),
  y = c(0.765, p2$y[1], p2$y[7], p2$y[2]),
  labels = c("north pole", "equator", "parallels", "meridians"),
  pos = c(3, 3, 2, 4),
  offset = .2
)
```
While altitude is an important quantity in many geospatial applications, when visualizing geospatial data in the form of maps we are primarily concerned with the other two dimensions, longitude and latitude. Both longitude and latitude are angles, expressed in degrees. Degrees longitude measure how far east or west a location lies. Lines of equal longitude are referred to as *meridians*, and all meridians terminate at the two poles (Figure \@ref(fig:world-orthographic)). The prime meridian, corresponding to 0&deg; longitude, runs through the village of Greenwich in the United Kingdom. The meridian opposite to the prime meridian lies at 180&deg; longitude (also referred to as 180&deg;E), which is equivalent to -180&deg; longitude (also referred to as 180&deg;W), near the international date line. Degrees latitude measure how far north or south a location lies. The equator corresponds to 0&deg; latitude, the north pole corresponds to 90&deg; latitude (also referred to as 90&deg;N), and the south pole corresponds to -90&deg; latitude (also referred to as 90&deg;S). Lines of equal latitude are referred to as *parallels*, since they run parallel to the equator.  All meridians have the same length, corresponding to half of a great circle around the globe, whereas the length of parallels depends on their latitude (Figure \@ref(fig:world-orthographic)). The longest parallel is the equator, at 0&deg; latitude, and the shortest parallels lie at the north and south poles, 90&deg;N and 90&deg;S, and have length zero.

The challenge in map-making is that we need to take the spherical surface of the earth and flatten it out so we can display it on a map. This process, called *projection,* necessarily introduces distortions, because a curved surface cannot be projected exactly onto a flat surface. Specifically, the projection can preserve either angles or areas but not both. A projection that does the former is called *conformal* and a projection that does the latter is called *equal-area*. Other projections may preserve neither angles nor areas but instead preserve other quantities of interest, such as distances to some reference point or line. Finally, some projections attempt to strike a compromise between preserving angles and areas. These compromise projections are frequently used to display the entire world in an aesthetically pleasing manner, and they accept some amount of both angular and area distortion (Figure \@ref(fig:worldmap-four-projections)). To systematize and keep track of different ways of projecting parts or all of the earth for specific maps, various standards bodies and organizations, such as the EPSG (European Petroleum Survey Group) or the ESRI (Environmental Systems Research Institute), maintain registries of projections. For example, EPSG:4326 represents unprojected longitude and latitude values in the WGS 84 coordinate system used by GPS. Several websites provide convenient access to these registered projections, including http://spatialreference.org/ and https://epsg.io/.

One of the earliest map projections in use, the Mercator projection, was developed in the 16th century for nautical navigation. It is a conformal projection that accurately represents shapes but introduces severe area distortions near the poles (Figure \@ref(fig:world-mercator)). The Mercator projection maps the globe onto a cylinder and then unrolls the cylinder to arrive at a rectangular map. Meridians in this projection are evenly spaced vertical lines, whereas parallels are horizontal lines whose spacing increases the further we move away from the equator (Figure \@ref(fig:world-mercator)). The spacing between parallels increases in proportion to the extent to which they have to be stretched closer to the poles to keep meridians perfectly vertical.

(ref:world-mercator) Mercator projection of the world. In this projection, parallels are straight horizontal lines and meridians are straight vertical lines. It is a conformal projection preserving local angles, but it introduces severe distortions in areas near the poles. For example, Greenland appears to be bigger than Africa in this projection, when in reality Africa is fourteen times bigger than Greenland (see Figures \@ref(fig:world-orthographic) and \@ref(fig:world-goode)).

```{r world-mercator, fig.width = 5, fig.asp = 0.85, fig.cap = '(ref:world-mercator)'}
world_sf <- sf::st_as_sf(rworldmap::getMap(resolution = "low"))

crs_longlat <- "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"

crs_mercator <- "+proj=merc"

# calculate bounding box in transformed coordinates
mercator_bbox <- 
  rbind(c(-180, -85), c(180, 85)) %>%
  st_multipoint() %>%
  st_sfc(
    crs = "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"
  ) %>% 
  st_transform(crs = crs_mercator)

ggplot(world_sf) + 
  geom_sf(fill = "#E69F00B0", color = "black", size = 0.5/.pt) + 
  scale_x_continuous(name = "longitude", breaks = seq(-120, 120, by = 60)) +
  scale_y_continuous(name = "latitude", breaks = seq(-80, 80, by = 20)) +
  coord_sf(
    xlim = mercator_bbox[[1]][, 1],
    ylim = mercator_bbox[[1]][, 2],
    expand = FALSE,
    crs = crs_mercator
  ) + 
  theme_dviz_grid(font_size = 12, rel_small = 1) +
  theme(
    panel.background = element_rect(fill = "#56B4E950", color = "#56B4E950"),
    panel.grid.major = element_line(color = "gray30", size = 0.25),
    axis.ticks = element_line(color = "gray30", size = 0.5/.pt)
  )
```

Because of the severe area distortions it produces, the Mercator projection has fallen out of favor for maps of the entire world. However, variants of this projection continue to live on. For example, the transverse Mercator projection is routinely used for large-scale maps that show moderately small areas (spanning less than a few degrees in longitude) at large magnification. Another variant, the web Mercator projection, was introduced by Google for Google Maps and is used by several online mapping applications. 

A whole-world projection that is perfectly area-preserving is the Goode homolosine (Figure \@ref(fig:world-goode)). It is usually shown in its interrupted form, which has one cut in the northern hemisphere and three cuts in the southern hemisphere, carefully chosen so they don't interrupt major land masses (Figure \@ref(fig:world-goode)). The cuts allow the projection to both preserve areas and approximately preserve angles, at the cost of non-contiguous oceans, a cut through the middle of Greenland, and several cuts through Antarctica. While the interrupted Goode homolosine has an unusual aesthetic and a strange name, it is a good choice for mapping applications that require accurate reproduction of areas on a global scale.

(ref:world-goode) Interrupted Goode homolosine projection of the world. This projection accurately preserves areas while minimizing angular distortions, at the cost of showing oceans and some land masses (Greenland, Antarctica) in a non-contiguous way.

```{r world-goode, fig.width = 5*6/4.2, fig.asp = 0.45, fig.cap = '(ref:world-goode)'}
crs_goode <- "+proj=igh"

# projection outline in long-lat coordinates
lats <- c(
  90:-90, # right side down
  -90:0, 0:-90, # third cut bottom
  -90:0, 0:-90, # second cut bottom
  -90:0, 0:-90, # first cut bottom
  -90:90, # left side up
  90:0, 0:90, # cut top
  90 # close
)
longs <- c(
  rep(180, 181), # right side down
  rep(c(80.01, 79.99), each = 91), # third cut bottom
  rep(c(-19.99, -20.01), each = 91), # second cut bottom
  rep(c(-99.99, -100.01), each = 91), # first cut bottom
  rep(-180, 181), # left side up
  rep(c(-40.01, -39.99), each = 91), # cut top
  180 # close
)

goode_outline <- 
  list(cbind(longs, lats)) %>%
  st_polygon() %>%
  st_sfc(
    crs = "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"
  ) %>% 
  st_transform(crs = crs_goode)

# bounding box in transformed coordinates
xlim <- c(-21945470, 21963330)
ylim <- c(-9538022, 9266738)
goode_bbox <- 
  list(
    cbind(
      c(xlim[1], xlim[2], xlim[2], xlim[1], xlim[1]), 
      c(ylim[1], ylim[1], ylim[2], ylim[2], ylim[1])
    )
  ) %>%
  st_polygon() %>%
  st_sfc(crs = crs_goode)

# area outside the earth outline
goode_without <- st_difference(goode_bbox, goode_outline)

ggplot(world_sf) + 
  geom_sf(fill = "#E69F00B0", color = "black", size = 0.5/.pt) +
  geom_sf(data = goode_without, fill = "white", color = NA) +
  geom_sf(data = goode_outline, fill = NA, color = "grey30", size = 0.5/.pt) +
  scale_x_continuous(name = NULL, breaks = seq(-120, 120, by = 60)) +
  scale_y_continuous(name = NULL, breaks = seq(-60, 60, by = 30)) +
  coord_sf(xlim = 0.95*xlim, ylim = 0.95*ylim, expand = FALSE, crs = crs_goode, ndiscr = 1000) + 
  theme_dviz_grid(font_size = 12, rel_small = 1) +
  theme(
    panel.background = element_rect(fill = "#56B4E950", color = "white", size = 1),
    panel.grid.major = element_line(color = "gray30", size = 0.25),
    axis.ticks = element_line(color = "gray30", size = 0.5/.pt)
  )
```

Shape or area distortions due to map projections are particularly prominent when we're attempting to make a map of the whole world, but they can cause trouble even at the scale of individual continents or countries. As an example, consider the United States, which consist of the "lower 48" (which are 48 contiguous states), Alaska, and Hawaii (Figure \@ref(fig:usa-orthographic)). While the lower 48 alone are reasonably easy to project onto a map, Alaska and Hawaii are so distant from the lower 48 that projecting all 50 states onto one map becomes awkward.

(ref:usa-orthographic) Relative locations of Alaska, Hawaii, and the lower 48 states shown on a globe.

```{r usa-orthographic, fig.width = 5.5, fig.asp = 1, fig.cap = '(ref:usa-orthographic)'}
cenlat <- 35
cenlong <- -130

draw_ocean(cenlat, cenlong, lwd = 0.25)
draw_land(map_polys$usa, cenlat, cenlong, col = "#D00000D0") 
draw_land(map_polys$world_no_usa, cenlat, cenlong, col = "#C0C0C0B0")
par(family = dviz_font_family_condensed, ps = 12)
text(
#  x = c(0.38, 0.05, -0.4),
#  y = c(0.15, 0.49, -0.1),
  x = c(0.36, -0.17, -0.4),
  y = c(0.13, 0.49, -0.1),
  labels = c("lower 48", "Alaska", "Hawaii"),
  col = c("white", "white", "black")
)
```

Figure \@ref(fig:usa-true-albers) shows a map of all 50 states, made using an equal-area Albers projection. This projection provides a reasonable representation of the relative shapes, areas, and locations of the 50 states, but we notice some issues. First, Alaska seems weirdly stretched out compared to how it looks, for example, in Figures \@ref(fig:world-mercator) or \@ref(fig:usa-orthographic). Second, the map is dominated by ocean/empty space. It would be preferable if we could zoom in further, so that the lower 48 states take up a larger proportion of the map area.

(ref:usa-true-albers) Map of the United States of America, using an area-preserving Albers projection (ESRI:102003, commonly used to project the lower 48 states). Alaska and Hawaii are shown in their true locations.

```{r usa-true-albers, fig.asp = 0.72, fig.cap = '(ref:usa-true-albers)'}
longs <- -180:-20
lats <- rep(89.9, length(longs))
earth_boundary <- sf::st_sfc(
    sf::st_linestring(
      cbind(longs, lats)
    ),
    crs = "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"
  )
earth_boundary <- sf::st_transform(earth_boundary, crs = "+proj=aea +lat_1=29.5 +lat_2=45.5 +lat_0=37.5 +lon_0=-96 +x_0=0 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs")
whiteout <- data.frame(
  x = earth_boundary[[1]][, 1],
  y = earth_boundary[[1]][, 2]
)

p <- ggplot(US_states_geoms$true_albers) + 
  geom_sf(fill = "#E69F00B0", color = "black", size = 0.5/.pt) +
  geom_polygon(
    data = whiteout, aes(x, y),
    fill = "white", color = "gray30",
    size = 0.5/.pt
  ) +
  coord_sf(xlim = c(-6721002, 2685733), ylim = c(-1634610, 4888053), expand = FALSE, ndiscr = 1000) +
  scale_x_continuous(name = "longitude", breaks = -20*c(3:10)) +
  scale_y_continuous(name = "latitude", breaks = (1:9)*10) +
  theme_dviz_grid(font_size = 12, rel_small = 1) +
  theme(
    #plot.background = element_rect(fill = "cornsilk"),
    panel.background = element_rect(fill = "#56B4E950"),
    panel.grid.major = element_line(color = "gray30", size = 0.25),
    axis.ticks = element_line(color = "gray30", size = 0.5/.pt)
  )

## work around bug in sf graticule code
## not needed anymore since sf 0.7-3
#b <- ggplot_build(p)
#b$layout$panel_params[[1]]$graticule$x_start[11] <- 0
#b$layout$panel_params[[1]]$graticule$y_start[11] <- 0.849
#ggdraw(ggplot_gtable(b))
p
```

To address the problem of uninteresting empty space, it is common practice to project Alaska and Hawaii separately (to minimize shape distortions) and then move them so they are shown underneath the lower 48 (Figure \@ref(fig:usa-albers)). You may notice in Figure \@ref(fig:usa-albers) that Alaska looks much smaller relative to the lower 48 than it does in Figure \@ref(fig:usa-true-albers). The reason for this discrepancy is that Alaska has not only been moved, it also has been scaled so it looks comparable in size to typical midwestern or western states. This scaling, while common practice, is highly misleading, and therefore I have labeled the figure as "bad."

(ref:usa-albers) Visualization of the United States, with the states of Alaska and Hawaii moved to lie underneath the lower 48 states. Alaska also has been scaled so its linear extent is only 35% of the state's true size. (In other words, the state's area has been reduced to approximately 12% of its true size.) Such a scaling is frequently applied to Alaska, to make it visually appear to be of similar size as typical midwestern or western states. However, the scaling is highly misleading, and therefore the figure has been labeled as "bad".

```{r usa-albers, fig.asp = 0.65, fig.cap = '(ref:usa-albers)'}
# standard US Albers map, with AK artificially small

# mimic color of transparent orange on top of transparent blue,
# as in previous maps drawn
# color was obtained by extraction from rendered png
brown <- "#deb664"

p <- ggplot(US_states_geoms$us_albers) + 
  geom_sf(fill = brown, color = "black", size = 0.5/.pt) +
  coord_sf(datum = NA, expand = FALSE) +
  theme_dviz_map() +
  theme(
    #plot.background = element_rect(fill = "cornsilk"),
    plot.margin = margin(6, 6, 1, 1.5) 
  )

stamp_bad(p)
```

Instead of both moving and scaling Alaska, we could just move it without changing its scale (Figure \@ref(fig:usa-albers-revised)). This visualization clearly shows that Alaska is the largest state, over twice the size of Texas. We are not used to seeing the U.S. shown in this way, but in my mind it is a much more reasonable representation of the 50 states than is Figure \@ref(fig:usa-albers).

(ref:usa-albers-revised) Visualization of the United States, with the states of Alaska and Hawaii moved to lie underneath the lower 48 states.

```{r usa-albers-revised, fig.asp = 0.75, fig.cap = '(ref:usa-albers-revised)'}
# revised US Albers map, with AK at its original size
ggplot(US_states_geoms$albers_revised) + 
  geom_sf(fill = brown, color = "black", size = 0.5/.pt) +
  coord_sf(datum = NA, expand = FALSE) +
  theme_dviz_map() +
  theme(
    #plot.background = element_rect(fill = "cornsilk")
  )
```


## Layers

To visualize geospatial data in the proper context, we usually create maps consisting of multiple layers showing different types of information. To demonstrate this concept, I will visualize the locations of wind turbines in the San Francisco Bay area. In the Bay Area, wind turbines are clustered in two locations. One location, which I will refer to as the Shiloh Wind Farm, lies near Rio Vista and the other lies east of Hayward near Tracy (Figure \@ref(fig:sfbay-overview)).

(ref:sfbay-overview) Wind turbines in the San Francisco Bay Area. Individual wind turbines are shown as purple-colored dots. Two regions with a high concentration of wind turbines are highlighted with black rectangles. I refer to the wind turbines near Rio Vista collectively as the Shiloh Wind Farm. Map tiles by Stamen Design, under CC BY 3.0. Map data by OpenStreetMap, under ODbL. Wind turbine data: United States Wind Turbine Database

```{r sfbay-overview, fig.width = 5.5*6/4.2, fig.asp = 0.75, fig.cap = '(ref:sfbay-overview)'}
# From http://www.csgnetwork.com/degreelenllavcalc.html
# Length Of A Degree Of Longitude In Meters at 38deg lat
m_per_deg <- 87832.42967867786

sfbay_scale = data.frame(
  x = -122.83,
  xend = -122.83 + 10000/m_per_deg,
  y = 37.24,
  yend = 37.24,
  label = "10km"
)

sfbay_bbox <- c(left = -122.88, bottom = 37.20, right = -120.88, top = 38.31)

wind_sfbay <- wind_turbines %>%
  filter(
    xlong < sfbay_bbox["right"],
    xlong > sfbay_bbox["left"],
    ylat > sfbay_bbox["bottom"],
    ylat < sfbay_bbox["top"]
  )

shiloh_bbox <- c(left = -121.9, bottom = 38.06, right = -121.71, top = 38.20)
tracy_bbox <- c(left = -121.73, bottom = 37.66, right = -121.55, top = 37.81)

p1 <- ggmap(sfbay_maps$sfbay_bg)  + 
  inset_ggmap(sfbay_maps$sfbay_lines) +
  geom_point(
    data = wind_sfbay,
    aes(x = xlong, y = ylat),
    size = 0.1,
    color = "#A825A8",
    alpha = 1/3
  ) +
  geom_rect(
    data = data.frame(rbind(t(shiloh_bbox), t(tracy_bbox))),
    aes(xmin = left, xmax = right, ymin = bottom, ymax = top),
    size = 0.5,
    color = "black",
    fill = NA,
    inherit.aes = FALSE
  ) +
  geom_text(
    data = data.frame(x = 0.5*(shiloh_bbox['left'] + shiloh_bbox['right']), y = shiloh_bbox['top'], label = "Shiloh Wind Farm"),
    aes(x, y, label = label),
    hjust = 0.512,
    vjust = -0.51,
    family = dviz_font_family,
    color = "white",
    size = 11/.pt
  ) +
  geom_text(
    data = data.frame(x = 0.5*(shiloh_bbox['left'] + shiloh_bbox['right']),
                      y = shiloh_bbox['top'], label = "Shiloh Wind Farm"),
    aes(x, y, label = label),
    hjust = 0.5,
    vjust = -0.5,
    family = dviz_font_family,
    size = 11/.pt
  ) +
  inset_ggmap(sfbay_maps$sfbay_labels) +
  geom_segment(
    data = sfbay_scale,
    aes(x, y, xend = xend, yend = yend),
    size = 1
  ) +
  geom_text(
    data = sfbay_scale,
    aes(0.5*(x+xend), y, label = label),
    hjust = 0.5,
    vjust = -0.5,
    family = dviz_font_family,
    size = 10/.pt
  ) +
  ggspatial::annotation_north_arrow(
    width = grid::unit(1, "cm"),
    height = grid::unit(1, "cm"),
    pad_x = grid::unit(0.25, "cm"),
    pad_y = grid::unit(0.5, "cm"),
    style = ggspatial::north_arrow_fancy_orienteering(
      line_width = 1,
      text_size = 12,
      text_family = dviz_font_family
    ),
    location ="tr"
  ) +
  theme_dviz_map()

p1
```

Figure \@ref(fig:sfbay-overview) consists of four separate layers. At the bottom, we have the terrain layer, which shows hills, valleys, and water. The next layer shows the road network. On top of the road layer, I have placed a layer indicating the location of individual wind turbines. This layer also contains the two rectangles highlighting the majority of the wind turbines. Finally, the top layer adds the locations and names of cities. These four layers are shown separately in Figure \@ref(fig:sfbay-layers). For any given map we want to make, we may want to add or remove some of these layers. For example, if we wanted to draw a map of voting districts, we might consider terrain information to be irrelevant and distracting. Alternatively, if we wanted to draw a map of exposed or covered roof areas to assess potential for solar power generation, we might want to replace terrain information with satellite imagery that shows individual roofs and actual vegetation. You can interactively try these different types of layers in most online map applications, such as Google Maps. I would like to emphasize that regardless of which layers we decide to keep or remove, it is generally recommended to add a scale bar and a north arrow. The scale bar helps readers understand the size of the spatial features shown in the map, while the north arrow clarifies the map's orientation.

(ref:sfbay-layers) The individual layers of Figure \@ref(fig:sfbay-overview). From bottom to top, the figure consists of a terrain layer, a roads layer, a layer showing the wind turbines, and a layer labeling cities and adding a scale bar and north arrow. Map tiles by Stamen Design, under CC BY 3.0. Map data by OpenStreetMap, under ODbL. Wind turbine data source: United States Wind Turbine Database

```{r sfbay-layers, fig.width = 5.5*6/4.2, fig.asp = 0.75, fig.cap = '(ref:sfbay-layers) '}
l1 <- ggmap(sfbay_maps$sfbay_bg) + labs(subtitle = "terrain") + 
  geom_rect(
    data = data.frame(t(sfbay_bbox)),
    aes(xmin = left, xmax = right, ymin = bottom, ymax = top),
    fill = NA, color = "black",
    size = 0.5,
    inherit.aes  = FALSE
  ) +
  theme_dviz_map() +
  theme(plot.subtitle = element_text(margin = margin(0, 0, 3, 0)))

l2 <- ggmap(sfbay_maps$sfbay_lines) + labs(subtitle = "roads") + 
  geom_rect(
    data = data.frame(t(sfbay_bbox)),
    aes(xmin = left, xmax = right, ymin = bottom, ymax = top),
    fill = NA, color = "black",
    size = 0.5,
    inherit.aes  = FALSE
  ) +
  theme_dviz_map() +
  theme(plot.subtitle = element_text(margin = margin(0, 0, 3, 0)))


l3 <- ggmap(sfbay_maps$sfbay_labels) + 
  geom_segment(
    data = sfbay_scale,
    aes(x, y, xend = xend, yend = yend),
    size = .5*1
  ) +
  geom_text(
    data = sfbay_scale,
    aes(0.5*(x+xend), y, label = label),
    hjust = 0.5,
    vjust = -0.5,
    family = dviz_font_family,
    size = .5*10/.pt
  ) +
  geom_rect(
    data = data.frame(t(sfbay_bbox)),
    aes(xmin = left, xmax = right, ymin = bottom, ymax = top),
    fill = NA, color = "black",
    size = 0.5,
    inherit.aes  = FALSE
  ) +
  ggspatial::annotation_north_arrow(
    width = grid::unit(.5*1, "cm"),
    height = grid::unit(.5*1, "cm"),
    pad_x = grid::unit(.5*0.25, "cm"),
    pad_y = grid::unit(.5*0.5, "cm"),
    style = ggspatial::north_arrow_fancy_orienteering(
      line_width = .5*1,
      text_size = .5*12,
      text_family = dviz_font_family
    ),
    location ="tr"
  ) +
 labs(subtitle = "city labels, scale bar") + 
 theme_dviz_map() +
 theme(plot.subtitle = element_text(margin = margin(0, 0, 3, 0)))


l4 <- ggmap(sfbay_maps$sfbay_bg) +
  geom_rect(
    data = data.frame(t(sfbay_bbox)),
    aes(xmin = left, xmax = right, ymin = bottom, ymax = top),
    fill = "white", color = "black",
    size = 0.5,
    inherit.aes  = FALSE
  ) + 
  geom_point(
    data = wind_sfbay,
    aes(x = xlong, y = ylat),
    size = .5*0.1,
    color = "#A825A8",
    alpha = 1/3
  ) +
  geom_rect(
    data = data.frame(rbind(t(shiloh_bbox), t(tracy_bbox))),
    aes(xmin = left, xmax = right, ymin = bottom, ymax = top),
    size = .5*0.5,
    color = "black",
    fill = NA,
    inherit.aes = FALSE
  ) +
  geom_text(
    data = data.frame(x = 0.5*(shiloh_bbox['left'] + shiloh_bbox['right']), y = shiloh_bbox['top'], label = "Shiloh Wind Farm"),
    aes(x, y, label = label),
    hjust = 0.5,
    vjust = -0.5,
    family = dviz_font_family,
    size = .5*11/.pt
  ) +
  labs(subtitle = "wind turbines") +
  theme_dviz_map() +
  theme(plot.subtitle = element_text(margin = margin(0, 0, 3, 0)))

plot_grid(
  l1, NULL, l2,
  l4, NULL, l3,
  rel_widths = c(1, .05, 1)
)

# fig asp: 418/519 ~= 0.805
```

All the concepts discussed in Chapter \@ref(aesthetic-mapping) of mapping data onto aesthetics carry over to maps. We can place data points into their geographic context and show other data dimensions via aesthetics such as color or shape. For example, Figure \@ref(fig:shiloh-map) provides a zoomed-in view of the rectangle labeled "Shiloh Wind Farm" in Figure \@ref(fig:sfbay-overview). Individual wind turbines are shown as dots, with the color of the dots representing when a specific turbine was built and the shape representing the project to which the wind turbine belongs. A map such as this one can provide a quick overview of how an area was developed. E.g., here we see that EDF Renewables is a relatively small project built before 2000, High Winds is a moderately sized project build between 2000 and 2004, and Shiloh and Solano are the largest two projects in the area, both built over an extended period of time.

(ref:shiloh-map) Location of individual wind turbines in the Shiloh Wind Farm. Each dot highlights the location of one wind turbine. The map area corresponds to the rectangle in Figure \@ref(fig:sfbay-overview). Dots are colored by when the wind turbine was built, and the shape of the dots represents the various projects to which the individual wind turbines belong. Map tiles by Stamen Design, under CC BY 3.0. Map data by OpenStreetMap, under ODbL. Wind turbine data source: United States Wind Turbine Database

```{r shiloh-map, fig.asp = 0.75, fig.cap = '(ref:shiloh-map)'}
# From http://www.csgnetwork.com/degreelenllavcalc.html
# Length Of A Degree Of Longitude In Meters at 38deg lat
m_per_deg <- 87832.42967867786

shiloh_scale = data.frame(
  x = -121.735,
  xend = -121.735 + 2000/m_per_deg,
  y = 38.064,
  yend = 38.064,
  label = "2000m"
)

#bbox <- c(left = -121.9, bottom = 38.06, right = -121.71, top = 38.20)
wind_shiloh <- wind_turbines %>%
  filter(
    xlong < shiloh_bbox["right"],
    xlong > shiloh_bbox["left"],
    ylat > shiloh_bbox["bottom"],
    ylat < shiloh_bbox["top"]
  ) %>%
  mutate(
    name = fct_relevel(fct_collapse(p_name,
      `EDF Renewables` = "EDF Renewable V",
      `High Winds` = "High Winds",
      `Shiloh` = c("Shiloh Wind Project", "Shiloh II", "Shiloh III", "Shiloh IV"),
      `Solano` = c("Solano Phase 3", "Solano Phase IIA", "Solano Wind Project", "Solano Wind Project, Phase I", "Solano Wind Project, Phase IA"),
      `other` = c("Montezuma", "Montezuma Winds II", "unknown Solano County")
    ), "EDF Renewables", "High Winds", "Shiloh", "Solano", "other"),
    year_range = cut(
      p_year,
      breaks = c(1980, 2000, 2005, 2010, 2015),
      labels = c("before 2000", "2000 to 2004", "2005 to 2009", "2010 to 2014"),
      right = FALSE
    )
  )

p2 <- ggmap(sfbay_maps$shiloh_terrain)  + 
  geom_point(
    data = wind_shiloh,
    aes(x = xlong, y = ylat, fill = year_range, shape = name),
    size = 1.5,
    color = "black", stroke = 0.2
  ) +
  geom_segment(
    data = shiloh_scale,
    aes(x, y, xend = xend, yend = yend),
    size = 1
  ) +
  geom_text(
    data = shiloh_scale,
    aes(0.5*(x+xend), y, label = label),
    hjust = 0.5,
    vjust = -0.5,
    family = dviz_font_family,
    size = 10/.pt
  ) +
  ggspatial::annotation_north_arrow(
    width = grid::unit(1, "cm"),
    height = grid::unit(1, "cm"),
    pad_x = grid::unit(0.2, "cm"),
    pad_y = grid::unit(0.2, "cm"),
    style = ggspatial::north_arrow_fancy_orienteering(
      line_width = 1,
      text_size = 12,
      text_family = dviz_font_family
    ),
    location ="bl"
  ) +
  xlab(label = NULL) +
  ylab(label = NULL) +
  scale_fill_viridis_d(
    name = "year built",
    option = "A", end = .95, begin = 0.3, direction = -1,
    guide = guide_legend(
      order = 2,
      reverse = FALSE,
      override.aes = list(shape = 22, size = 4, stroke = 0))
  ) +
  scale_shape_manual(
    name = "project name",
    values = 21:25,
    guide = guide_legend(
      order = 1,
      override.aes = list(
        fill = "grey70",
        size = 2
      )
    )
  ) +
  theme_dviz_map(12) +
  theme(
    legend.key.width = grid::unit(12, "pt")
  )

p2
```


## Choropleth mapping {#choropleth-mapping}

We frequently want to show how some quantity varies across locations. We can do so by coloring individual regions in a map according to the data dimension we want to display. Such maps are called *choropleth maps.*

As a simple example, consider the population density (persons per square kilometer) across the United States. We take the population number for each county in the U.S., divide it by the county's surface area, and then draw a map where the color of each county corresponds to the ratio between population number and area (Figure \@ref(fig:population-density-counties)). We can see how the the major cities on the east and the west coast are the most populated areas of the U.S., the great plains and western states have low population densities, and the state of Alaska is the least populated of all.

(ref:population-density-counties) Population density in every U.S. county, shown as a choropleth map. Population density is reported as persons per square kilometer. Data source: 2015 Five-Year American Community Survey

```{r population-density-counties, fig.asp = 0.73, fig.cap = '(ref:population-density-counties)'}
# x range: -3683715  2258154
# y range: -2839538  1558935

US_counties_income <- mutate(US_counties_income, logdens = log(as.numeric(popdens)*1e6))

p <- ggplot(US_counties_income) + 
  geom_sf(aes(color = logdens, fill = logdens), size = 0.1) + 
  geom_sf(data = US_states_geoms$albers_revised, fill = NA, color = "grey30", size = 0.2) +
  coord_sf(datum = NA, expand = FALSE) +
  scale_x_continuous(limits = c(-4000000, 2300000)) +
  scale_fill_continuous_sequential(
    aesthetics = c("color", "fill"),
    palette = "YlGnBu", rev = TRUE, cmax = 20, c2 = 20, p2 = 1.75,
    name = "population density\n(persons / square km)",
    limits = log(c(0.01, 30000)),
    breaks = log(c(0.01, 1, 100, 10000)),
    labels = c("0.01", "1", "100", "10,000"),
    guide = guide_colorbar(
      frame.colour = "black",
      ticks.colour = "white",
      barwidth = grid::unit(15, "pt"),
      barheight = grid::unit(90, "pt")
    )
  ) +
  theme_dviz_map(12, rel_small = 1) +
  theme(
    #plot.background = element_rect(fill = "cornsilk"),
    legend.position = c(0, 1),
    legend.justification = c(0, 1),
    legend.spacing.x = grid::unit(3, "pt"),
    legend.title = element_text(hjust = 0.5),
    plot.margin = margin(3, 3, 3, 1.5)
  )

ggdraw(align_legend(p))
```


Figure \@ref(fig:population-density-counties) uses light colors to represent low population densities and dark colors to represent high densities, so that high-density metropolitan areas stand out as dark colors on a background of light colors. We tend to associate darker colors with higher intensities when the background color of the figure is light. However, we can also pick a color scale where high values light up on a dark background (Figure \@ref(fig:population-density-counties2)). As longs as the lighter colors fall into the red-yellow spectrum, so that they appear to be glowing, they can be perceived as representing higher intensities. As a general principle, when figures are meant to be printed on white paper then light-colored background areas (as in Figure \@ref(fig:population-density-counties)) will typically work better. For online viewing or on a dark background, dark-colored background areas (as in Figure \@ref(fig:population-density-counties2)) may be preferable.

(ref:population-density-counties2) Population density in every U.S. county, shown as a choropleth map. This map is identical to Figure \@ref(fig:population-density-counties) except that now the color scale uses light colors for high population densities and dark colors for low population densities. Data source: 2015 Five-Year American Community Survey

```{r population-density-counties2, fig.asp = 0.73, fig.cap = '(ref:population-density-counties2)'}
# x range: -3683715  2258154
# y range: -2839538  1558935

p <- ggplot(US_counties_income) + 
  geom_sf(aes(color = logdens, fill = logdens), size = 0.1) +
  # state boundaries don't look good with this color scale
  geom_sf(data = US_states_geoms$albers_revised, fill = NA, color = "black", size = 0.2) +
  coord_sf(datum = NA, expand = FALSE) +
  scale_x_continuous(limits = c(-4000000, 2300000)) +
  scale_fill_continuous_sequential(
    aesthetics = c("color", "fill"),
    palette = "Lajolla", rev = TRUE, p1 = 2, p2 = 1.3,
    name = "population density\n(persons / square km)",
    limits = log(c(0.01, 30000)),
    breaks = log(c(0.01, 1, 100, 10000)),
    labels = c("0.01", "1", "100", "10,000"),
    guide = guide_colorbar(
      frame.colour = "black",
      ticks.colour = "white",
      barwidth = grid::unit(15, "pt"),
      barheight = grid::unit(90, "pt")
    )
  ) +
  theme_dviz_map(12, rel_small = 1) +
  theme(
    #plot.background = element_rect(fill = "cornsilk"),
    legend.position = c(0, 1),
    legend.justification = c(0, 1),
    legend.spacing.x = grid::unit(3, "pt"),
    legend.title = element_text(hjust = 0.5),
    plot.margin = margin(3, 3, 3, 1.5)
  )

ggdraw(align_legend(p))
```


(ref:median-income-counties) Median income in every U.S. county, shown as a choropleth map. Data source: 2015 Five-Year American Community Survey


```{r median-income-counties, eval = FALSE, fig.asp = 0.73, fig.cap = '(ref:median-income-counties)'}
# x range: -3683715  2258154
# y range: -2839538  1558935

p <- ggplot(US_counties_income) + 
  geom_sf(aes(color = median_income, fill = median_income), size = 0.1) + 
  geom_sf(data = US_states_geoms$albers_revised, fill = NA, color = "grey30", size = 0.2) +
  coord_sf(datum = NA, expand = FALSE) +
  scale_x_continuous(limits = c(-4000000, 2300000)) +
  scale_fill_continuous_sequential(
    aesthetics = c("color", "fill"),
    #palette = "Lajolla", rev = TRUE, p1 = 2, p2 = 1.3,
    #palette = "BlueYellow", rev = TRUE, l1 = 15, p2 = 1.7,
    h1 = -83, h2 = 20, c1 = 30, cmax = 40, c2 = 0, l1 = 20, l2 = 100, p1 = 1, p2 = 1.2, rev = TRUE, 
    name = "median income",
    limits = c(0, 130000),
    breaks = c(0, 50000, 100000),
    labels = c("$0", "$50,000", "$100,000"),
    guide = guide_colorbar(
      frame.colour = "black",
      ticks.colour = "white",
      barwidth = grid::unit(15, "pt"),
      barheight = grid::unit(90, "pt")
    )
  ) +
  theme_dviz_map(12, rel_small = 1) +
  theme(
    #plot.background = element_rect(fill = "cornsilk"),
    legend.position = c(0, 1),
    legend.justification = c(0, 1),
    legend.spacing.x = grid::unit(3, "pt"),
    legend.title = element_text(hjust = 0.5),
    plot.margin = margin(3, 3, 3, 1.5)
  )

ggdraw(align_legend(p))
```

Choropleths work best when the coloring represents a density (i.e., some quantity divided by surface area, as in Figures \@ref(fig:population-density-counties) and \@ref(fig:population-density-counties2)). We perceive larger areas as corresponding to larger amounts than smaller areas (see also the chapter on proportional ink, Chapter \@ref(proportional-ink)), and shading by density corrects for this effect. However, in practice, we often see choropleths colored according to some quantity that is not a density. For example, in Figure \@ref(fig:map-Texas-income) I showed a choropleth of median annual income in Texas counties. Such choropleth maps can be appropriate when they are prepared with caution. There are two conditions under which we can color-map quantities that are not densities: First, if all the individual areas we color have approximately the same size and shape, then we don't have to worry about some areas drawing disproportionate attention solely due to their size. Second, if the individual areas we color are relatively small compared to the overall size of the map and if the quantity that color represents changes on a scale larger than the individual colored areas, then again we don't have to worry about some areas drawing disproportionate attention solely due to their size. Both of these conditions are approximately met in Figure \@ref(fig:map-Texas-income).

It is also important to consider the effect of continuous versus discrete color scales in choropleth mapping. While continuous color scales tend to look visually appealing (e.g., Figures \@ref(fig:population-density-counties) and \@ref(fig:population-density-counties2)), they can be difficult to read. We are not very good at recognizing a specific color value and matching it against a continuous scale. Therefore, it is often appropriate to bin the data values into discrete groups that are represented with distinct colors. On the order of four to six bins is a good choice. The binning sacrifices some information, but on the flip side the binned colors can be uniquely recognized. As an example, Figure \@ref(fig:median-income-counties-binned) expands the map of median income in Texas counties (Figure \@ref(fig:map-Texas-income)) to all counties in the U.S., and it uses a color scale consisting of five distinct income bins. 

(ref:median-income-counties-binned) Median income in every U.S. county, shown as a choropleth map. The median income values have been binned into five distinct groups, because binned color scales are generally easier to read than continuous color scales. Data source: 2015 Five-Year American Community Survey

```{r median-income-counties-binned, fig.asp = 0.73, fig.cap = '(ref:median-income-counties-binned)'}
# x range: -3683715  2258154
# y range: -2839538  1558935

US_counties_income <- mutate(
  US_counties_income,
  income_bins = cut(
      ifelse(is.na(median_income), 35000, median_income), # hide missing value
      breaks = c(0, 30000, 55000, 80000, 105000, 150000),
      labels = c("< $30k", "$30k to $60k", "$60k to $85k", "$85k to $105k", "> $105k"),
      right = FALSE
    )
  )

p <- ggplot(US_counties_income) + 
  geom_sf(aes(color = income_bins, fill = income_bins), size = 0.1) + 
  geom_sf(data = US_states_geoms$albers_revised, fill = NA, color = "grey30", size = 0.2) +
  coord_sf(datum = NA, expand = FALSE) +
  scale_x_continuous(limits = c(-4000000, 2300000)) +
  scale_fill_discrete_sequential(
    aesthetics = c("color", "fill"),
    h1 = -83, h2 = 20, c1 = 30, cmax = 40, c2 = 0, l1 = 20, l2 = 100, p1 = 1, p2 = 1.2, rev = TRUE, 
    name = "median income",
    nmax = 6,
    order = 2:6,
    guide = guide_legend(
      override.aes = list(colour = "white", size = 1),
      reverse = TRUE
    )
  ) +
  theme_dviz_map(12, rel_small = 1) +
  theme(
    #plot.background = element_rect(fill = "cornsilk"),
    legend.position = c(0, 1),
    legend.justification = c(0, 1),
    legend.spacing.x = grid::unit(3, "pt"),
    legend.spacing.y = grid::unit(3, "pt"),
    legend.title = element_text(hjust = 0.5),
    legend.key.width = grid::unit(18, "pt"),
    legend.key.height = grid::unit(15, "pt"),
    plot.margin = margin(3, 3, 3, 1.5)
  )

p
```

Even though counties are not quite as equal-sized and even-shaped across the entire U.S. as they are just within Texas, I think Figure \@ref(fig:median-income-counties-binned) still works as a choropleth map. No individual county overly dominates the map. However, things look different when we draw a comparable map at the state level (Figure \@ref(fig:median-income-states)). Then Alaska dominates the choropleth and, because of its size, suggests that median incomes above \$70,000 are common. Yet Alaska is very sparsely populated (see Figures \@ref(fig:population-density-counties) and \@ref(fig:population-density-counties2)), and thus the income levels in Alaska apply only to a small portion of the U.S. population. The vast majority of U.S. counties, which are nearly all more populous than counties in Alaska, have a median income of below \$60,000.

(ref:median-income-states) Median income in every U.S. state, shown as a choropleth map. This map is visually dominated by the state of Alaska, which has a high median income but very low population density. At the same time, the densely populated high-income states on the East Coast do not appear very prominent on this map. In aggregate, this map provides a poor visualization of the income distribution in the U.S., and therefore I have labeled it as "bad." Data source: 2015 Five-Year American Community Survey


```{r median-income-states, fig.asp = 0.73, fig.cap = '(ref:median-income-states)'}
# x range: -3683715  2258154
# y range: -2839538  1558935

US_income <- mutate(
  US_income,
  income_bins = cut(
      ifelse(is.na(median_income), 25000, median_income), # hide missing value
      breaks = c(0, 40000, 50000, 60000, 70000, 80000),
      labels = c("< $40k", "$40k to $50k", "$50k to $60k", "$60k to $70k", "> $70k"),
      right = FALSE
    )
  )

p <- ggplot(US_income, aes(fill = income_bins)) + 
  geom_sf(color = "grey30", size = 0.2) + 
  coord_sf(datum = NA, expand = FALSE) +
  scale_x_continuous(limits = c(-4000000, 2300000)) +
  scale_fill_discrete_sequential(
    h1 = -83, h2 = 20, c1 = 30, cmax = 40, c2 = 0, l1 = 20, l2 = 100, p1 = 1, p2 = 1.2, rev = TRUE, 
    name = "median income",
    nmax = 7,
    order = 2:6,
    guide = guide_legend(
      override.aes = list(colour = "white", size = 1),
      reverse = TRUE
    )
  ) +
  theme_dviz_map(12, rel_small = 1) +
  theme(
    #plot.background = element_rect(fill = "cornsilk"),
    legend.position = c(0, 1),
    legend.justification = c(0, 1),
    legend.spacing.x = grid::unit(3, "pt"),
    legend.spacing.y = grid::unit(3, "pt"),
    legend.title = element_text(hjust = 0.5),
    legend.key.width = grid::unit(18, "pt"),
    legend.key.height = grid::unit(15, "pt"),
    plot.margin = margin(3, 3, 3, 1.5)
  )

stamp_bad(p)
```


## Cartograms

Not every map-like visualization has to be geographically accurate to be useful. For example, the problem with Figure \@ref(fig:median-income-states) is that some states take up a comparatively large area but are sparsely populated while others take up a small area yet have a large number of inhabitants. What if we deformed the states so their size was proportional to their number of inhabitants? Such a modified map is called a *cartogram*, and Figure \@ref(fig:median-income-cartogram) shows what it can look like for the median income dataset. We can still recognize individual states, yet we also see how the adjustment for population numbers has introduced important modifications. The east coast states, Florida, and California have grown a lot in size, whereas the other western states and Alaska have collapsed.

(ref:median-income-cartogram) Median income in every U.S. state, shown as a cartogram. The shapes of individual states have been modified such that their area is proportional to their number of inhabitants. Data source: 2015 Five-Year American Community Survey

```{r median-income-cartogram, fig.asp = 0.73, fig.cap = '(ref:median-income-cartogram)'}
# copy binned data over, order of both data frames is the same
US_income_cartogram$income_bins <- US_income$income_bins

p <- ggplot(US_income_cartogram, aes(fill = income_bins)) + 
  geom_sf(color = "grey30", size = 0.2) + coord_sf(datum = NA, expand = FALSE) +
  scale_x_continuous(limits = c(-3900000, 2500000)) +
  scale_fill_discrete_sequential(
    h1 = -83, h2 = 20, c1 = 30, cmax = 40, c2 = 0, l1 = 20, l2 = 100, p1 = 1, p2 = 1.2, rev = TRUE, 
    name = "median income",
    nmax = 7,
    order = 2:6,
    guide = guide_legend(
      override.aes = list(colour = "white", size = 1),
      reverse = TRUE
    )
  ) +
  theme_dviz_map(12, rel_small = 1) +
  theme(
    #plot.background = element_rect(fill = "cornsilk"),
    legend.position = c(0, 1),
    legend.justification = c(0, 1),
    legend.spacing.x = grid::unit(3, "pt"),
    legend.spacing.y = grid::unit(3, "pt"),
    legend.title = element_text(hjust = 0.5),
    legend.key.width = grid::unit(18, "pt"),
    legend.key.height = grid::unit(15, "pt"),
    plot.margin = margin(3, 3, 3, 1.5)
  )

p
```

As an alternative to a cartogram with distorted shapes, we can also draw a much simpler *cartogram heatmap,* where each state is represented by a colored square (Figure \@ref(fig:median-income-statebins)). While this representation does not correct for the population number in each state, and thus underrepresents more populous states and overrepresents less populous states, at least it treats all states equally and doesn't weigh them arbitrarily by their shape or size.

(ref:median-income-statebins) Median income in every U.S. state, shown as a cartogram heatmap. Each state is represented by an equally sized square, and the squares are arranged according to the approximate position of each state relative to the other states. This representation gives the same visual weight to each state. Data source: 2015 Five-Year American Community Survey

```{r median-income-statebins, fig.asp = 0.62, fig.cap = '(ref:median-income-statebins)'}
filter(US_income, name != "Puerto Rico", GEOID != "11") %>% # remove Puerto Rico and DC
  ggplot(aes(state = name, fill = income_bins)) +
  geom_statebins(family = dviz.supp::dviz_font_family,
                 lbl_size = 14/.pt) +
  expand_limits(x = -1.3) + # make space for legend
  coord_equal(expand = FALSE) +
   scale_fill_discrete_sequential(
    h1 = -83, h2 = 20, c1 = 30, cmax = 40, c2 = 0, l1 = 20, l2 = 100, p1 = 1, p2 = 1.2, rev = TRUE, 
    name = "median income",
    nmax = 7,
    order = 2:6,
    guide = guide_legend(
      override.aes = list(colour = "white", size = 1),
      reverse = TRUE
    )
  ) +
  theme_dviz_map(12, rel_small = 1) +
  theme(
    #plot.background = element_rect(fill = "cornsilk"),
    legend.background = element_blank(),
    legend.position = c(0, 1),
    legend.justification = c(0, 1),
    legend.spacing.x = grid::unit(3, "pt"),
    legend.spacing.y = grid::unit(3, "pt"),
    legend.title = element_text(hjust = 0.5),
    legend.key.width = grid::unit(18, "pt"),
    legend.key.height = grid::unit(15, "pt")
  )
```

Finally, we can draw more complex cartograms by placing individual plots at the location of each state. For example, if we want to visualize the evolution of the unemployment rate over time for each state, it can help to draw an individual graph for each state and then arrange the graphs based on the approximate relative position of the states to each other (Figure \@ref(fig:unemployment-geofacet)). For somebody who is familiar with the geography of the United States, this arrangement may make it easier to find the graphs for specific states than arranging them, for example, in alphabetical order. Furthermore, one would expect neighboring states to display similar patterns, and Figure \@ref(fig:unemployment-geofacet) shows that this is indeed the case. 

(ref:unemployment-geofacet) Unemployment rate leading up to and following the 2008 financial crisis, by state. Each panel shows the unemployment rate for one state, including the District of Columbia (DC), from January 2007 through May 2013. Vertical grid lines mark January of 2008, 2010, and 2012. States that are geographically close tend to show similar trends in the unemployment rate. Data source: U.S. Bureau of Labor Statistics

```{r unemployment-geofacet, fig.width = 5.5*6/4.2, fig.asp = 0.75, fig.cap = '(ref:unemployment-geofacet)'}
adjust_labels <- as_labeller(
  function(x) {
    case_when(
      x == "New Hampshire" ~ "N. Hampshire",
      x == "District of Columbia" ~ "DC",
      TRUE ~ x
    )
  }
)

house_prices %>% 
  filter(
    date >= ymd("2007-01-01"),
    date <= ymd("2013-05-31")
  ) %>%
  ggplot(aes(date, unemploy_perc)) + 
  geom_area(fill = "#56B4E9", alpha = 0.7) +
  geom_line() + 
  scale_y_continuous(
    name = "unemployment rate",
    limits = c(0, 16), expand = c(0, 0),
    breaks = c(0, 5, 10, 15),
    labels = c("0%", "5%", "10%", "15%")
  ) +
  scale_x_date(
    name = NULL,
    breaks = ymd(c("2008-01-01", "2010-01-01", "2012-01-01")),
    labels = c("'08", "'10", "'12"),
    expand = c(0, 0)
  ) +
  coord_cartesian(clip = "off") +
  facet_geo(~state, grid = "us_state_grid1", labeller = adjust_labels) +
  theme_dviz_grid(12, dviz_font_family_condensed, rel_small = 10/12) +
  theme(
    strip.text = element_text(
      family = dviz_font_family_condensed,
      margin = margin(3, 3, 3, 3)
    ),
    axis.line.x = element_blank(),
    panel.spacing.x = grid::unit(5, "pt"),
    panel.spacing.y = grid::unit(5, "pt"),
    panel.grid.major = element_line(color = "gray80"),
    panel.background = element_rect(fill = "gray90")
  ) -> p

p
```


<!--chapter:end:geospatial_data.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")
```

# (PART\*) Part III: Miscellaneous topics {-}

# Understanding the most commonly used image file formats {#image-file-formats}

Anybody who is making figures for data visualization will eventually have to know a few things about how figures are stored on the computer. There are many different image file formats, and each has its own set of benefits and disadvantages. Choosing the right file format and the right workflow can alleviate many figure-preparation headaches. 

My own preference is to use pdf for high-quality publication-ready files and generally whenever possible, png for online documents and other scenarios where bitmap graphics are required, and jpeg as the final resort if the png files are too large. In the following, I explain the key differences between these file formats and their respective benefits and drawbacks.

## Bitmap and vector graphics

The most important difference between the various graphics formats is whether they are bitmap or vector (Table \@ref(tab:file-formats)). Bitmaps or raster graphics store the image as a grid of individual points (called pixels), each with a specified color. By contrast, vector graphics store the geometric arrangement of individual graphical elements in the image. Thus, a vector image contains information such as "there's a black line from the top left corner to the bottom right corner, and a red line from the bottom left corner to the top right corner," and the actual image is recreated on the fly as it is displayed on screen or printed.

Table: (\#tab:file-formats) Commonly used image file formats

Acronym   Name                              Type         Application
--------- --------------------------------- ------------ ------------------------
pdf       Portable Document Format          vector       general purpose
eps       Encapsulated PostScript           vector       general purpose, outdated; use pdf
svg       Scalable Vector Graphics          vector       online use
png       Portable Network Graphics         bitmap       optimized for line drawings
jpeg      Joint Photographic Experts Group  bitmap       optimized for photographic images
tiff      Tagged Image File Format          bitmap       print production, accurate color reproduction
raw       Raw Image File                    bitmap       digital photography, needs post-processing
gif       Graphics Interchange Format       bitmap       outdated for static figures, Ok for animations

Vector graphics are also called "resolution-independent," because they can be magnified to arbitrary size without losing detail or sharpness. See Figure \@ref(fig:bitmap-zoom) for a demonstration.

(ref:bitmap-zoom) Illustration of the key difference between vector graphics and bitmaps. (a) Original image. The black square indicates the area we are magnifying in parts (b) and (c). (b) Increasing magnification of the highlighted area from part (a) when the image has been stored as a bitmap graphic. We can see how the image becomes increasingly pixelated as we zoom in further. (c) Increasing magnification of a vector representation of the image. The image maintains perfect sharpness at arbitrary magnification levels.

```{r bitmap-zoom, fig.width = 5.5*6/4.2, fig.asp = 1/2.8, fig.cap='(ref:bitmap-zoom)'}
library(magick)

# support functions ---------------------------------------------------------

# cut square out of larger image
# x and y are center coordinates
cut_npc_square <- function(img, x, y, w) {
  info <- magick::image_info(img)
  width <- w*info$width
  
  xpx <- x*info$width - 0.5*width
  ypx <- (1-y)*info$height-0.5*width # reverse coordinate system
    
  image_crop(img, geometry_area(width = width, height = width, x_off = xpx, y_off = ypx))
}

# add an image to a plot
# x and y are center coordinates, height is calculated from image aspect ratio
add_image <- function(img, x, y, width = NULL, interpolate = TRUE) {
  info <- magick::image_info(img)
  if (is.null(width)) {
    width <- info$width
  }
  height <- width*info$height/info$width
  x <- x - 0.5*width
  y <- y - 0.5*height
  draw_image(img, x, y, width, height, interpolate = interpolate)
}

# adds a box corresponding to the size of an image
# x and y are center coordinates
add_image_box <- function(img, x, y, width = NULL) {
  info <- image_info(img)
  if (is.null(width)) {
    width <- info$width
  }
  height <- width*info$height/info$width 
  geom_rect(
    data = data.frame(
      xmin = x - 0.5*width, xmax = x + 0.5*width,
      ymin = y - 0.5*height, ymax = y + 0.5*height
    ),
    aes(xmin = xmin, xmax = xmax, ymin = ymin, ymax = ymax),
    fill = NA,
    color = "black"
  )
}

# image drawing code ---------------------------------------------------------

img_hires <- image_read("figures/lincoln-temps.png")
xc <- 0.73 # relative x center
yc <- 0.58 # relative y center
bw <- 0.16  # relative box width

img <- image_scale(img_hires, "1800")
c5 <- cut_npc_square(img, xc, yc, bw)
img <- image_scale(img_hires, "3600")
c6 <- cut_npc_square(img, xc, yc, bw/2)
img <- image_scale(img_hires, "7200")
c7 <- cut_npc_square(img, xc, yc, bw/4)
c8 <- cut_npc_square(img_hires, xc, yc, bw/8)

img <- image_scale(img_hires, "1200")
c1 <- cut_npc_square(img, xc, yc, bw)
c2 <- cut_npc_square(img, xc, yc, bw/2)
c3 <- cut_npc_square(img, xc, yc, bw/4)
c4 <- cut_npc_square(img, xc, yc, bw/8)

info <- image_info(img)
p1 <- ggplot() + 
  expand_limits(
    x = c(0, info$width-1),
    y = c(-0.3*info$height, info$height-1) # add 30% vertical space below
  ) +
  coord_fixed(expand = FALSE) +
  draw_image(img, x = 0, y = 0, width = info$width, height = info$height) +
  add_image_box(c1, xc*info$width, yc*info$height, width = bw*info$width) +
  theme_nothing()

p2 <- ggplot() + 
  expand_limits(x = c(-0.01, 4.61), y = c(-0.01, 1.01)) +
  coord_fixed(expand = FALSE) +
  add_image(c1, x = 0.5, y = 0.5, width = 1, interpolate = FALSE) +
  add_image(c2, x = 1.7, y = 0.5, width = 1, interpolate = FALSE) +
  add_image(c3, x = 2.9, y = 0.5, width = 1, interpolate = FALSE) +
  add_image(c4, x = 4.1, y = 0.5, width = 1, interpolate = FALSE) +
  add_image_box(c1, x = 0.5, y = 0.5, width = 1) +
  add_image_box(c2, x = 1.7, y = 0.5, width = 1) +
  add_image_box(c3, x = 2.9, y = 0.5, width = 1) +
  add_image_box(c4, x = 4.1, y = 0.5, width = 1) +
  theme_nothing()

p3 <- ggplot() + 
  expand_limits(x = c(-0.01, 4.61), y = c(-0.01, 1.01)) +
  coord_fixed(expand = FALSE) +
  add_image(c5, x = 0.5, y = 0.5, width = 1) +
  add_image(c6, x = 1.7, y = 0.5, width = 1) +
  add_image(c7, x = 2.9, y = 0.5, width = 1) +
  add_image(c8, x = 4.1, y = 0.5, width = 1) +
  add_image_box(c5, x = 0.5, y = 0.5, width = 1) +
  add_image_box(c6, x = 1.7, y = 0.5, width = 1) +
  add_image_box(c7, x = 2.9, y = 0.5, width = 1) +
  add_image_box(c8, x = 4.1, y = 0.5, width = 1) +
  theme_nothing()

col <- plot_grid(p2, p3, labels = c("b", "c"), ncol = 1)
plot_grid(p1, col, labels = c("a", ""), nrow = 1, rel_widths = c(1, 1.5))


```

Vector graphics have two downsides that can and often do cause trouble in real-world applications. First, because vector graphics are redrawn on the fly by the graphics program with which they are displayed, it can happen that there are differences in how the same graphic looks in two different programs, or on two different computers. This problem occurs most frequently with text, for example when the required font is not available and the rendering software substitutes a different font. Font substitutions will typically allow the viewer to read the text as intended, but the resulting image rarely looks good. There are ways to avoid these problems, such as outlining or embedding all fonts in a pdf file, but they may require special software and/or special technical knowledge to achieve. By contrast, bitmap images will always look the same.

Second, for very large and/or complex figures, vector graphics can grow to enormous file sizes and be slow to render. For example, a scatter plot of millions of data points will contain the x and y coordinates of every individual point, and each point needs to be drawn when the image is rendered, even if points overlap and/or are hidden by other graphical elements. As a consequence, the file may be many megabytes in size, and it may take the rendering software some time to display the figure. When I was a postdoc in the early 2000s, I once created a pdf file that at the time took almost an hour to display in the Acrobat reader. While modern computers are much faster and rendering times of many minutes are all but unheard of these days, even a rendering time of a few seconds can be disruptive if you want to embed your figure into a larger document and your pdf reader grinds to a halt every time you display the page with that one offending figure. Of course, on the flip side, simple figures with only a small number of elements (a few data points and some text, say) will often be much smaller as vector graphics than as bitmaps, and the viewing software may even render such figures faster than it would the corresponding bitmap images.

## Lossless and lossy compression of bitmap graphics

Most bitmap file formats employ some form of data compression to keep file sizes manageable. There are two fundamental types of compression: lossless and lossy. Lossless compression guarantees that the compressed image is pixel-for-pixel identical to the original image, whereas lossy compression accepts some image degradation in return for smaller file sizes.

To understand when using either lossless or lossy compression is appropriate, it is helpful to have a basic understanding of how these different compression algorithms work. Let's first consider lossless compression. Imagine an image with a black background, where large areas of the image are solid black and thus many black pixels appear right next to each other. Each black pixel can be represented by three zeroes in a row: 0 0 0, representing zero intensities in the red, green, and blue color channels of the image. The areas of black background in the image correspond to thousands of zeros in the image file. Now assume somewhere in the image are 1000 consecutive black pixels, corresponding to 3000 zeros. Instead of writing out all these zeros, we could store simply the total number of zeros we need, e.g. by writing 3000 0. In this way, we have conveyed the exact same information with only two numbers, the count (here, 3000) and the value (here, 0). Over the years, many clever tricks along these lines have been developed, and modern lossless image formats (such as png) can store bitmap data with impressive efficiency. However, all lossless compression algorithms perform best when images have large areas of uniform color, and therefore Table \@ref(tab:file-formats) lists png as optimized for line drawings.

Photographic images rarely have multiple pixels of identical color and brightness right next to each other. Instead they have gradients and other somewhat regular patterns on many different scales. Therefore, lossless compression of these images often doesn't work very well, and lossy compression has been developed as an alternative. The key idea of lossy compression is that some details in an image are too subtle for the human eye, and those can be discarded without obvious degradation in the image quality. For example, consider a gradient of 1000 pixels, each with a slightly different color value. Chances are the gradient will look nearly the same if it is drawn with only 200 different colors and coloring every five adjacent pixels in the exact same color.

The most widely used lossy image format is jpeg (Table \@ref(tab:file-formats)), and indeed many digital cameras output images as jpeg by default. Jpeg compression works exceptionally well for photographic images, and huge reductions in file size can often be obtained with very little degradation in image quality. However, jpeg compression fails when images contain sharp edges, such as created by line drawings or by text. In those cases, jpeg compression can result in very noticeable artifacts (Figure \@ref(fig:jpeg-example)). 


(ref:jpeg-example) Illustration of jpeg artifacts. (a) The same image is reproduced multiple times using increasingly severe jpeg compression. The resulting file size is shown in red text above each image. A reduction in file size by a factor of 10, from 432KB in the original image to 43KB in the compressed image, results in only minor perceptible reduction in image quality. However, a further reduction in file size by a factor of 2, to a mere 25KB, leads to numerous visible artifacts. (b) Zooming in to the most highly compressed image reveals the various compression artifacts. Photo credit: Claus O. Wilke 

```{r jpeg-example, fig.cap='(ref:jpeg-example)'}
knitr::include_graphics("figures/jpeg_example_combined.png", dpi=210)
```

Even if jpeg artifacts are sufficiently subtle that they are not immediately visible to the naked eye they can cause trouble, for example in print production. Therefore, it is a good idea to avoid the jpeg format whenever possible. In particular, you should avoid it for images containing line drawings or text, as is the case for data visualizations or screen shots. The appropriate format for those images is png or tiff. I use the jpeg format exclusively for photographic images. And if an image contains both photographic elements and line drawings or text, you should still use png or tiff. The worst case scenario with those file formats is that your image files grow large, whereas the worst case scenario with jpeg is that your final product looks ugly.

## Converting between image formats

It is generally possible to convert any image format into any other image format. For example, on a Mac, you can open an image with Preview and then export to a number of different formats. In this process, though, important information can get lost, and information is never regained. For example, after saving a vector graphic into a bitmap format, e.g. a pdf file as a jpeg, the resolution independence that is a key feature of the vector graphic has been lost. Conversely, saving a jpeg image into a pdf file does not magically turn the image into a vector graphic. The image will still be a bitmap image, just stored inside the pdf file. Similarly, converting a jpeg file into a png file does not remove any artifacts that may have been introduced by the jpeg compression algorithm.

It is therefore a good rule of thumb to always store the original image in the format that maintains maximum resolution, accuracy, and flexibility. Thus, for data visualizations, either create your figure as pdf and then convert into png or jpg when necessary, or alternatively store as high-resolution png. Similarly, for images that are only available as bitmaps, such as digital photographs, store them in a format that doesn't use lossy compression, or if that can't be done, compress as little as possible. Also, store the image in as high a resolution as possible, and downscale when needed.  


<!--chapter:end:image_file_formats.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")
```


# Introduction {#introduction}

Data visualization is part art and part science. The challenge is to get the art right without getting the science wrong and vice versa. A data visualization first and foremost has to accurately convey the data. It must not mislead or distort. If one number is twice as large as another, but in the visualization they look to be about the same, then the visualization is wrong. At the same time, a data visualization should be aesthetically pleasing. Good visual presentations tend to enhance the message of the visualization. If a figure contains jarring colors, imbalanced visual elements, or other features that distract, then the viewer will find it harder to inspect the figure and interpret it correctly.

In my experience, scientists frequently (though not always!) know how to visualize data without being grossly misleading. However, they may not have a well developed sense of visual aesthetics, and they may inadvertently make visual choices that detract from their desired message. Designers, on the other hand, may prepare visualizations that look beautiful but play fast and loose with the data. It is my goal to provide useful information to both groups. 

The book attempts to cover the key principles, methods, and concepts required to visualize data for publications, reports, or presentations. Because data visualization is a vast field, and in its broadest definition could include topics as varied as schematic technical drawings, 3D animations, and user interfaces, I necessarily had to limit my scope for this book. I am specifically covering the case of static visualizations presented in print, online, or as slides. The book does not cover interactive visuals or movies, except in one brief section in the chapter on visualizing uncertainty. Therefore, throughout this book, I will use the words "visualization" and "figure" somewhat interchangeably. The book also does not provide any instruction on *how* to make figures with existing visualization softwares or programming libraries. The annotated bibliography at the end of the book includes pointers to appropriate texts covering these topics.

The book is divided into three parts. The first, "From data to visualization," describes different types of plots and charts, such as bar graphs, scatter plots, or pie charts. Its primary emphasis is the science of visualization. In this part, rather than attempting to provide encyclopedic coverage of every conceivable visualization approach, I discuss a core set of visuals that you will likely encounter in publications and/or need in your own work. In organizing this part, I have attempted to group visualizations by the type of message they convey rather than by the type of data being visualized. Statistical texts often describe data analysis and visualization by type of data, organizing the material by number and type of variables (one continuous variable, one discrete variable, two continuous variables, one continuous and one discrete variable, etc.). I believe that only statisticians find this organization helpful. Most other people think in terms of a message, such as how large something is, how it is composed of parts, how it relates to something else, and so on.

The second part, "Principles of figure design," discusses various design issues that arise when assembling data visualizations. Its primary but not exclusive emphasis is the aesthetic aspect of data visualization. Once we have chosen the correct type of plot or chart that is appropriate for our dataset, we have to make aesthetic choices about the visual elements, such as colors, symbols, and font sizes. These choices can affect both how clear a visualization is and how elegant it looks. The chapters in this second part address the most common issues that I have seen arise repeatedly in practical applications.

The third part, "Miscellaneous topics," covers a few remaining issues that didn't fit into the first two parts. It discusses file formats commonly used to store images and plots, it provides thoughts about the choice of visualization software, and it explains how to place individual figures into the context of a larger document.

## Ugly, bad, and wrong figures {-}

Throughout this book, I frequently show different versions of the same figures, some as examples of how to make a good visualization and some as examples of how not to. To provide a simple visual guideline of which examples should be emulated and which should be avoided, I am clearly labeling problematic figures as "ugly", "bad", or "wrong" (Figure \@ref(fig:ugly-bad-wrong-examples)):

- **ugly**---A figure that has aesthetic problems but otherwise is clear and informative.
- **bad**---A figure that has problems related to perception; it may be unclear, confusing, overly complicated, or deceiving.
- **wrong**---A figure that has problems related to mathematics; it is objectively incorrect.


(ref:ugly-bad-wrong-examples) Examples of ugly, bad, and wrong figures. (a) A bar plot showing three values (A = 3, B = 5, and C = 4). This is a reasonable visualization with no major flaws. (b) An ugly version of part (a). While the plot is technically correct, it is not aesthetically pleasing. The colors are too bright and not useful. The background grid is too prominent. The text is displayed using three different fonts in three different sizes. (c) A bad version of part (a). Each bar is shown with its own *y*-axis scale. Because the scales don't align, this makes the figure misleading. One can easily get the impression that the three values are closer together than they actually are. (d) A wrong version of part (a). Without an explicit *y* axis scale, the numbers represented by the bars cannot be ascertained. The bars appear to be of lengths 1, 3, and 2, even though the values displayed are meant to be 3, 5, and 4.


```{r ugly-bad-wrong-examples, fig.asp = 3/4, fig.cap = '(ref:ugly-bad-wrong-examples)'}
df <- tibble(
  type = c("A", "B", "C"),
  value = c(3, 5, 4),
  expand = c(4, 5, 4.5)
)


p1 <- ggplot(df, aes(type, value)) + 
  geom_col(fill = "#56B4E9", width = 0.65, alpha = 0.9) +
  scale_y_continuous(limits = c(0, 5.2), expand = c(0, 0)) +
  scale_x_discrete(name = NULL) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line = element_blank(),
    plot.margin = margin(18, 12, 0, 0)
  )

p2 <- ggplot(df, aes(type, value)) + 
  geom_col(fill = c("#CCFF00FF", "#00FFFFFF", "#CC00FFFF"), width = 0.75) +
  scale_y_continuous(limits = c(0, 5.2), expand = c(0, 0)) +
  scale_x_discrete(name = NULL) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line = element_blank(),
    axis.title = element_text(family = "Comic Sans MS", size = 15),
    axis.text.x = element_text(family = "Times", size = 10),
    axis.text.y = element_text(family = "Arial", size = 13),
    panel.grid = element_line(color = "black"),
    axis.ticks = element_line(color = "black"),
    plot.margin = margin(18, 12, 1.5, 1.5)
  )


p3 <- ggplot(df, aes(type, value)) + 
  geom_col(fill = "#56B4E9", width = 0.65, alpha = 0.9) +
  geom_point(aes(y = expand), shape = NA) +
  scale_y_continuous(expand = c(0, 0)) +
  scale_x_discrete(name = NULL) +
  facet_wrap(~type, scales = "free") +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line = element_blank(),
    strip.text = element_blank(),
    plot.margin = margin(18, 12, 1.5, 1.5)
  )

p3a <- ggplot(df, aes(type, value)) + 
  geom_col(color = "black", fill = NA, width = .5) +
  scale_y_continuous(limits = c(0, 5.2), expand = c(0, 0)) +
  scale_x_discrete(name = NULL) +
  coord_cartesian(clip = "off") +
  theme_dviz_grid(12) +
  background_grid(
    major = "y", minor = "none",
    colour.major = "grey30", colour.minor = "black",
    size.major = 0.5,
    size.minor = 0.2
  ) +
  theme(
    axis.ticks = element_line(color = "grey30"),
    plot.margin = margin(18, 12, 1.5, 1.5)
  )

p4 <- ggplot(df, aes(type, value)) + 
  geom_col(fill = "#56B4E9", width = 0.65, alpha = 0.9) +
  coord_cartesian(xlim = c(0.4, 3.6), ylim = c(2, 6.2), expand = FALSE, clip = "on") +
  scale_y_continuous(breaks = 2:4, name = "", labels = c("", "", "")) +
  scale_x_discrete(name = NULL) +
  theme_dviz_hgrid(12) +
  theme(
    panel.grid.major.y = element_blank(),
    axis.ticks.y = element_blank(),
    plot.margin = margin(18, 12, 1.5, 1.5)
  )

plot_grid(
  p1, NULL, stamp_ugly(p2),
  NULL, NULL, NULL,
  stamp_bad(p3), NULL, stamp_wrong(p4),
  rel_widths = c(1, .1, 1),
  rel_heights = c(1, .15, 1),
  labels = c("a", "", "b", "", "", "", "c", "", "d")
)
```

I am not explicitly labeling good figures. Any figure that isn't clearly labeled as flawed  should be assumed to be at least acceptable. It is a figure that is informative, looks appealing, and could be printed as is. Note that among the good figures, there will still be differences in quality, and some good figures will be better than others.

I generally provide my rationale for specific ratings, but some are a matter of taste. In general, the "ugly" rating is more subjective than the "bad" or "wrong" rating. Moreover, the boundary between "ugly" and "bad" is somewhat fluid. Sometimes poor design choices can interfere with human perception to the point where a "bad" rating is more appropriate than an "ugly" rating. In any case, I encourage you to develop your own eye and to critically evaluate my choices.


<!--chapter:end:introduction.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")

library(forcats)
library(stringr)
library(ggridges)
```

# Multi-panel figures {#multi-panel-figures}

When datasets become large and complex, they often contain much more information than can reasonably be shown in a single figure panel. To visualize such datasets, it can be helpful to create multi-panel figures. These are figures that consist of multiple figure panels where each one shows some subset of the data. There are two distinct categories of such figures: 1. Small multiples are plots consisting of multiple panels arranged in a regular grid. Each panel shows a different subset of the data but all panels use the same type of visualization. 2. Compound figures consist of separate figure panels assembled in an arbitrary arrangement (which may or may not be grid based) and showing entirely different visualizations, or possibly even different datasets. 

We have encountered both types of multi-panel figures in many places throughout this book. In general, these figures are intuitive and straightforward to interpret. However, when preparing such figures, there are a few issues we need to pay attention to, such as appropriate axis scaling, alignment, and consistency between separate panels.

## Small multiples

The term "small multiple" was popularized by @TufteEnvisioning. An alternative term, "trellis plot", was popularized around the same time by Cleveland, Becker, and colleagues at Bell Labs [@Cleveland1993; @Becker-Cleveland-Shyu-1996]. Regardless of terminology, the key idea is to slice the data into parts according to one or more data dimensions, visualize each data slice separately, and then arrange the individual visualizations into a grid. Columns, rows, or individual panels in the grid are labeled by the values of the data dimensions that define the data slices. More recently, this technique is also sometimes referred to as "faceting", named after the methods that create such plots in the widely used ggplot2 plot library (e.g., `facet_grid()`, see @Wickham2016).

As a first example, we will apply this technique to the dataset of Titanic passengers. We can subdivide this dataset by the class in which each passenger travelled and by whether a passenger survived or not. Within each of these six slices of data, there are both male and female passengers, and we can visualize their numbers using bars. The result is six bar plots, which we arrange in two columns (one for passengers who died and one for those who survived) of three rows (one for each class) (Figure \@ref(fig:titanic-passenger-breakdown)). The columns and rows are labeled, so it is immediately clear which of the six plots corresponds to which combination of survival status and class.

(ref:titanic-passenger-breakdown) Breakdown of passengers on the Titanic by gender, survival, and class in which they traveled (1st, 2nd, or 3rd).

```{r titanic-passenger-breakdown, fig.width = 5, fig.asp = 3/4, fig.cap = '(ref:titanic-passenger-breakdown)'}
titanic %>% mutate(surv = ifelse(survived == 0, "died", "survived")) %>%
  ggplot(aes(sex, fill = sex)) + geom_bar() +
    facet_grid(class ~ surv, scales = "free_x") +
    scale_x_discrete(name = NULL) + 
    scale_y_continuous(limits = c(0, 195), expand = c(0, 0)) +
    scale_fill_manual(values = c("#D55E00D0", "#0072B2D0"), guide = "none") +
    theme_dviz_hgrid(rel_small = 1) +
    theme(
      axis.line = element_blank(),
      axis.ticks.length = grid::unit(0, "pt"),
      axis.ticks = element_blank(),
      axis.text.x = element_text(margin = margin(7, 0, 0, 0)),
      strip.text = element_text(margin = margin(3.5, 3.5, 3.5, 3.5)),
      strip.background  = element_rect(
        fill = "grey85", colour = "grey85",
        linetype = 1, size = 0.25
      ),
      panel.border = element_rect(
        colour = "grey85", fill = NA, linetype = 1,
        size = 1.)
      )
```

This visualization provides an intuitive and highly interpretable visualization of the fate of the Titanic passengers. We see clearly that most men died and most women survived. Further, and among the women who died nearly all were traveling in 3rd class.

Small multiples are a powerful tool to visualize very large amounts of data at once. Figure \@ref(fig:titanic-passenger-breakdown) uses six separate panels, but we can use many more. Figure \@ref(fig:movie-rankings) shows the relationship between the average ranking of a movie on the Internet Movie Database (IMDB) and the number of votes the movie has received, separately for movies released over a 100 year time period. Here, the dataset is sliced by only one dimension, the year, and panels for each year are arranged in rows from top left to bottom right. This visualization shows that there is an overall relationship between average ranking and number of votes, such that movies with more votes tend to have higher rankings. However, the strength of this trend varies with year, and for movies released in the early 2000s there is no relationship or even a negative one.

(ref:movie-rankings) Average movie rankings versus number of votes, for movies from 1906 to 2005. Dots represent individual movies, and lines represent the linear regression of the average ranking of each movie versus the logarithm of the number of votes the movie has received. In most years, movies with a higher number of votes have, on average, a higher average ranking. However, this trend has weakened towards the end of the 20th century, and a negative relationship can be seen for movies released in the early 2000s. Data Source: Internet Movie Database (IMDB, http://imdb.com/)

```{r movie-rankings, fig.width = 5.5*6/4.2, fig.asp = 1, fig.cap = '(ref:movie-rankings)'}
library(ggplot2movies)

ggplot(filter(movies, year > 1905), aes(y = rating, x = votes)) + 
  geom_point(color = "#0072B250", size = 0.1) + 
  geom_smooth(
    method = 'lm', se = FALSE, size = 1.25, color = '#D55E00',
    fullrange = TRUE
  ) + 
  scale_x_log10(labels = label_log10, name = "number of votes", breaks = c(10, 1000, 100000)) + 
  scale_y_continuous(
    limits = c(0, 10), expand = c(0, 0),
    breaks = c(0, 5, 10), name = "average rating"
  ) + 
  facet_wrap(~year, ncol = 10) +
  theme_dviz_grid(10, rel_small = 1, line_size = 0.25) +
  theme(
    axis.title = element_text(size = 14),
    axis.ticks = element_blank(),
    axis.ticks.length = unit(0, "pt"),
    strip.text = element_text(margin = margin(3.5, 3.5, 3.5, 3.5)),
    panel.border = element_rect(
      colour = "grey80", fill = NA, linetype = 1, size = 1.
    ),
    plot.margin = margin(3, 5, 3, 1.5)
  )
```

For such large plots to be easily understandable, it is important that each panel uses the same axis ranges and scalings. The human mind expects this to be the case. When it is not, there is a good chance that a reader will mis-interpret what the figure shows. For example, consider Figure \@ref(fig:BA-degrees-variable-y-lims), which presents how the proportion of Bachelor's degrees in different degree areas has changed over time. The figure shows the nine degree areas that have represented, on average, more than 4% of all degrees between 1971 to 2015. The *y* axis of panel is scaled such that the curve for each degree field covers the entire *y*-axis range. As a consequence, a cursory examination of Figure \@ref(fig:BA-degrees-variable-y-lims) suggests that the nine degree areas are all equally popular and have all experienced variation in popularity of a similar magnitude.

(ref:BA-degrees-variable-y-lims) Trends in Bachelor's degrees conferred by U.S. institutions of higher learning. Shown are all degree areas that represent, on average, more than 4% of all degrees. This figure is labeled as "bad" because all panels use different *y*-axis ranges. This choice obscures the relative sizes of the different degree areas and it over-exaggerates the changes that have happened in some of the degree areas. Data Source: National Center for Education Statistics

```{r BA-degrees-variable-y-lims, fig.width = 5.5*6/4.2, fig.asp = 0.8, fig.cap = '(ref:BA-degrees-variable-y-lims)'}
BA_degrees %>% 
  mutate(field = ifelse(field == "Communication, journalism, and related programs",
                        "Communication, journalism, and related", field)) -> BA_df

BA_df %>% group_by(field) %>%
  summarize(mean_perc = mean(perc)) %>%
  arrange(desc(mean_perc)) -> BA_top

top_fields <- filter(BA_top, mean_perc>0.04)$field

BA_top_degrees <- filter(BA_df, field %in% top_fields) %>%
  mutate(field = factor(field, levels = top_fields)) %>%
  arrange(field)

p <- ggplot(BA_top_degrees, aes(year, perc)) + 
  geom_line(color = "#0072B2") + 
  facet_wrap(~field, labeller = label_wrap_gen(width = 25), ncol = 3,
             scales = "free") +
  ylab("proportion of degrees") +
  scale_y_continuous(labels = scales::percent_format(accuracy = .1)) +
  theme_dviz_hgrid() +
  theme(strip.text = element_text(margin = margin(7, 7, 3, 7)),
        panel.spacing.x = grid::unit(14, "pt"),
        plot.margin = margin(3.5, 14, 3.5, 1.5)) 

stamp_bad(p)
```

Placing all panels onto the same *y* axis reveals, however, that this interpretation is highly misleading (Figure \@ref(fig:BA-degrees-fixed-y-lims)). Some degree areas are much more popular than others, and similarly some areas have grown or shrunk much more than others. For example, education has declined a lot, whereas visual and performing arts have remained approximately constant or maybe seen a small increase.

(ref:BA-degrees-fixed-y-lims) Trends in Bachelor's degrees conferred by U.S. institutions of higher learning. Shown are all degree areas that represent, on average, more than 4% of all degrees. Data Source: National Center for Education Statistics

```{r BA-degrees-fixed-y-lims, fig.width = 5.5*6/4.2, fig.asp = 0.8, fig.cap = '(ref:BA-degrees-fixed-y-lims)'}
ggplot(BA_top_degrees, aes(year, perc)) + 
  geom_line(color = "#0072B2") + 
  facet_wrap(~field, labeller = label_wrap_gen(width = 25), ncol = 3,
             scales = "free") +
  scale_y_continuous(
    limits = c(0, 0.241), expand = c(0, 0),
    name = "proportion of degrees",
    labels = scales::percent_format(accuracy = 1)
  ) +
  theme_dviz_hgrid() +
  theme(strip.text = element_text(margin = margin(7, 7, 3, 7)),
        panel.spacing.x = grid::unit(14, "pt"),
        plot.margin = margin(3.5, 1.5, 3.5, 1.5)) 

```

I generally recommend against using different axis scalings in separate panels of a small multiples plot. However, on occasion, this problem truly cannot be avoided. If you encounter such a scenario, then I think at a minimum you need to draw the reader's attention to this issue in the figure caption. For example, you could add a sentence such as: "Notice that the *y*-axis scalings differ among the different panels of this figure."

It is also important to think about the ordering of the individual panels in a small multiples plot. The plot will be easier to interpret if the ordering follows some logical principle. In Figure \@ref(fig:titanic-passenger-breakdown), I arranged the rows from the highest class (1st class) to the lowest class (3rd class). In Figure \@ref(fig:movie-rankings), I arranged the panels by increasing years from the top left to the bottom right. In Figure \@ref(fig:BA-degrees-fixed-y-lims), I arranged the panels by decreasing average degree popularity, such that the most popular degrees are in the top row and/or to the left and the least popular degrees are in the bottom row and/or to the right.

```{block type='rmdtip', echo=TRUE}
Always arrange the panels in a small multiples plot in a meaningful and logical order. 
```

## Compound figures {#compound-figures}

Not every figure with multiple panels fits the pattern of small multiples. Sometimes we simply want to combine several independent panels into a combined figure that conveys one overarching point. In this case, we can take the individual plots and arrange them in rows, columns, or other, more complex arrangements, and call the entire arrangement one figure. For an example, see Figure \@ref(fig:BA-degrees-compound), which continues the analysis of trends in Bachelor's degrees conferred by U.S. institutions of higher learning. Panel (a) of Figure \@ref(fig:BA-degrees-compound) shows the growth in total number of degrees awarded from 1971 to 2015, a time span during which the number of degrees awarded approximately doubled. Panel (b) instead shows the change in the percent of degrees awarded over the same time period in the five most popular degree areas. We can see that social sciences, history, and education have experienced massive declines from 1971 to 2015, whereas business and health professions have seen substantial growth.

Notice how unlike in my small multiples examples, the individual panels of the compound figure are labeled alphabetically. It is conventional to use lower or upper case letters from the Latin alphabet. The labeling is needed to uniquely specify a particular panel. For example, when I want to talk about the part of Figure \@ref(fig:BA-degrees-compound) showing the changes in percent of degrees awarded, I can refer to panel (b) of that figure or simply to Figure \@ref(fig:BA-degrees-compound)b. Without labeling, I would have to awkwardly talk about the "right panel" or the "left panel" of Figure \@ref(fig:BA-degrees-compound), and referring to specific panels would be even more awkward for more complex panel arrangements. Labeling is not needed and not normally done for small multiples because there each panel is uniquely specified by the faceting variable(s) that are provided as figure labels.

(ref:BA-degrees-compound) Trends in Bachelor's Degrees conferred by U.S. institutions of higher learning. (a) From 1970 to 2015, the total number of degrees nearly doubled. (b) Among the most popular degree areas, social sciences, history, and education experienced a major decline, while business and health professions grew. Data Source: National Center for Education Statistics

```{r BA-degrees-compound, fig.width =  5.5*6/4.2, fig.asp = 0.4, fig.cap = '(ref:BA-degrees-compound)'}
BA_degrees %>% 
  mutate(field = ifelse(field == "Communication, journalism, and related programs",
                        "Communication, journalism, and related", field)) -> BA_df

BA_df %>% group_by(year) %>%
  summarize(total = sum(count)) -> BA_totals

textcol <- "gray30"

p1 <- ggplot(BA_totals, aes(year, total/1e6)) + 
  geom_density_line(stat = "identity", color = "#0072B2",
                    fill = desaturate(lighten("#0072B280", .3), .4)) + 
  scale_y_continuous(limits = c(0, 2.05), expand = c(0, 0),
                     name = "degrees awarded (millions)") +
  scale_x_continuous(limits = c(1970, 2016), expand = c(0, 0), name = NULL) +
  theme_dviz_hgrid() +
  theme(axis.title = element_text(color = textcol),
        axis.text = element_text(color = textcol),
        plot.margin = margin(3, 7, 3, 1.5))

BA_df %>% group_by(field) %>%
  summarize(mean_perc = mean(perc)) %>%
  arrange(desc(mean_perc)) -> BA_top

top_fields <- filter(BA_top, mean_perc>0.055)$field

BA_top_pairs <- filter(BA_df, field %in% top_fields,
                       year %in% c(1971, 2015)) %>%
  mutate(field_wrapped = str_wrap(field, 25))

p2 <- ggplot(BA_top_pairs, aes(x = year, y = perc)) +
  geom_line(aes(group = field), color = "gray60") +
  geom_point(fill = "#0072B2", color = "white", shape = 21, size = 3, stroke = 1.5) +
  scale_x_continuous(limits = c(1971, 2015), breaks = c(1971, 2015),
                     labels = c("1970-71", "2014-15"),
                     expand = expand_scale(mult = c(0.1, 0.04)),
                     name = NULL,
                     position = "top") +
  scale_y_continuous(
    limits = c(0.02, 0.22), expand = c(0, 0),
    name = "proportion of degrees",
    labels = scales::percent_format(accuracy = 1),
    sec.axis = dup_axis(
      breaks = filter(BA_top_pairs, year == 2015)$perc + 0.0001,
      labels = filter(BA_top_pairs, year == 2015)$field_wrapped,
      name = NULL)
  ) +
  theme_dviz_open() +
  theme(axis.line.x = element_blank(),
        axis.ticks.x = element_blank(),
        axis.title.y = element_text(color = textcol),
        axis.text.y = element_text(color = textcol),
        axis.line.y.left = element_line(color = textcol),
        axis.text.y.right = element_text(hjust = 0, vjust = .5,
                                         margin = margin(0, 0, 0, 0),
                                         color = "black",
                                         lineheight = 0.8
                                         ),
        axis.line.y.right = element_blank(),
        axis.ticks.y.right = element_blank(),
        plot.margin = margin(3, 7, 3, 1.5))

plot_grid(p1, p2, labels = "auto", rel_widths = c(1.2, 1), align = 'h')
```

When labeling the different panels of a compound figure, pay attention to how the labels fit into the overall figure design. I often see figures where the labels look like they were slapped onto the figure after the fact by a different person. It's not uncommon to see labels made overly large and prominent, placed in an awkward location, or typeset in a different font than the rest of the figure. (See Figure \@ref(fig:BA-degrees-compound-bad) for an example.) The labels should not be the first thing you see when you look at a compound figure. In fact, they don't need to stand out at all. We generally know which figure panel has which label, since the  convention is to start in the top-left corner with "a" and label consecutively left to right and top to bottom. I think of these labels as equivalent to page numbers. You don't normally read the page numbers, and there is no surprise in which page has which number, but on occasion it can be helpful to use page numbers to refer to a particular place in a book or article.

(ref:BA-degrees-compound-bad) Variation of Figure \@ref(fig:BA-degrees-compound) with poor labeling. The labels are too large and thick, they are in the wrong font, and they are placed in an awkward location. Also, while labeling with capital letters is fine and is in fact quite common, labeling needs to be consistent across all figures in a document. In this book, the convention is that multi-panel figures use lower lower-case labels, and thus this figure is inconsistent with the other figures in this book.

```{r BA-degrees-compound-bad, fig.width =  5.5*6/4.2, fig.asp = 0.4, fig.cap = '(ref:BA-degrees-compound-bad)'}
stamp_ugly(plot_grid(p1, p2, labels = "AUTO", rel_widths = c(1.2, 1), align = 'h',
                     label_fontfamily = "Palatino", label_fontface = "bold",
                     label_x = 0.8,
                     label_y = 0.2,
                     label_size = 23))
```


We also need to pay attention to how the individual panels of a compound figure fit together. It is possible to make a set of figure panels that individually are fine but jointly don't work. In particular, we need to employ a consistent visual language. By "visual language," I mean the colors, symbols, fonts, and so on that we use to display the data. And keeping the language consistent means, in a nutshell, that the same things look the same or at least substantively similar across figures.

Let's look at an example that violates this principle. Figure \@ref(fig:athletes-composite-inconsistent) is a three-panel figure visualizing a dataset about the physiology and body-composition of male and female athletes. Panel (a) shows the number of men and women in the dataset, panel (b) shows the counts of red and white blood cells for men and women, and panel (c) shows the body fat percentage of men and women, broken down by sport. Each panel individually is an acceptable figure. However, in combination the three panels do not work, because they don't share a common visual language. First, panel (a) uses the same blue color for both male and female athletes, panel (b) uses it only for male athletes, and panel (c) uses it for female athletes. Moreover, panels (b) and (c) introduce additional colors, but these colors differ between the two panels. It would have been better to use the same two colors consistently for male and female athletes, and to apply the same coloring scheme to panel (a) as well. Second, in panels (a) and (b) women are on the left and men on the right, but in panel (c) the order is reversed. The order of the boxplots in panel (c) should be switched so it matches panels (a) and (b).

(ref:athletes-composite-inconsistent) Physiology and body-composition of male and female athletes. (a) The data set encompasses 73 female and 85 male professional athletes. (b) Male athletes tend to have higher red blood cell (RBC, reported in units of $10^{12}$ per liter) counts than female athletes, but there are no such differences for white blood cell counts (WBC, reported in units of $10^{9}$ per liter). (c) Male athletes tend to have a lower body fat percentage than female athletes performing in the same sport. Data source: @Telford-Cunningham-1991

```{r athletes-composite-inconsistent, fig.width = 5*6/4.2, fig.asp = 0.75, fig.cap = '(ref:athletes-composite-inconsistent)'}
male_sport <- unique(filter(Aus_athletes, sex=="m")$sport)
female_sport <- unique(filter(Aus_athletes, sex=="f")$sport)
both_sport <- male_sport[male_sport %in% female_sport]
athletes_df <- filter(Aus_athletes, sport %in% both_sport) %>%
  mutate(
    sport = case_when(
      sport == "track (400m)" ~ "track",
      sport == "track (sprint)" ~ "track",
      TRUE ~ sport
    ),
    sex = factor(sex, levels = c("f", "m"))
  )

p1 <- ggplot(athletes_df, aes(x = sex)) + 
  geom_bar(fill = "#56B4E9E0") +
  scale_y_continuous(limits = c(0, 95), expand = c(0, 0), name = "number") +
  scale_x_discrete(name = NULL, labels = c("female", "male")) +
  theme_dviz_hgrid(12, rel_small = 1) + 
  theme(
    axis.ticks.x = element_blank(),
    #axis.ticks.length = grid::unit(0, "pt"),
    plot.margin = margin(3, 0, 0, 0)
  )

p2 <- ggplot(athletes_df, aes(x = rcc, y = wcc, shape = sex, color = sex, fill = sex)) + 
  geom_point(size = 2.5) +
  scale_x_continuous(limits = c(3.8, 6.75), name = NULL) +
  scale_y_continuous(limits = c(2.2, 11.), expand = c(0, 0), name = "WBC count") +
  scale_shape_manual(
    values = c(21, 22),
    labels = c("female   ", "male"), name = NULL,
    guide = guide_legend(direction = "horizontal")
  ) +
  scale_color_manual(
    values = c("#CC79A7", "#56B4E9"), name = NULL,
    labels = c("female   ", "male"),
    guide = guide_legend(direction = "horizontal")
  ) +
  scale_fill_manual(
    values = c("#CC79A780", "#56B4E980"), name = NULL,
    labels = c("female   ", "male"),
    guide = guide_legend(direction = "horizontal")
  ) +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    legend.position = c(1, .1),
    legend.justification = "right",
    legend.box.background = element_rect(fill = "white", color = "white"),
    plot.margin = margin(3, 0, 0, 0)
  )

p_row <- plot_grid(
  p1, NULL, p2, 
  labels = c("a", "", "b"),
  align = 'h',
  nrow = 1,
  rel_widths = c(0.7, 0.02, 1)
) +
  draw_text(
    "RBC count", x = 1, y = 0.01, size = 12, hjust = 1, vjust = 0,
    family = dviz_font_family
  )

p3 <- ggplot(
  athletes_df,
  aes(
    x = sport, y = pcBfat, color = fct_relevel(sex, "m"),
    fill = fct_relevel(sex, "m")
  )
) + 
  geom_boxplot(width = 0.5) +
  scale_color_manual(
    values = c("#009E73", "#56B4E9"), name = NULL,
    labels = c("male", "female")
  ) +
  scale_fill_manual(
    values = c("#009E7340", "#56B4E940"), name = NULL,
    labels = c("male", "female")
  ) +
  scale_x_discrete(name = NULL) +
  scale_y_continuous(name = "% body fat") +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank()
    #axis.ticks.length = grid::unit(0, "pt")
  )

stamp_bad(
  plot_grid(
    p_row, NULL, p3, 
    ncol = 1,
    rel_heights = c(1, .04, 1),
    labels = c("", "", "c")
  ) +
  theme(plot.margin = margin(6, 6, 3, 1.5))
)
```

Figure \@ref(fig:athletes-composite-good) fixes all these issues. In this figure, female athletes are consistently shown in orange and to the left of male athletes, who are shown in blue. Notice how much easier it is to read this figure than Figure \@ref(fig:athletes-composite-inconsistent). When we use a consistent visual language, it doesn't take much mental effort to determine which visual elements in the different panels represent women and which men. Figure \@ref(fig:athletes-composite-inconsistent), on the other hand, can be quite confusing. In particular, on first glance it may generate the impression that men tend to have higher body fat percentages than women. Notice also that we need only a single legend in Figure \@ref(fig:athletes-composite-good) but needed two in Figure \@ref(fig:athletes-composite-inconsistent). Since the visual language is consistent, the same legend works for panels (b) and (c).

(ref:athletes-composite-good) Physiology and body-composition of male and female athletes. This figure shows the exact same data as Figure \@ref(fig:athletes-composite-inconsistent), but now using a consistent visual language. Data for female athletes is always shown to the left of the corresponding data for male athletes, and genders are consistently color-coded throughout all elements of the figure. Data source: @Telford-Cunningham-1991

```{r athletes-composite-good, fig.width = 5*6/4.2, fig.asp = 0.75, fig.cap = '(ref:athletes-composite-good)'}
p1 <- ggplot(athletes_df, aes(x = sex, fill = sex)) + 
  geom_bar() +
  scale_y_continuous(limits = c(0, 95), expand = c(0, 0), name = "number") +
  scale_x_discrete(name = NULL, labels = c("female", "male")) +
  scale_fill_manual(values = c("#D55E00D0", "#0072B2D0"), guide = "none") +
  theme_dviz_hgrid(12, rel_small = 1) + 
  theme(
    #axis.line.x = element_blank(),
    axis.ticks.x = element_blank(),
    #axis.ticks.length = grid::unit(0, "pt"),
    plot.margin = margin(3, 0, 0, 0)
  )

p2 <- ggplot(athletes_df, aes(x = rcc, y = wcc, fill = sex)) + 
  geom_point(pch = 21, color = "white", size = 2.5) +
  scale_x_continuous(limits = c(3.8, 6.75), name = NULL) +
  scale_y_continuous(limits = c(2.2, 11.), expand = c(0, 0), name = "WBC count") +
  scale_fill_manual(values = c("#D55E00D0", "#0072B2D0"), guide = "none") +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(plot.margin = margin(3, 0, 0, 0))

p_row <- plot_grid(
  p1, NULL, p2,
  labels = c("a", "", "b"),
  align = 'h', 
  nrow = 1,
  rel_widths = c(0.7, 0.02, 1)
) +
  draw_text(
    "RBC count", x = 1, y = 0.01, size = 12, hjust = 1, vjust = 0,
    family = dviz_font_family
  )

GeomBP <- GeomBoxplot
GeomBP$draw_key <- draw_key_polygon

p3 <- ggplot(athletes_df, aes(x = sport, y = pcBfat, color = sex, fill = sex)) + 
  stat_boxplot(width = 0.5, geom = GeomBP) +
  scale_color_manual(
    values = c("#D55E00", "#0072B2"), name = NULL,
    labels = c("female   ", "male")) +
  scale_fill_manual(values = c("#D55E0040", "#0072B240"), guide = "none") +
  scale_x_discrete(name = NULL) +
  scale_y_continuous(name = "% body fat") +
  guides(color = guide_legend(
    override.aes = list(
      fill = c("#D55E00D0", "#0072B2D0"),
      color = "white", size = 2
    ),
    direction = "horizontal")
  ) +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank(),
    #axis.ticks.length = grid::unit(0, "pt"),
    legend.position = c(1., .96),
    legend.justification = "right"
  )

plot_grid(
  p_row, NULL, p3,
  ncol = 1,
  rel_heights = c(1, .04, 1),
  labels = c("", "", "c")
) +
  theme(plot.margin = margin(6, 6, 3, 1.5))
```

Finally, we need to pay attention to the alignment of individual figure panels in a compound figure. The axes and other graphical elements of the individual panels should all be aligned to each other. Getting the alignment right can be quite tricky, in particular if individual panels are prepared separately, possibly by different people and/or in different programs, and then pasted together in an image manipulation program. To draw your attention to such alignment issues, Figure \@ref(fig:athletes-composite-misaligned) shows a variation of Figure \@ref(fig:athletes-composite-good) where now all figure elements are slightly out of alignment. I have added axis lines to all panels of Figure \@ref(fig:athletes-composite-misaligned) to emphasize these alignment problems. Notice how no axis line is aligned with any other axis line for any other panel of the figure.

(ref:athletes-composite-misaligned) Variation of Figure \@ref(fig:athletes-composite-good) where all figure panels are slightly misaligned. Misalignments are ugly and should be avoided.

```{r athletes-composite-misaligned, fig.width = 5*6/4.2, fig.asp = 0.75, fig.cap = '(ref:athletes-composite-misaligned)'}
p1 <- ggplot(athletes_df, aes(x = sex, fill = sex)) + 
  geom_bar() +
  scale_y_continuous(limits = c(0, 95), expand = c(0, 0), name = "number") +
  scale_x_discrete(name = NULL, labels = c("female", "male")) +
  scale_fill_manual(values = c("#D55E00D0", "#0072B2D0"), guide = "none") +
  theme_dviz_open(12, rel_small = 1) +
  background_grid(major = "y") +
  theme(
    #axis.line.x = element_blank(),
    #axis.ticks.x = element_blank(),
    #axis.ticks.length = grid::unit(0, "pt"),
    plot.margin = margin(3, 6, 6, 0)
  )

p2 <- ggplot(athletes_df, aes(x = rcc, y = wcc, fill = sex)) + 
  geom_point(pch = 21, color = "white", size = 2.5) +
  scale_x_continuous(limits = c(3.8, 6.75), name = "RBC count") +
  scale_y_continuous(limits = c(2.2, 11.), expand = c(0, 0), name = "WBC count") +
  scale_fill_manual(values = c("#D55E00D0", "#0072B2D0"), guide = "none") +
  theme_dviz_open(12, rel_small = 1) +
  background_grid(major = "y") +
  theme(plot.margin = margin(3, 18, 0, 0))

p_row <- plot_grid(
  NULL, p1, p2, labels = c("", "a", "b"), nrow = 1,
  rel_widths = c(0.03, 0.7, 1)
)

GeomBP <- GeomBoxplot
GeomBP$draw_key <- draw_key_polygon

p3 <- ggplot(athletes_df, aes(x = sport, y = pcBfat, color = sex, fill = sex)) + 
  stat_boxplot(width = 0.5, geom = GeomBP) +
  scale_color_manual(
    values = c("#D55E00", "#0072B2"), name = NULL,
    labels = c("female   ", "male")
  ) +
  scale_fill_manual(values = c("#D55E0040", "#0072B240"), guide = "none") +
  scale_x_discrete(name = NULL) +
  scale_y_continuous(name = "% body fat") +
  guides(color = guide_legend(
    override.aes = list(
      fill = c("#D55E00D0", "#0072B2D0"),
      color = "white", size = 2
    ),
    direction = "horizontal")
  ) +
  theme_dviz_open(12, rel_small = 1) +
  background_grid(major = "y") +
  theme(
    #axis.line.x = element_blank(),
    #axis.ticks.x = element_blank(),
    #axis.ticks.length = grid::unit(0, "pt"),
    legend.position = c(1., 0.95),
    legend.justification = "right"
  )

stamp_ugly(
  plot_grid(
    p_row, p3, ncol = 1, labels = c("", "c")
  ) +
  theme(plot.margin = margin(6, 6, 0, 1.5))
)
```

<!--chapter:end:multi-panel_figures.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")

library(ggforce)
library(treemapify)
```

# Visualizing nested proportions {#nested-proportions}

In the preceding chapter, I discussed scenarios where a dataset is broken into pieces defined by one categorical variable, such as political party, company, or health status. It is not uncommon, however, that we want to drill down further and break down a dataset by multiple categorical variables at once. For example, in the case of parliamentary seats, we could be interested in the proportions of seats by party and by the gender of the representatives. Similarly, in the case of people's health status, we could ask how health status further breaks down by marital status. I refer to these scenarios as nested proportions, because each additional categorical variable that we add creates a finer subdivision of the data nested within the previous proportions. There are several suitable approaches to visualize such nested proportions, including  mosaic plots, treemaps, and parallel sets.

## Nested proportions gone wrong

I will begin by demonstrating two flawed approaches to visualizing nested proportions. While these approaches may seem nonsensical to any experienced data scientist, I have seen them in the wild and therefore think they warrant discussion. Throughout this chapter, I will work with a dataset of 106 bridges in Pittsburgh. This dataset contains various pieces of information about the bridges, such as the material from which they are constructed (steel, iron, or wood) and the year when they were erected. Based on the year of erection, bridges are grouped into distinct categories, such as crafts bridges that were erected before 1870 and modern bridges that were erected after 1940.

Let's assume we want to visualize both the fraction of bridges made from steel, iron, or wood and the fraction that are crafts or modern. We might be tempted to do so by drawing a combined pie chart (Figure \@ref(fig:bridges-pie-wrong)). However, this visualization is not valid. All the slices in a pie chart must add up to 100%, and here the slices add up to 135%. We reach a total percentage in excess of 100% because we are double-counting bridges. Every bridge in the dataset is made of steel, iron, or wood, so these three slices of the pie already represent 100% of the bridges. Every crafts or modern bridge is also a steel, iron, or wood bridge, and hence is counted twice in the pie chart.

(ref:bridges-pie-wrong)  Breakdown of bridges in Pittsburgh by construction material (steel, wood, iron) and by date of construction (crafts, before 1870, and modern, after 1940), shown as a pie chart. Numbers represent the percentages of bridges of a given type among all bridges. This figure is invalid, because the percentages add up to more than 100%. There is overlap between construction material and date of construction. For example, all modern bridges are made of steel, and the majority of crafts bridges are made of wood. Data source: Yoram Reich and Steven J. Fenves, via the UCI Machine Learning Repository [@UCI_repo_2017]


```{r bridges-pie-wrong, fig.width = 5, fig.asp = 0.7, fig.cap = '(ref:bridges-pie-wrong)'}

# crafts: before 1870
# emerging: 1870 -- 1889
# mature: 1890 -- 1939
# modern: after 1940

select(bridges, MATERIAL, ERECTED) %>%
  table() %>% 
  reshape2::melt() %>%
  rename(material = MATERIAL, erected = ERECTED, count = value) %>%
  mutate(
    material = case_when(
      material == "IRON" ~ "iron",
      material == "STEEL" ~ "steel",
      material == "WOOD" ~ "wood"
    ),
    erected = case_when(
      erected == "CRAFTS" ~ "crafts",
      erected == "EMERGING" ~ "emerging",
      erected == "MATURE" ~ "mature",
      erected == "MODERN" ~ "modern"
   )
  ) %>%
  group_by(erected) %>%
  mutate(group_count = sum(count)) -> bridges_tidy


n_total <- sum(bridges_tidy$count)

bridges_erected <- filter(bridges_tidy, erected %in% c("crafts", "modern")) %>%
  group_by(erected) %>%
  summarize(
    count = sum(count),
    percent = round(100*count/n_total, 1)
  ) %>%
  rename(type = erected)

bridges_material <- group_by(bridges_tidy, material) %>%
  summarize(
    count = sum(count),
    percent = round(100*count/n_total, 1)
  ) %>%
  rename(type = material)

bridges_material_erected <- rbind(bridges_material, bridges_erected) %>%
  mutate(
    type = factor(type, levels = c("steel", "wood", "iron", "modern", "crafts"))
  ) %>%
  arrange(type)

bridges_pie <- bridges_material_erected %>%
  mutate(
    count_total = sum(count),
    end_angle = 2*pi*cumsum(count)/count_total,   # ending angle for each pie slice
    start_angle = lag(end_angle, default = 0),   # starting angle for each pie slice
    mid_angle = 0.5*(start_angle + end_angle),   # middle of each pie slice, for the text label
    hjust = ifelse(mid_angle>pi, 1, 0),
    vjust = ifelse(mid_angle<pi/2 | mid_angle>3*pi/2, 0, 1)
  )

rpie = 1
rlabel = 1.05 * rpie

p_bridges_pie <- ggplot(bridges_pie) + 
  geom_arc_bar(
    aes(
      x0 = 0, y0 = 0, r0 = 0, r = rpie,
      start = start_angle, end = end_angle, fill = type
    ),
    color = "white", size = 0.5
  ) +
  geom_text(
    aes(
      x = rlabel*sin(mid_angle),
      y = rlabel*cos(mid_angle),
      label = type,
      hjust = hjust, vjust = vjust
    ),
    family = dviz_font_family,
    size = 14/.pt
  ) +
  geom_text(
    aes(
      x = 0.6*sin(mid_angle),
      y = 0.6*cos(mid_angle),
      label = paste0(percent, "%")
    ),
    family = dviz_font_family,
    size = 12/.pt,
    color = c("white", "white", "white", "black", "black")
  ) +
  coord_fixed(clip = "off") +
  scale_x_continuous(
    limits = c(-1.5, 1.5), expand = c(0, 0), name = "", breaks = NULL, labels = NULL
  ) +
  scale_y_continuous(
    limits = c(-1.15, 1.15), expand = c(0, 0), name = "", breaks = NULL, labels = NULL
  ) +
  scale_fill_manual(
    values = c(iron = "#D55E00D0", wood = "#009E73D0", steel = "#0072B2D0",
               crafts = "#F0E442D0", modern = "#56B4E9D0")
  ) +
  theme_dviz_map() +
  theme(legend.position = "none")

stamp_wrong(p_bridges_pie)
```

Double-counting is not necessarily a problem if we choose a visualization that does not require the proportions to add to 100%. As discussed in the preceding chapter, side-by-side bars meet this criterion. We can show the various proportions of bridges as bars in a single plot, and this plot is not technically wrong (Figure \@ref(fig:bridges-bars-bad)). Nevertheless, I have labeled it as "bad", because it does not immediately show that there is overlap among some of the categories shown. A casual observer might conclude from Figure \@ref(fig:bridges-bars-bad) that there are five separate categories of bridges, and that, for example, modern bridges are neither made of steel nor of wood or iron.

(ref:bridges-bars-bad) Breakdown of bridges in Pittsburgh by construction material (steel, wood, iron) and by date of construction (crafts, before 1870, and modern, after 1940), shown as a bar plot. Unlike Figure \@ref(fig:bridges-pie-wrong), this visualization is not technically wrong, since it doesn't imply that the bar heights need to add up to 100%. However, it also does not clearly indicate the overlap among different groups, and therefore I have labeled it "bad". Data source: Yoram Reich and Steven J. Fenves, via the UCI Machine Learning Repository [@UCI_repo_2017]


```{r bridges-bars-bad, fig.cap = '(ref:bridges-bars-bad)'}
p_bridges_bars <- ggplot(bridges_material_erected) +
  aes(type, percent, fill = type) +
  geom_col() +
  scale_y_continuous(
    limits = c(0, 75),
    expand = c(0, 0),
    labels = function(x) paste0(x, "%"),
    name = "proportion of bridges"
  ) +
  scale_x_discrete(name = NULL) +
  scale_fill_manual(
    values = c(iron = "#D55E00D0", wood = "#009E73D0", steel = "#0072B2D0",
               crafts = "#F0E442D0", modern = "#56B4E9D0")
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank(),
    legend.position = "none",
    plot.margin = margin(3.5, 7, 3.5, 1.5)
  )

stamp_bad(p_bridges_bars)
```


## Mosaic plots and treemaps

Whenever we have categories that overlap, it is best to show clearly how they relate to each other. This can be done with a mosaic plot (Figure \@ref(fig:bridges-mosaic)). On first glance, a mosaic plot looks similar to a stacked bar plot (e.g., Figure \@ref(fig:marketshare-stacked)). However, unlike in a stacked bar plot, in a mosaic plot both the heights and the widths of individual shaded areas vary. Note that in Figure \@ref(fig:bridges-mosaic), we see two additional construction eras, *emerging* (from 1870 to 1889) and *mature* (1890 to 1939). In combination with crafts and modern, these construction eras cover all bridges in the dataset, as do the three building materials. This is a critical condition for a mosaic plot: Every categorical variable shown must cover all the observations in the dataset.


(ref:bridges-mosaic) Breakdown of bridges in Pittsburgh by construction material (steel, wood, iron) and by era of construction (crafts, emerging, mature, modern), shown as a mosaic plot. The widths of each rectangle are proportional to the number of bridges constructed in that era, and the heights are proportional to the number of bridges constructed from that material. Numbers represent the counts of bridges within each category. Data source: Yoram Reich and Steven J. Fenves, via the UCI Machine Learning Repository [@UCI_repo_2017]

```{r bridges-mosaic, fig.cap = '(ref:bridges-mosaic)'}
select(bridges, MATERIAL, ERECTED) %>%
  table() %>% 
  reshape2::melt() %>%
  rename(material = MATERIAL, erected = ERECTED, count = value) %>%
  mutate(
    material = case_when(
      material == "IRON" ~ "iron",
      material == "STEEL" ~ "steel",
      material == "WOOD" ~ "wood"
    ),
    erected = case_when(
      erected == "CRAFTS" ~ "crafts",
      erected == "EMERGING" ~ "emerging",
      erected == "MATURE" ~ "mature",
      erected == "MODERN" ~ "modern"
   )
  ) %>%
  group_by(erected) %>%
  mutate(group_count = sum(count)) -> bridges_tidy

labels_df <- group_by(bridges_tidy, erected) %>%
  filter(count != 0) %>%
  arrange(desc(material)) %>%
  mutate(
    y = (cumsum(count) - 0.5*count)/group_count,
    y = ifelse(
      erected == "mature" & material == "wood", NA, y
    )
  )

ggplot(bridges_tidy) +
  aes(x = erected, y = count, width = group_count, fill = material) +
  geom_bar(stat = "identity", position = "fill", colour = "white", size = 1) +
  geom_text(
    data = labels_df,
    aes(y = y, label = count, color = material),
    na.rm = TRUE,
    size = 12/.pt,
    family = dviz_font_family
  ) +
  facet_grid(~erected, scales = "free_x", space = "free_x") +
  scale_y_continuous(
    name = NULL,
    #breaks = NULL,
    expand = c(0, 0),
    breaks = filter(labels_df, erected == "crafts")$y,
    labels = filter(labels_df, erected == "crafts")$material,
    sec.axis = dup_axis(
      breaks = filter(labels_df, erected == "modern")$y,
      labels = filter(labels_df, erected == "modern")$material
    )
  ) +
  scale_x_discrete(
    name = NULL
  ) +
  scale_fill_manual(
    values = c("#D55E00D0", "#0072B2D0", "#009E73D0"), 
    guide = "none"
  ) +
  scale_color_manual(
    values = c(iron = "white", wood = "white", steel = "white"),
    guide = "none"
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_grid(rel_small = 1) +
  theme(
    line = element_blank(),
    strip.text = element_blank(),
    axis.ticks.length = unit(0, "pt"),
    panel.spacing.x = unit(0, "pt")
  )
```


To draw a mosaic plot, we begin by placing one categorical variable along the *x* axis (here, era of bridge construction) and subdivide the *x* axis by the relative proportions that make up the categories. We then place the other categorical variable along the *y* axis (here, building material) and, within each category along the *x* axis, subdivide the *y* axis by the relative proportions that make up the categories of the *y* variable. The result is a set of rectangles whose areas are proportional to the number of cases representing each possible combination of the two categorical variables.


The bridges dataset can also be visualized in a related but distinct format called a *treemap*. In a treemap, just as is the case in a mosaic plot, we take an enclosing rectangle and subdivide it into smaller rectangles whose areas represent the proportions. However, the method of placing the smaller rectangles into the larger one is different compared to the mosaic plot. In a treemap, we recursively nest rectangles inside each other. For example, in the case of the Pittsburgh bridges, we can first subdivide the total area into three parts representing the three building materials wood, iron, and steel. Then, we subdivide each of those areas further to represent the construction eras represented for each building material (Figure \@ref(fig:bridges-treemap)). In principle, we could keep going with nesting ever more smaller subdivisions inside each other, though relatively quickly the result would become unwieldy or confusing.

(ref:bridges-treemap) Breakdown of bridges in Pittsburgh by construction material (steel, wood, iron) and by era of construction (crafts, emerging, mature, modern), shown as a treemap. The area of each rectangle is proportional to the number of bridges of that type. Data source: Yoram Reich and Steven J. Fenves, via the UCI Machine Learning Repository [@UCI_repo_2017]

```{r bridges-treemap, fig.asp = 3/4, fig.cap = '(ref:bridges-treemap)'}
filcols <- c("#D55E00D0", "#0072B2D0", "#009E73D0")
filcols <- c(vapply(filcols, function(x) c(lighten(x, .9), lighten(x, .6), lighten(x, .3), x), character(4)))

ggplot(bridges_tidy, aes(area = count, subgroup = material, fill = interaction(erected, material))) +
  geom_treemap(color = "white", size = 0.5*.pt, alpha = NA) + 
  geom_treemap_subgroup_text(
    family = dviz_font_family,
    colour = "grey50",
    place = "centre", alpha = 0.7,
    grow = TRUE
  ) +
  geom_treemap_subgroup_border(color = "white") +
  geom_treemap_text(
    aes(label = erected, color = interaction(erected, material)),
    family = dviz_font_family,
    place = "centre",
    grow = FALSE
  ) +
  scale_fill_manual(values = filcols) +
  scale_color_manual(values = c(
    crafts.iron = "black", crafts.steel = "black", crafts.wood = "black",
    emerging.iron = "black", emerging.steel = "black", emerging.wood = "black",
    mature.iron = "black", mature.steel = "black", mature.wood = "black",
    modern.iron = "white", modern.steel = "white", modern.wood = "white")
  ) +
  coord_cartesian(clip = "off") +
  guides(colour = "none", fill = "none")

```

While mosaic plots and treemaps are closely related, they have different points of emphasis and different application areas. Here, the mosaic plot (Figure \@ref(fig:bridges-mosaic))  emphasizes the temporal evolution in building-material use from the crafts era to the modern era, whereas the treemap (Figure \@ref(fig:bridges-treemap)) emphasizes the total number of steel, iron, and wood bridges.

More generally, mosaic plots assume that all of the proportions shown can be identified via combinations of two or more orthogonal categorical  variables. For example, in Figure \@ref(fig:bridges-mosaic), every bridge can be described by a choice of building material (wood, iron, steel) and a choice of time period (crafts, emerging, mature, modern). Moreover, in principle every combination of these two variable is possible, even though in practice this need not be the case. (Here, there are no steel crafts bridges and no wood or iron modern bridges.) By contrast, such a requirement does not exist for treemaps. In fact, treemaps tend to work well when the proportions cannot meaningfully be described by combining multiple categorical variables. For example, we can separate the U.S. into four regions (West, Northeast, Midwest, and South) and each region into distinct states, but the states in one region have no relationship to the states in another region (Figure \@ref(fig:US-states-treemap)).

(ref:US-states-treemap) States in the U.S. visualized as a treemap. Each rectangle represents one state, and the area of each rectangle is proportional to the state's land surface area. The states are grouped into four regions, West, Northeast, Midwest, and South. The coloring is proportional to the number of inhabitants for each state, with darker colors representing larger numbers of inhabitants. Data source: 2010 U.S. Census

```{r US-states-treemap, fig.width = 5.5*6/4.2, fig.cap = '(ref:US-states-treemap)'}
population_df <- left_join(US_census, US_regions) %>%
    group_by(region, division, state) %>%
    summarize(
      pop2000 = sum(pop2000, na.rm = TRUE),
      pop2010 = sum(pop2010, na.rm = TRUE),
      area = sum(area)
    ) %>%
    ungroup() %>%
    mutate(
      state = factor(state, levels = state),
      region = factor(region, levels = c("West", "South", "Midwest", "Northeast"))
    )

## manually add colors
# hues
hues <- c(300, 50, 250, 100) # purple, brown, blue, green
hues <- c(50, 100, 250, 300) # brown, green, blue, purple

# minimum and maximum population density
minpop <- min(population_df$pop2010)
maxpop <- max(population_df$pop2010)

# turn pop density into color
population_df_color <- population_df %>%
  mutate(index = as.numeric(factor(region))) %>%
  group_by(index) %>%
  mutate(
    value = (pop2010-minpop)/(maxpop-minpop),
    fill = scales::gradient_n_pal(
      colorspace::sequential_hcl(
        6,
        h = hues[index[1]],
        c = c(45, 20),
        l = c(30, 80),
        power = .5
      )
    )(1-value)
  )

ggplot(population_df_color, aes(area = area, subgroup = region, fill = fill)) +
  geom_treemap(color = "white", size = 0.5*.pt, alpha = NA) + 
  geom_treemap_subgroup_text(
    family = dviz_font_family,
    colour = "white",
    place = "centre", alpha = 0.7,
    grow = TRUE
  ) +
  geom_treemap_subgroup_border(color = "white") +
  geom_treemap_text(
    aes(label = state),
    color = "black",
    family = dviz_font_family,
    place = "centre",
    grow = FALSE
  ) +
  scale_fill_identity() +
  coord_cartesian(clip = "off") +
  guides(colour = "none", fill = "none")
```

Both mosaic plots and treemaps are commonly used and can be illuminating, but they have similar limitations as do stacked bars (Chapter \@ref(tab:pros-cons-pie-bar)): A direct comparison among conditions can be difficult, because different rectangles do not necessarily share baselines that enable visual comparison. In mosaic plots or treemaps, this problem is exacerbated by the fact that the shapes of the different rectangles can vary. For example, there are the same number of iron bridges (three) among the emerging and the mature bridges, but this is difficult to discern in the mosaic plot (Figure \@ref(fig:bridges-mosaic)), because the two rectangles representing these two groups of three bridges have entirely different shapes. There isn't necessarily a solution to this problem---visualizing nested proportions can be tricky. Whenever possible, I recommend showing the actual counts or percentages on the plot, so readers can verify that their intuitive interpretation of the shaded areas is correct.

## Nested pies

At the beginning of this chapter, I visualized the bridges dataset with a flawed pie chart (Figure \@ref(fig:bridges-pie-wrong)), and I then argued that a mosaic plot or a treemap are more appropriate. However, both of these latter plot types are closely related to pie charts, since they all use area to represent data values. The primary difference is the type of coordinate system, polar in the case of a pie chart versus cartesian in the case of a mosaic plot or treemap. This close relationship between these different plots begs the question whether some variant of a pie chart can be used to visualize this dataset.

There are two possibilities. First, we can draw a pie chart composed of an inner and an outer circle (Figure \@ref(fig:bridges-nested-pie)). The inner circle shows the breakdown of the data by one variable (here, building material) and the outer circle shows the breakdown of each slice of the inner circle by the second variable (here, era of bridge construction). This visualization is reasonable but I have my reservations, and therefore I have labeled it "ugly". Most importantly, the two separate circles obscure the fact that each bridge in the dataset has both a building material and an era of bridge construction. In effect, in Figure \@ref(fig:bridges-nested-pie), we are still double-counting each bridge. If we add up all the numbers shown in the two circles we obtain 212, which is twice the number of bridges in the dataset.

(ref:bridges-nested-pie) Breakdown of bridges in Pittsburgh by construction material (steel, wood, iron, inner circle) and by era of construction (crafts, emerging, mature, modern, outer circle). Numbers represent the counts of bridges within each category. Data source: Yoram Reich and Steven J. Fenves, via the UCI Machine Learning Repository [@UCI_repo_2017]

```{r bridges-nested-pie, fig.asp = 3/4, fig.cap = '(ref:bridges-nested-pie)'}
bridges_arranged <-
  ungroup(bridges_tidy) %>%
  mutate(material = factor(material, levels = c("wood", "iron", "steel"))) %>%
  arrange(material)

bridges_pie_outer <- bridges_arranged %>%
  mutate(
    count_total = sum(count),
    end_angle = 2*pi*cumsum(count)/count_total,   # ending angle for each pie slice
    start_angle = lag(end_angle, default = 0),   # starting angle for each pie slice
    mid_angle = 0.5*(start_angle + end_angle),   # middle of each pie slice, for the text label
    hjust = ifelse(mid_angle>pi, 1, 0),
    vjust = ifelse(mid_angle<pi/2 | mid_angle>3*pi/2, 0, 1),
    type = erected,
    label = paste0(erected, " (", material, ")"),
    angle_off = ifelse(
      label == "emerging (wood)", -0.0175/2,
      ifelse(label == "mature (wood)", 2*0.0175, 0)
    )
  ) %>%
  #filter(erected %in% c("crafts", "modern"), count != 0)
  filter(count != 0)


bridges_pie_inner <- bridges_arranged %>%
  group_by(material) %>%
  summarize(count = sum(count)) %>%
  mutate(
    count_total = sum(count),
    end_angle = 2*pi*cumsum(count)/count_total,   # ending angle for each pie slice
    start_angle = lag(end_angle, default = 0),   # starting angle for each pie slice
    mid_angle = 0.5*(start_angle + end_angle),   # middle of each pie slice, for the text label
    hjust = ifelse(mid_angle>pi, 1, 0),
    vjust = ifelse(mid_angle<pi/2 | mid_angle>3*pi/2, 0, 1),
    type = material
  ) 

rpie <- 1
rpie1 <- 0.6
rpie2 <- 1
rlabel <- 1.02 * rpie

bridges_nested_pie <- ggplot() + 
  geom_arc_bar(data = bridges_pie_outer,
    aes(
      x0 = 0, y0 = 0, r0 = rpie1, r = rpie2,
      start = start_angle, end = end_angle, fill = type
    ),
    color = "white", size = 0.5
  ) +
  geom_arc_bar(data = bridges_pie_inner,
    aes(
      x0 = 0, y0 = 0, r0 = 0, r = rpie1,
      start = start_angle, end = end_angle, fill = type
    ),
    color = "white", size = 0.5
  ) +
  geom_text(data = bridges_pie_outer,
    aes(
      x = rlabel*sin(mid_angle + angle_off),
      y = rlabel*cos(mid_angle + angle_off),
      label = label,
      hjust = hjust, vjust = vjust
    ),
    family = dviz_font_family,
    size = 12/.pt
  ) +
  geom_text(data = bridges_pie_outer,
    aes(
      x = 0.78*sin(mid_angle),
      y = 0.78*cos(mid_angle),
      label = count
    ),
    family = dviz_font_family,
    size = 10/.pt,
    hjust = 0.5, vjust = 0.5
  ) +
  geom_text(data = bridges_pie_inner,
    aes(
      x = 0.32*sin(mid_angle),
      y = 0.32*cos(mid_angle),
      label = count
    ),
    family = dviz_font_family,
    size = 10/.pt,
    hjust = 0.5, vjust = 0.5
  ) +
  coord_fixed(clip = "off") +
  scale_x_continuous(
    limits = c(-1.5, 1.8), expand = c(0, 0), name = "", breaks = NULL, labels = NULL
  ) +
  scale_y_continuous(
    limits = c(-1.15, 1.15), expand = c(0, 0), name = "", breaks = NULL, labels = NULL
  ) +
  scale_fill_manual(
    values = c(iron = "#D55E00D0", wood = "#009E73D0", steel = "#0072B2D0",
               crafts = "#F0E442D0", modern = "#56B4E9D0", emerging = "#E69F00D0",
               mature = "#CC79A7D0")
  ) +
  theme_dviz_map() +
  theme(legend.position = "none")

stamp_ugly(bridges_nested_pie)
```

Alternatively, we can first slice the pie into pieces representing the proportions according to one variable (e.g. material) and then subdivide these slices further according to the other variable (construction era) (Figure \@ref(fig:bridges-nested-pie2)). In this way, in effect we are making a normal pie chart with a large number of small pie slices. However, we can then use coloring to indicate the nested nature of the pie. In Figure \@ref(fig:bridges-nested-pie2), green colors represent wood bridges, orange colors represent iron bridges, and blue colors represent steel bridges. The darkness of each color represents the construction era, with darker colors corresponding to more recently constructed bridges. By using a nested color scale in this way, we can visualize the breakdown of the data both by the primary variable (construction material) and by the secondary variable (construction era).

(ref:bridges-nested-pie2) Breakdown of bridges in Pittsburgh by construction material (steel, wood, iron) and by era of construction (crafts, emerging, mature, modern). Numbers represent the counts of bridges within each category.  Data source: Yoram Reich and Steven J. Fenves, via the UCI Machine Learning Repository [@UCI_repo_2017]

```{r bridges-nested-pie2, fig.asp = 3/4, fig.cap = '(ref:bridges-nested-pie2)'}
rpie <- 1
rpie1 <- 0
rpie2 <- 1
rlabel <- 1.02 * rpie

bridges_nested_pie2 <- ggplot() + 
  geom_arc_bar(data = bridges_pie_outer,
    aes(
      x0 = 0, y0 = 0, r0 = rpie1, r = rpie2,
      start = start_angle, end = end_angle, fill = label
    ),
    color = "white", size = 0.5
  ) +
  geom_text(data = bridges_pie_outer,
    aes(
      x = rlabel*sin(mid_angle + angle_off),
      y = rlabel*cos(mid_angle + angle_off),
      label = label,
      hjust = hjust, vjust = vjust
    ),
    family = dviz_font_family,
    size = 12/.pt
  ) +
  geom_text(data = bridges_pie_outer,
    aes(
      x = 0.6*sin(mid_angle),
      y = 0.6*cos(mid_angle),
      label = count
    ),
    color = c(rep("black", 8), "white"),
    family = dviz_font_family,
    size = 10/.pt,
    hjust = 0.5, vjust = 0.5
  ) +
  coord_fixed(clip = "off") +
  scale_x_continuous(
    limits = c(-1.5, 1.8), expand = c(0, 0), name = "", breaks = NULL, labels = NULL
  ) +
  scale_y_continuous(
    limits = c(-1.15, 1.15), expand = c(0, 0), name = "", breaks = NULL, labels = NULL
  ) +
  scale_fill_manual(
    values = c(
      `crafts (wood)` = lighten("#009E73D0", .9),
      `emerging (wood)` = lighten("#009E73D0", .6),
      `mature (wood)` = lighten("#009E73D0", .3),
      `crafts (iron)` = lighten("#D55E00D0", .9),
      `emerging (iron)` = lighten("#D55E00D0", .6),
      `mature (iron)` = lighten("#D55E00D0", .3),
      `emerging (steel)` = lighten("#0072B2D0", .6),
      `mature (steel)` = lighten("#0072B2D0", .3),
      `modern (steel)` = "#0072B2D0"
    )
  ) +
  theme_dviz_map() +
  theme(legend.position = "none")

bridges_nested_pie2
```

The pie chart of Figure \@ref(fig:bridges-nested-pie2) represents a reasonable visualization of the bridges dataset, but in a direct comparison to the equivalent treemap (Figure \@ref(fig:bridges-treemap)) I think the treemap is preferable. First, the rectangular shape of the treemap allows it to make better use of the available space. Figures \@ref(fig:bridges-treemap) and \@ref(fig:bridges-nested-pie2) are of exactly equal size, but in Figure \@ref(fig:bridges-nested-pie2) much of the figure is wasted as white space. Figure \@ref(fig:bridges-treemap), the treemap, has virtually no superfluous white space. This matters because it enables me to place the labels inside the shaded areas in the treemap. Inside labels always create a stronger visual unit with the data than outside labels and hence are preferred. Second, some of the pie slices in Figure \@ref(fig:bridges-nested-pie2) are very thin and thus hard to see. By contrast, every rectangle in Figure \@ref(fig:bridges-treemap) is of a reasonable size.

## Parallel sets

When we want to visualize proportions described by more than two categorical variables, mosaic plots, treemaps, and pie charts all can quickly become unwieldy. A viable alternative in this case can be a *parallel sets plot*. In a parallel sets plot, we show how the total dataset breaks down by each individual categorical variable, and then we draw shaded bands that show how the subgroups relate to each other. See Figure \@ref(fig:bridges-parallel-sets1) for an example. In this figure, I have broken down the bridges dataset by construction material (iron, steel, wood), length of each bridge (long, medium, short), the era during which each bridge was constructed (crafts, emerging, mature, modern), and the river each bridge spans (Allegheny, Monongahela, Ohio). The bands that connect the parallel sets are colored by construction material. This shows, for example, that wood bridges are mostly of medium length (with a few short bridges), were primarily erected during the crafts period (with a few bridges of medium length erected during the emerging and mature periods), and span primarily the Allegheny river (with a few crafts bridges spanning the Monongahela river). By contrast, iron bridges are all of medium length, were primarily erected during the crafts period, and span the Allegheny and Monongahela rivers in approximately equal proportions.

(ref:bridges-parallel-sets1) Breakdown of bridges in Pittsburgh by construction material, length, era of construction, and the river they span, shown as a parallel sets plot. The coloring of the bands highlights the construction material of the different bridges. Data source: Yoram Reich and Steven J. Fenves, via the UCI Machine Learning Repository [@UCI_repo_2017]

```{r bridges-parallel-sets1, fig.width = 5*6/4.2, fig.cap = '(ref:bridges-parallel-sets1)'}
select(bridges, MATERIAL, ERECTED, RIVER, LENGTH) %>%
  filter(RIVER != "Y") %>%
  table() %>% 
  reshape2::melt() %>%
  rename(material = MATERIAL, erected = ERECTED, length = LENGTH, river = RIVER, count = value) %>%
  mutate(
    material = factor(
      case_when(
        material == "IRON" ~ "iron",
        material == "STEEL" ~ "steel",
        material == "WOOD" ~ "wood"
      ),
      levels = c("wood", "steel", "iron")
    ),
    erected = factor(
      case_when(
        erected == "CRAFTS" ~ "crafts",
        erected == "EMERGING" ~ "emerging",
        erected == "MATURE" ~ "mature",
        erected == "MODERN" ~ "modern"
      ),
      levels = c("modern", "mature", "emerging", "crafts")
    ),
    length = factor(
      case_when(
        length == "LONG" ~ "long",
        length == "MEDIUM" ~ "medium",
        length == "SHORT" ~ "short"
      ),
      levels = c("short", "medium", "long")
    ),
    river = factor(
      case_when(
        river == "A" ~ "Allegheny",
        river == "M" ~ "Monongahela",
        river == "O" ~ "Ohio"
      ),
      levels = c("Ohio", "Monongahela", "Allegheny")
    )
  ) -> data

data <- gather_set_data(data, 1:4)
data$x <- factor(data$x, levels = c("material", "length", "erected", "river"))

ggplot(data, aes(x, id = id, split = y, value = count)) +
  geom_parallel_sets(aes(fill = material), alpha = 0.5, axis.width = 0.13) +
  geom_parallel_sets_axes(axis.width = 0.1, fill = "grey80", color = "grey80") +
  geom_parallel_sets_labels(
    color = 'black',
    family = dviz_font_family,
    size = 10/.pt,
    angle = 90
  ) +
  scale_x_discrete(
    name = NULL,
    expand = c(0, 0.2)
  ) +
  scale_y_continuous(breaks = NULL, expand = c(0, 0))+
  scale_fill_manual(
    values = c(iron = "#D55E00D0", wood = "#009E73D0", steel = "#0072B2D0"),
    guide = "none"
  ) +
  theme_dviz_open() +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_blank(),
    plot.margin = margin(14, 1.5, 2, 1.5)
  )

```

The same visualization looks quite different if we color by a different criterion, for example by river (Figure \@ref(fig:bridges-parallel-sets2)). This figure is visually busy, with many criss-crossing bands, but we do see that nearly any bridge of any type can be found to span each river.


(ref:bridges-parallel-sets2) Breakdown of bridges in Pittsburgh by construction material, length, era of construction, and the river they span. This figure is similar to Figure \@ref(fig:bridges-parallel-sets1) but now the coloring of the bands highlights the river spanned by the different bridges. This figure is labeled "ugly" because the arrangement of the colored bands in the middle of the figure is very busy, and also because the bands need to be read from right to left. Data source: Yoram Reich and Steven J. Fenves, via the UCI Machine Learning Repository [@UCI_repo_2017]

```{r bridges-parallel-sets2, fig.width = 5*6/4.2, fig.cap = '(ref:bridges-parallel-sets2)'}
data$x <- factor(data$x, levels = c("material", "length", "erected", "river"))

p_ugly <- ggplot(data, aes(x, id = id, split = y, value = count)) +
  geom_parallel_sets(aes(fill = river), alpha = 0.5, axis.width = 0.13) +
  geom_parallel_sets_axes(axis.width = 0.1, fill = "grey80", color = "grey80") +
  geom_parallel_sets_labels(
    color = 'black',
    family = dviz_font_family,
    size = 10/.pt,
    angle = 90
  ) +
  scale_x_discrete(
    name = NULL,
    expand = c(0, 0.2)
  ) +
  scale_y_continuous(breaks = NULL, expand = c(0, 0))+
  scale_fill_manual(
    values = c("#CC79A7D0", "#56B4E9D0", "#F0E442D0"),
    guide = "none"
  ) +
  theme_dviz_open() +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_blank(),
    plot.margin = margin(14, 1.5, 2, 1.5)
  )

stamp_ugly(p_ugly)
```

I have labeled Figure \@ref(fig:bridges-parallel-sets2) as "ugly" because I think it is overly complex and confusing. First, since we are used to reading from left to right I think the sets that define the coloring should appear all the way to the left, not on the right. This will make it easier to see where the coloring originates and how it flows through the dataset. Second, it is a good idea to change the order of the sets such that the amount of criss-crossing bands is minimized. Following these principles, I arrive at Figure \@ref(fig:bridges-parallel-sets3), which I consider preferable to Figure \@ref(fig:bridges-parallel-sets2).

(ref:bridges-parallel-sets3) Breakdown of bridges in Pittsburgh by river, era of construction, length, and construction material. This figure differs from Figure \@ref(fig:bridges-parallel-sets2) only in the order of the parallel sets. However, the modified order results in a figure that is easier to read and less busy. Data source: Yoram Reich and Steven J. Fenves, via the UCI Machine Learning Repository [@UCI_repo_2017]

```{r bridges-parallel-sets3, fig.width = 5*6/4.2, fig.cap = '(ref:bridges-parallel-sets3)'}
#data$x <- factor(data$x, levels = c("river", "length", "material", "erected"))
data$x <- factor(data$x, levels = rev(c("material", "length", "erected", "river")))
data$x <- factor(data$x, levels = c("river", "length", "material", "erected"))

ggplot(data, aes(x, id = id, split = y, value = count)) +
  geom_parallel_sets(aes(fill = river), alpha = 0.5, axis.width = 0.13) +
  geom_parallel_sets_axes(axis.width = 0.1, fill = "grey80", color = "grey80") +
  geom_parallel_sets_labels(
    color = 'black',
    family = dviz_font_family,
    size = 10/.pt,
    angle = 90
  ) +
  scale_x_discrete(
    name = NULL,
    expand = c(0, 0.2)
  ) +
  scale_y_continuous(breaks = NULL, expand = c(0, 0))+
  scale_fill_manual(
    values = c("#CC79A7D0", "#56B4E9D0", "#F0E442D0"),
    guide = "none"
  ) +
  theme_dviz_open() +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_blank(),
    plot.margin = margin(14, 1.5, 2, 1.5)
  )
```

<!--chapter:end:nested_proportions.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")

library(tidyr)
```

# Don't go 3D {#no-3d}

3D plots are quite popular, in particular in business presentations but also among academics. They are also almost always inappropriately used. It is rare that I see a 3D plot that couldn't be improved by turning it into a regular 2D figure. In this chapter, I will explain why 3D plots have problems, why they generally are not needed, and in what limited circumstances 3D plots may be appropriate.

## Avoid gratuitous 3D

Many visualization softwares enable you to spruce up your plots by turning the plots' graphical elements into three-dimensional objects. Most commonly, we see pie charts turned into disks rotated in space, bar plots turned into columns, and line plots turned into bands. Notably, in none of these cases does the third dimension convey any actual data. 3D is used simply to decorate and adorn the plot. I consider this use of 3D as gratuitous. It is unequivocally bad and should be erased from the visual vocabulary of data scientists.

The problem with gratuitous 3D is that the projection of 3D objects into two dimensions for printing or display on a monitor distorts the data. The human visual system tries to correct for this distortion as it maps the 2D projection of a 3D image back into a 3D space. However, this correction can only ever be partial. As an example, let's take a simple pie chart with two slices, one representing 25% of the data and one 75%, and rotate this pie in space (Figure \@ref(fig:rotated-pie)). As we change the angle at which we're looking at the pie, the size of the slices seems to change as well. In particular, the 25% slice, which is located in the front of the pie, looks much bigger than 25% when we look at the pie from a flat angle (Figure \@ref(fig:rotated-pie)a).

(ref:rotated-pie) The same 3D pie chart shown from four different angles. Rotating a pie into the third dimension makes pie slices in the front appear larger than they really are and pie slices in the back appear smaller. Here, in parts (a), (b), and (c), the blue slice corresponding to 25% of the data visually occupies more than 25% of the area representing the pie. Only part (d) is an accurate representation of the data.

```{r rotated-pie, fig.asp = 5.1/6, fig.cap = '(ref:rotated-pie)'}
ggdraw() + draw_image("figures/3d/3d-pie-assembled.png")
```

Similar problems arise for other types of 3D plot. Figure \@ref(fig:titanic-3d) shows the breakdown of Titanic passengers by class and gender using 3D bars. Because of the way the bars are arranged relative to the axes, the bars all look shorter than they actually are. For example, there were 322 passengers total traveling in 1st class, yet Figure \@ref(fig:titanic-3d) suggests that the number was less than 300. This illusion arises because the columns representing the data are located at a distance from the two back surfaces on which the gray horizontal lines are drawn. To see this effect, consider extending any of the bottom edges of one of the columns until it hits the lowest gray line, which represents 0. Then, imagine doing the same to any of the top edges, and you'll see that all columns are taller than they appear at first glance. (See Figure \@ref(fig:titanic-passengers-by-class-sex) in Chapter \@ref(visualizing-amounts) for a more reasonable 2D version of this figure.) 

(ref:titanic-3d) Numbers of female and male passengers on the Titanic traveling in 1st, 2nd, and 3rd class, shown as a 3D stacked bar plot. The total numbers of passengers in 1st, 2nd, and 3rd class are 322, 279, and 711, respectively (see Figure \@ref(fig:titanic-passengers-by-class-sex)). Yet in this plot, the 1st class bar appears to represent fewer than 300 passengers, the 3rd class bar appears to represent fewer than 700 passengers, and the 2nd class bar seems to be closer to 210--220 passengers than the actual 279 passengers. Furthermore, the 3rd class bar visually dominates the figure and makes the number of passengers in 3rd class appear larger than it actually is. 

```{r titanic-3d, fig.asp = 4.5/6, fig.cap = '(ref:titanic-3d)'}
stamp_bad(ggdraw() + draw_image("figures/3d/titanic-3d-bars-assembled.png"))
```


## Avoid 3D position scales

While visualizations with gratuitous 3D can easily be dismissed as bad, it is less clear what to think of visualizations using three genuine position scales (*x*, *y*, and *z*) to represent data. In this case, the use of the third dimension serves an actual purpose. Nevertheless, the resulting plots are frequently difficult to interpret, and in my mind they should be avoided.

Consider a 3D scatter plot of fuel efficiency versus displacement and power for 32 cars. We have seen this dataset previously in Chapter \@ref(aesthetic-mapping), Figure \@ref(fig:mtcars-five-scale).  Here, we plot displacement along the *x* axis, power along the *y* axis, and fuel efficiency along the *z* axis, and we represent each car with a dot (Figure \@ref(fig:mtcars-3d)). Even though this 3D visualization is shown from four different perspectives, it is difficult to envision how exactly the points are distributed in space. I find part (d) of Figure \@ref(fig:mtcars-3d) particularly confusing. It almost seems to show a different dataset, even though nothing has changed other than the angle from which we look at the dots.

(ref:mtcars-3d) Fuel efficiency versus displacement and power for 32 cars (1973–74 models). Each dot represents one car, and the dot color represents the number of cylinders of the car. The four panels (a)--(d) show exactly the same data but use different perspectives. Data source: *Motor Trend,* 1974.


```{r mtcars-3d, fig.width = 5*6/4.2, fig.asp = 1.1, fig.cap = '(ref:mtcars-3d)'}
library(plot3D)
library(cowplot)
set_null_device("png")

colors <- c("#0072B2", "#CC79A7", "#E69F00")

cyls <- data.frame(cyl = factor(c(4, 6, 8)))

p <- ggplot(cyls, aes(cyl, cyl, color = cyl)) + 
  geom_point(size = 2.5) +
  scale_color_manual(
    values = colors,
    name = "cylinders ",
    labels = c("4  ", "6  ", "8")
  ) +
  theme_dviz_open(font_size = 14, rel_small = 1, font_family = dviz_font_family) +
  theme(
    legend.position = "top",
    legend.justification = "right",
    legend.key.width = unit(4, "pt")
  )
legend <- get_legend(p)

pfun <- function(theta = 30, phi = 20) {
  function() {
    par(
      xpd = NA,
      bg = "transparent",
      mai = c(0, 0.1, 0, 0),
      family = dviz_font_family_condensed
    )
    scatter3D(
      mtcars$disp, mtcars$hp, mtcars$mpg, colvar = mtcars$cyl,
      col = colors,
      pch = 19, bty ="b2", theta = theta, phi = phi, colkey = FALSE, 
      xlab = "displacement (cu. in.)",
      ylab ="power (hp)",
      zlab = "efficiency (mpg)",
      cex.lab = 1.17
    )
  }
}

plot_grid(pfun(30, 20), pfun(-30, 20), 
          NULL, legend,
          pfun(30, 40), pfun(-30, 40),
          rel_heights = c(1, 0.1, 1), ncol = 2,
          labels = c("a", "b", "", "", "c", "d"),
          label_fontface = "plain", label_fontfamily = dviz_font_family)
```

The fundamental problem with such 3D visualizations is that they require two separate, successive data transformations. The first transformation maps the data from the data space into the 3D visualization space, as discussed in Chapters \@ref(aesthetic-mapping) and \@ref(coordinate-systems-axes) in the context of position scales. The second one maps the data from the 3D visualization space into the 2D space of the final figure. (This second transformation obviously does not occur for visualizations shown in a true 3D environment, such as when shown as physical sculptures or 3D-printed objects. My primary objection here is to 3D visualizations shown on 2D displays.) The second transformation is non-invertible, because each point on the 2D display corresponds to a line of points in the 3D visualization space. Therefore, we cannot uniquely determine where in 3D space any particular data point lies.

Our visual system nevertheless attempts to invert the 3D to 2D transformation. However, this process is unreliable, fraught with error, and highly dependent on appropriate cues in the image that convey some sense of three-dimensionality. When we remove these cues the inversion becomes entirely impossible. This can be seen in Figure \@ref(fig:mtcars-3d-no-axes), which is identical to Figure \@ref(fig:mtcars-3d) except all depth cues have been removed. The result is four random arrangements of points that we cannot interpret at all and that aren't even easily relatable to each other. Could you tell which points in part (a) correspond to which points in part (b)? I certainly cannot.

(ref:mtcars-3d-no-axes) Fuel efficiency versus displacement and power for 32 cars (1973–74 models). The four panels (a)--(d) correspond to the same panels in Figure \@ref(fig:mtcars-3d), only that all grid lines providing depth cues have been removed. Data source: *Motor Trend,* 1974.

```{r mtcars-3d-no-axes, fig.width = 5*6/4.2, fig.asp = 1.1, fig.cap = '(ref:mtcars-3d-no-axes)'}
pfun2 <- function(theta = 30, phi = 20) {
  function() {
    par(
      xpd = NA,
      bg = "transparent",
      mai = c(0, 0.1, 0, 0),
      family = dviz_font_family_condensed
    )
    scatter3D(
      mtcars$disp, mtcars$hp, mtcars$mpg, colvar = mtcars$cyl,
      col = colors,
      pch = 19, axes = FALSE, theta = theta, phi = phi,
      colkey = FALSE, box = FALSE,
      cex.lab = 1.17
    )
  }
}


plot_grid(
  pfun2(30, 20), pfun2(-30, 20), 
  NULL, legend,
  pfun2(30, 40), pfun2(-30, 40),
  rel_heights = c(1, 0.1, 1), ncol = 2,
  labels = c("a", "b", "", "", "c", "d"),
  label_fontface = "plain", label_fontfamily = dviz_font_family
)

```

Instead of applying two separate data transformations, one of which is non-invertible, I think it is generally better to just apply one appropriate, invertible transformation and map the data directly into 2D space. It is rarely necessary to add a third dimension as a position scale, since variables can also be mapped onto color, size, or shape scales. For example, in Chapter \@ref(aesthetic-mapping), I plotted five variables of the fuel-efficiency dataset at once yet used only two position scales (Figure \@ref(fig:mtcars-five-scale)).

Here, I want to show two alternative ways of plotting exactly the variables used in Figure \@ref(fig:mtcars-3d). First, if we primarily care about fuel efficiency as the response variable, we can plot it twice, once against displacement and once against power (Figure \@ref(fig:mtcars-2d-multiple)). Second, if we are more interested in how displacement and power relate to each other, with fuel efficiency as a secondary variable of interest, we can plot power versus displacement and map fuel efficiency onto the size of the dots (Figure \@ref(fig:mtcars-2d-size)). Both figures are more useful and less confusing than Figure \@ref(fig:mtcars-3d).

(ref:mtcars-2d-multiple) Fuel efficiency versus displacement (a) and power (b). Data source: *Motor Trend,* 1974.

```{r mtcars-2d-multiple, fig.width = 5*6/4.2, fig.asp = .45, fig.cap = '(ref:mtcars-2d-multiple)'}
p1 <- ggplot(mtcars, aes(x = disp, y = mpg, color = factor(cyl))) +
  geom_point(size = 1.5) + 
  scale_color_manual(values = colors, name = "cylinders", guide = "none") +
  xlab("displacement (cu. in.)") +
  ylab("efficiency (mpg)") +
  theme_dviz_open(12)

p2 <- ggplot(mtcars, aes(x = hp, y = mpg, color = factor(cyl))) +
  geom_point(size = 1.5) + 
  scale_color_manual(values = colors, name = "cylinders") +
  xlab("power (hp)") +
  ylab("efficiency (mpg)") +
  theme_dviz_open(12) +
  theme(
    legend.position = c(1, 1),
    legend.justification = c(1, 1),
    legend.spacing.y = grid::unit(3, "pt"),
    legend.key.width = unit(6, "pt")
  )


plot_grid(p1, align_legend(p2), labels = "auto")
```

(ref:mtcars-2d-size) Power versus displacement for 32 cars, with fuel efficiency represented by dot size. Data source: *Motor Trend,* 1974.

```{r mtcars-2d-size, fig.width = 5.5, fig.asp = 0.75, fig.cap = '(ref:mtcars-2d-size)'}
p <- ggplot(mtcars, aes(x = disp, y = hp, size = mpg, fill = factor(cyl))) +
  geom_point(color = "white", pch = 21) + 
  scale_fill_manual(
    values = colors, name = "cylinders",
    guide = guide_legend(override.aes = list(size = 3))
  ) +
  scale_size(
    name = "    mpg    ",
    range = c(1, 8),
    limits = c(8, 40),
    breaks = c(5, 10, 20, 40),
    guide = guide_legend(override.aes = list(fill = "gray50"))
  ) +
  xlab("displacement (cu. in.)") +
  ylab("power (hp)") +
  theme_dviz_open() +
  theme(
    legend.title.align = 0.5,
    legend.spacing.y = grid::unit(4, "pt")
  )

ggdraw(align_legend(p))
```

You may wonder whether the problem with 3D scatter plots is that the actual data representation, the dots, do not themselves convey any 3D information. What happens, for example, if we use 3D bars instead? Figure \@ref(fig:VA-death-rates-3d) shows a typical dataset that one might visualize with 3D bars, the mortality rates in 1940 Virginia stratified by age group and by gender and housing location. We can see that indeed the 3D bars help us interpret the plot. It is unlikely that one might mistake a bar in the foreground for one in the background or vise versa. Nevertheless, the problems discussed in the context of Figure \@ref(fig:titanic-3d) exist here as well. It is difficult to judge exactly how tall the individual bars are, and it is also difficult to make direct comparisons. For example, was the mortality rate of urban females in the 65--69 age group higher or lower than that of urban males in the 60--64 age group?

(ref:VA-death-rates-3d) Mortality rates in Virginia in 1940, visualized as a 3D bar plot. Mortality rates are shown for four groups of people (urban and rural females and males) and five age categories (50--54, 55--59, 60--64, 65--69, 70--74), and they are reported in units of deaths per 1000 persons. This figure is labeled as "bad" because the 3D perspective makes the plot difficult to read. Data source: @Molyneaux-et-al-1947

```{r VA-death-rates-3d, fig.width = 5, fig.asp = 0.8, fig.cap = '(ref:VA-death-rates-3d)'}
pfun3 <- function() {

  par(xpd = NA,
      bg = "transparent",
      mai = c(0.1, 0.1, 0, 0),
      family = dviz_font_family_condensed
  )

  hist3D(x = 1:5, y = 1:4, z = VADeaths,
         bty = "b2", phi = 20,  theta = -65,
         xlab = "", ylab = "", zlab = "deaths / 1000",
         col = "#56B4E9", border = darken("#56B4E9", .5), 
         shade = 0.2,
         ticktype = "detailed", space = .3, d = 2, cex.axis = 1e-9
  )
  # Use text3D to label x axis
  text3D(x = 1:5, y = rep(0.5, 5), z = rep(3, 5),
         labels = rownames(VADeaths),
         add = TRUE, adj = -0.2)
  # Use text3D to label y axis
  text3D(x = rep(0., 3), y = rep(5, 3), z = 20*(1:3),
         labels  = 20*(1:3),
         add = TRUE, adj = -1.2)
  # Use text3D to label z axis
  text3D(x = rep(1, 4), y = 1:4, z = rep(0, 4),
         labels  = colnames(VADeaths),
         add = TRUE, adj = 1)
}

# Doesn't currently work, due to bug in gridGraphics. Revisit later.
# For now, we do a work-around by rendering to png.
#stamp_bad(pfun3)

png("figures/VA-deaths.png", width = 4.64, height = 0.618*6, units = "in",
    res = 600)
pfun3()
null <- dev.off()

stamp_bad(ggdraw() +  draw_image("figures/VA-deaths.png"))
```

In general, it is better to use Trellis plots (Chapter \@ref(multi-panel-figures)) instead of 3D visualizations. The Virginia mortality dataset requires only four panels when shown as Trellis plot (Figure \@ref(fig:VA-death-rates-Trellis)). I consider this figure clear and easy to interpret. It is immediately obvious that mortality rates were higher among men than among women, and also that urban males seem to have had higher mortality rates than rural males whereas no such trend is apparent for urban and rural females.

(ref:VA-death-rates-Trellis) Mortality rates in Virginia in 1940, visualized as a Trellis plot. Mortality rates are shown for four groups of people (urban and rural females and males) and five age categories (50--54, 55--59, 60--64, 65--69, 70--74), and they are reported in units of deaths per 1000 persons.  Data source: @Molyneaux-et-al-1947

```{r VA-death-rates-Trellis, fig.width = 5*6/4.2, fig.cap = '(ref:VA-death-rates-Trellis)'}
df <- data.frame(VADeaths)
df$age <- row.names(df)
row.names(df) <- NULL
df_long <- gather(df, type, rate, -age) %>%
  mutate(type = 
    case_when(
      type == "Urban.Male" ~ "urban male",
      type == "Urban.Female" ~ "urban female",
      type == "Rural.Male" ~ "rural male",
      type == "Rural.Female" ~ "rural female"
    )
  )
ggplot(df_long, aes(age, rate)) + 
  geom_col(fill = "#56B4E9D0") + 
  facet_wrap(~type) +
  scale_y_continuous(name = "deaths / 1000", expand = c(0, 0)) +
  scale_x_discrete(name = "age group") +
  theme_dviz_hgrid()
```

## Appropriate use of 3D visualizations

Visualizations using 3D position scales can sometimes be appropriate, however. First, the issues described in the preceding section are of lesser concern if the visualization is interactive and can be rotated by the viewer, or alternatively, if it is shown in a VR or augmented reality environment where it can be inspected from multiple angles. Second, even if the visualization isn't interactive, showing it slowly rotating, rather than as a static image from one perspective, will allow the viewer to discern where in 3D space different graphical elements reside. The human brain is very good at reconstructing a 3D scene from a series of images taken from different angles, and the slow rotation of the graphic provides exactly these images.

Finally, it makes sense to use 3D visualizations when we want to show actual 3D objects and/or data mapped onto them. For example, showing the topographic relief of a mountainous island is a reasonable choice (Figure \@ref(fig:corsica-relief)). Similarly, if we want to visualize the evolutionary sequence conservation of a protein mapped onto its structure, it makes sense to show the structure as a 3D object (Figure \@ref(fig:protein-3d)). In either case, however, these visualizations would still be easier to interpret if they were shown as rotating animations. While this is not possible in traditional print publications, it can be done easily when posting figures on the web or when giving presentations.

(ref:corsica-relief) Relief of the Island of Corsica in the Mediterranean Sea. Data source: Copernicus Land Monitoring Service

```{r corsica-relief, fig.cap = '(ref:corsica-relief)'}
# figure rendered with rayshader; it's too slow to do while rendering the R markdown
knitr::include_graphics("figures/3d/Corsica.png", auto_pdf = FALSE)
```


(ref:protein-3d) Patterns of evolutionary variation in a protein. The colored tube represents the backbone of the protein Exonuclease III from the bacterium *Escherichia coli* (Protein Data Bank identifier: 1AKO). The coloring indicates the evolutionary conservation of the individual sites in this protein, with dark coloring indicating conserved amino acids and light coloring indicating variable amino acids. Data source: @Marcos-Echave-2015

```{r protein-3d, fig.width = 4.5, fig.asp = (16+4)/19, fig.cap = '(ref:protein-3d)'}
# Make legend via ggplot2
df <- data.frame(x = 1:10,
                 fill = runif(10))

p <- ggplot(df, aes(x, y = 1, fill = fill)) + geom_tile() +
  scale_fill_gradient2(low = darken("#A6522B", .07), mid = darken("#FFFF00", .05),
                       high = darken("#FFFFFF", .02),
                       midpoint = .5,
                       limits = c(0, 1),
                       breaks = c(0, 1),
                       labels = c("highly\nconserved", "highly\nvariable"),
                       name = "sequence conservation",
                       guide = guide_colorbar(direction = "horizontal",
                                              label.position = "bottom",
                                              title.position = "top",
                                              ticks = FALSE,
                                              barwidth = grid::unit(3.5, "in"),
                                              barheight = grid::unit(0.2, "in"))) +
  theme_dviz_open(12) +
  theme(legend.title.align = 0.5,
        legend.background = element_blank(),
        legend.box.background = element_blank(),
        legend.justification = "center")

legend <- get_legend(p)

plot_grid(ggdraw() +  draw_image("figures/3d/1AKO-cropped.png"),
          legend, ncol = 1, rel_heights = c(16, 4))
```

<!--chapter:end:no_3d.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(lubridate)
```

# Handling overlapping points {#overlapping-points}

When we want to visualize large or very large datasets, we often experience the challenge that simple *x*--*y* scatter plots do not work very well because many points lie on top of each other and partially or fully overlap. And similar problems can arise even in small datasets if data values were recorded with low precision or rounded, such that multiple observations have exactly the same numeric values. The technical term commonly used to describe this situation is "overplotting", i.e., plotting many points on top of each other. Here I describe several strategies you can pursue when encountering this challenge.


## Partial transparency and jittering

We first consider a scenario with only a moderate number of data points but with extensive rounding. Our dataset contains fuel economy during city driving and engine displacement for 234 popular car models released between 1999 and 2008 (Figure \@ref(fig:mpg-cty-displ-solid)). In this dataset, fuel economy is measured in miles per gallon (mpg) and is rounded to the nearest integer value. Engine displacement is measured in liters and is rounded to the nearest deciliter. Due to this rounding, many car models have exactly identical values. For example, there are 21 cars total with 2.0 liter engine displacement, and as a group they have only four different fuel economy values, 19, 20, 21, or 22 mpg. Therefore, in Figure \@ref(fig:mpg-cty-displ-solid) these 21 cars are represented by only four distinct points, so that 2.0 liter engines appear much less popular than they actually are. Moreover, the dataset contains two four-wheel drive cars with 2.0 liter engines, which are represented by black dots. However, these black dots are fully occluded by yellow dots, so that it looks like there are no four-wheel drive cars with a 2.0 liter engine.

(ref:mpg-cty-displ-solid) City fuel economy versus engine displacement, for popular cars released between 1999 and 2008. Each point represents one car. The point color encodes the drive train: front-wheel drive (FWD), rear-wheel drive (RWD), or four-wheel drive (4WD). The figure is labeled "bad" because many points are plotted on top of others and obscure them. 

```{r mpg-cty-displ-solid, fig.width=5.5, fig.asp=.7416, fig.cap='(ref:mpg-cty-displ-solid)'}
p_mpg_solid <- ggplot(mpg, aes(y = cty, x = displ, color = drv, fill = drv)) +
  geom_point(size = 3, shape = 21) + 
  ylab("fuel economy (mpg)") +
  xlab("displacement (l)") +
  scale_color_manual(values=c("#202020", "#E69F00", "#56B4E9"), 
                     name="drive train",
                     breaks=c("f", "r", "4"),
                     labels=c("FWD", "RWD", "4WD")) +
  scale_fill_manual(values=c("#202020", "#E69F00", "#56B4E9"), 
                     name="drive train",
                     breaks=c("f", "r", "4"),
                     labels=c("FWD", "RWD", "4WD")) +
  theme_dviz_open() +
  theme(legend.position = c(.7, .8),
        plot.margin = margin(3, 7, 3, 1.5))

stamp_bad(p_mpg_solid)
```

One way to ameliorate this problem is to use partial transparency. If we make individual points partially transparent, then overplotted points appear as darker points and thus the shade of the points reflects the density of points in that location of the graph (Figure \@ref(fig:mpg-cty-displ-transp)).

(ref:mpg-cty-displ-transp) City fuel economy versus engine displacement. Because points have been made partially transparent, points that lie on top of other points can now be identified by their darker shade. 

```{r mpg-cty-displ-transp, fig.width=5.5, fig.asp=.7416, fig.cap='(ref:mpg-cty-displ-transp)'}
p_mpg_transp <- ggplot(mpg, aes(y = cty, x = displ, color = drv, fill = drv)) +
  geom_point(size = 3, shape = 21) + 
  ylab("fuel economy (mpg)") +
  xlab("displacement (l)") +
  scale_color_manual(values=c("#202020", "#E69F00", "#56B4E9"), 
                     name="drive train",
                     breaks=c("f", "r", "4"),
                     labels=c("FWD", "RWD", "4WD")) +
  scale_fill_manual(values=c("#20202080", "#E69F0080", "#56B4E980"), 
                     name="drive train",
                     breaks=c("f", "r", "4"),
                     labels=c("FWD", "RWD", "4WD")) +
  theme_dviz_open() +
  theme(legend.position = c(.7, .8),
        plot.margin = margin(3, 7, 3, 1.5))

p_mpg_transp
```

However, making points partially transparent is not always sufficient to solve the issue of overplotting. For example, even though we can see in Figure \@ref(fig:mpg-cty-displ-transp) that some points have a darker shade than others, it is difficult to estimate how many points were plotted on top of each other in each location. In addition, while the differences in shading are clearly visible, they are not self-explanatory. A reader who sees this figure for the first time will likely wonder why some points are darker than others and will not realize that those points are in fact multiple points stacked on top of each other. A simple trick that helps in this situation is to apply a small amount of jitter to the points, i.e., to displace each point randomly by a small amount in either the *x* or the *y* direction or both. With jitter, it is immediately apparent that the darker areas arise from points that are plotted on top of each other (Figure \@ref(fig:mpg-cty-displ-jitter)). Also, now, for the first time the black dots representing four-wheel drive cars with 2.0 liter engines are clearly visible.

(ref:mpg-cty-displ-jitter) City fuel economy versus engine displacement. By adding a small amount of jitter to each point, we can make the overplotted points more clearly visible without substantially distorting the message of the plot.

```{r mpg-cty-displ-jitter, fig.width=5.5, fig.asp=.7416, fig.cap='(ref:mpg-cty-displ-jitter)'}
p_mpg_jitter <- ggplot(mpg, aes(y = cty, x = displ, color = drv, fill = drv)) +
  geom_point(size = 3, shape = 21,
             position = position_jitter(width = 0.01 * diff(range(mpg$displ)),
                                        height = 0.01 * diff(range(mpg$cty)),
                                        seed = 7384)) + 
  ylab("fuel economy (mpg)") +
  xlab("displacement (l)") +
  scale_color_manual(values=c("#202020", "#E69F00", "#56B4E9"), 
                     name="drive train",
                     breaks=c("f", "r", "4"),
                     labels=c("FWD", "RWD", "4WD")) +
  scale_fill_manual(values=c("#20202080", "#E69F0080", "#56B4E980"), 
                     name="drive train",
                     breaks=c("f", "r", "4"),
                     labels=c("FWD", "RWD", "4WD")) +
  theme_dviz_open() +
  theme(legend.position = c(.7, .8),
        plot.margin = margin(3, 7, 3, 1.5))

p_mpg_jitter
```

One downside of jittering is that it does change the data and therefore has to be performed with care. If we jitter too much, we end up placing points in locations that are not representative of the underlying dataset. The result is a misleading visualization of the data. See Figure \@ref(fig:mpg-cty-displ-jitter-extreme) as an example.

(ref:mpg-cty-displ-jitter-extreme) City fuel economy versus engine displacement. By adding too much jitter to the points, we have created a visualization that does not accurately reflect the underlying dataset.

```{r mpg-cty-displ-jitter-extreme, fig.width=5.5, fig.asp=.7416, fig.cap='(ref:mpg-cty-displ-jitter-extreme)'}
p_mpg_jitter_extreme <- ggplot(mpg, aes(y = cty, x = displ, color = drv, fill = drv)) +
  geom_point(size = 3, shape = 21,
             position = position_jitter(width = 0.1 * diff(range(mpg$displ)),
                                        height = 0.1 * diff(range(mpg$cty)))) + 
  scale_x_continuous(breaks = 2:7) +
  ylab("fuel economy (mpg)") +
  xlab("displacement (l)") +
  scale_color_manual(values=c("#202020", "#E69F00", "#56B4E9"), 
                     name="drive train",
                     breaks=c("f", "r", "4"),
                     labels=c("FWD", "RWD", "4WD")) +
  scale_fill_manual(values=c("#20202080", "#E69F0080", "#56B4E980"), 
                     name="drive train",
                     breaks=c("f", "r", "4"),
                     labels=c("FWD", "RWD", "4WD")) +
  theme_dviz_open() +
  theme(legend.position = c(.7, .8),
        plot.margin = margin(3, 7, 3, 1.5))

stamp_bad(p_mpg_jitter_extreme)
```

## 2D histograms

When the number of individual points gets very large, partial transparency (with or without jittering) will not be sufficient to resolve the overplotting issue. What will typically happen is that areas with high point density will appear as uniform blobs of dark color while in areas with low point density the individual points are barely visible (Figure \@ref(fig:nycflights-points)). And changing the transparency level of individual points will either ameliorate one or the other of these problems while worsening the other; no transparency setting can address both at the same time. 

(ref:nycflights-points) Departure delay in minutes versus the flight departure time, for all flights departing Newark airport (EWR) in 2013. Each dot represents one departure.

```{r nycflights-points, fig.asp = 0.75, fig.cap = '(ref:nycflights-points)'}
# break points along the x axis
breaks_x <- c("0:00", "6:00", "12:00", "18:00", "24:00")

p_flights_base <- ggplot(flight_delays, aes(`departure time`, `departure delay (minutes)`)) + 
  geom_abline(slope = 0, intercept = 0, color="grey70") +
  scale_x_time(
    name = "departure time",
    breaks = hm(breaks_x),
    labels = breaks_x
  ) +
  scale_y_continuous(
    name = "departure delay (minutes)"
  ) +
  theme_dviz_open() +
  theme(plot.margin = margin(3, 7, 3, 1.5))
  
p_flights_scatter <- p_flights_base + geom_point(alpha = 0.1, stroke = 0, size = 2)

stamp_bad(p_flights_scatter)
```

Figure \@ref(fig:nycflights-points) shows departure delays for over 100,000 individual flights, with each dot representing one flight departure. Even though we have made the individual dots fairly transparent, the majority of them just forms a black band between 0 and 300 minutes departure delay. This band obscures whether most flights depart approximately on time or with substantial delay (say 50 minutes or more). At the same time, the most delayed flights (with delays of 400 minutes or more) are barely visible due to the transparency of the dots.

In such cases, instead of plotting individual points, we can make a 2D histogram. A 2D histogram is conceptually similar to a 1D histogram as discussed in Chapter \@ref(histograms-density-plots), but now we bin the data in two dimensions. We subdivide the entire *x*--*y* plane into small rectangles, count how many observations fall into each one, and then color the rectangles by that count. Figure \@ref(fig:nycflights-2d-bins) shows the result of this approach for the departure-delay data. This visualization clearly highlights several important features of the flight-departure data. First, the vast majority of departures during the day (6am to about 9pm) actually depart without delay or even early (negative delay). However, a modest number of departures has a substantial delay. Moreover, the later a plane departs in the day the more of a delay it can have. Importantly, the departure time is the actual time of departure, not the scheduled time of departure. So this figure does not necessarily tell us that planes scheduled to depart early never experience delay. What it does tell us, though, is that if a plane departs early it either has little delay or, in very rare cases, a delay of around 900 minutes.

(ref:nycflights-2d-bins) Departure delay in minutes versus the flight departure time. Each colored rectangle represents all flights departing at that time with that departure delay. Coloring represents the number of flights represented by that rectangle.

```{r nycflights-2d-bins, fig.asp = 0.75, fig.cap = '(ref:nycflights-2d-bins)'}
p_flights_2d_bins <- p_flights_base +
  geom_bin2d(bins=50) +
  #scale_fill_continuous_sequential(palette = "Blue-Yellow", l2 = 90, c2 = 20) +
  scale_fill_continuous_sequential(
    h1 = -83, h2 = 20, c1 = 30, cmax = 40, c2 = 0, l1 = 20, l2 = 94, p1 = 1, p2 = 1.2, 
    rev = TRUE,
    begin = 0.2,
    name = "departures"
  ) +
  theme(legend.position = c(0.85, .85))

p_flights_2d_bins
```


As an alternative to binning the data into rectangle, we can also bin into hexagons. This approach, first proposed by 
@Carr-et-al-1987, has the advantage that the points in a hexagon are, on average, closer to the hexagon center than the points in an equal-area square are to the center of the square. Therefore, the colored hexagon represents the data slightly more accurately than the colored rectangle does. Figure \@ref(fig:nycflights-hex-bins) shows the flight departure data with hexagon binning rather than rectangular binning.

(ref:nycflights-hex-bins) Departure delay in minutes versus the flight departure time. Each colored hexagon represents all flights departing at that time with that departure delay. Coloring represents the number of flights represented by that hexagon.

```{r nycflights-hex-bins, fig.asp = 0.75, fig.cap = '(ref:nycflights-hex-bins)'}
p_flights_hex_bins <- p_flights_base +
  geom_hex(aes(colour = stat(count)), bins = 50) +
  #scale_fill_continuous_sequential(palette = "Blue-Yellow", l2 = 90, c2 = 20) +
  scale_fill_continuous_sequential(
    aesthetics = c("colour", "fill"),
    h1 = -83, h2 = 20, c1 = 30, cmax = 40, c2 = 0, l1 = 20, l2 = 94, p1 = 1, p2 = 1.2, 
    rev = TRUE,
    begin = 0.2,
    name = "departures"
  ) +
  theme(legend.position = c(0.85, .85))

p_flights_hex_bins
```


## Contour lines

Instead of binning data points into rectangles or hexagons, we can also estimate the point density across the plot area and indicate regions of different point densities with contour lines. This technique works well when the point density changes slowly across both the *x* and the *y* dimensions.

As an example for this approach, we return to the blue jays dataset from Chapter \@ref(visualizing-associations). Figure \@ref(fig:blue-jays-scatter) showed the relationship between head length and body mass for 123 blue jays, and there was some amount of overlap among the points. We can highlight the distribution of points more clearly by making the points smaller and partially transparent and plotting them on top of contour lines that delineate regions of similar point density (Figure \@ref(fig:blue-jays-contour)). We can further enhance the perception of changes in the point density by shading the regions enclosed by the contour lines, using darker colors for regions representing higher point densities (Figure \@ref(fig:blue-jays-contour-filled)).

(ref:blue-jays-contour) Head length versus body mass for 123 blue jays, as in Figure \@ref(fig:blue-jays-scatter). Each dot corresponds to one bird, and the lines indicate regions of similar point density. The point density increases towards the center of the plot, near a body mass of 75g and a head length between 55mm and 57.5mm. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-contour, fig.width = 6, fig.asp = 3/4, fig.cap='(ref:blue-jays-contour)'}

blue_jays_base <- ggplot(blue_jays, aes(Mass, Head)) + 
  scale_x_continuous(
    limits = c(57, 82),
    expand = c(0, 0),
    name = "body mass (g)") +
  scale_y_continuous(
    limits = c(49, 61),
    expand = c(0, 0),
    name = "head length (mm)"
  ) +
  theme_dviz_grid()

blue_jays_base + 
  stat_density_2d(color = "black", size = 0.4, binwidth = 0.004) +
  geom_point(color = "black", size = 1.5, alpha = 1/3)
```

(ref:blue-jays-contour-filled) Head length versus body mass for 123 blue jays. This figure is nearly identical to Figure \@ref(fig:blue-jays-scatter), but now the areas enclosed by the contour lines are shaded with increasingly darker shades of gray. This shading creates a stronger visual impression of increasing point density towards the center of the point cloud. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-contour-filled, fig.width = 6, fig.asp = 3/4, fig.cap='(ref:blue-jays-contour-filled)'}
blue_jays_base + 
  stat_density_2d(aes(fill = ..level..), geom = "polygon", color = "black", size = 0.15, binwidth = 0.004) +
  geom_point(color = "black", size = 1.5, alpha = .4) +
  scale_fill_gradient(low = "grey95", high = "grey70", guide = "none")
```

In Chapter \@ref(visualizing-associations), we also looked at the relationship between head length and body mass separately for male and female birds (Figure \@ref(fig:blue-jays-scatter-sex)). We can do the same with contour lines, by drawing separately colored contour lines for male and female birds (Figure \@ref(fig:blue-jays-contour-by-sex)).

(ref:blue-jays-contour-by-sex) Head length versus body mass for 123 blue jays. As in Figure \@ref(fig:blue-jays-scatter-sex), we can also indicate the birds' sex by color when drawing contour lines. This figure highlights how the point distribution is different for male and female birds. In particular, male birds are more densely clustered in one region of the plot area whereas female birds are more spread out. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-contour-by-sex, fig.width = 6, fig.asp = 3/4, fig.cap='(ref:blue-jays-contour-by-sex)'}
blue_jays_base + 
  aes(color = KnownSex) +
  stat_density_2d(size = 0.4, binwidth = 0.006) +
  geom_point(size = 1.5, alpha = 0.7) +
  scale_color_manual(
    values = c(F = "#D55E00", M = "#0072B2"),
    breaks = c("F", "M"),
    labels = c("female birds   ", "male birds"),
    name = NULL,
    guide = guide_legend(
      direction = "horizontal",
      override.aes = list(size = 2, linetype = 0)
    )
  ) +
  theme_dviz_grid() +
  theme(
    legend.position = c(1, 0),
    legend.justification = c(1, 0),
    #legend.position = "top",
    #legend.justification = "right",
    #legend.box.spacing = unit(3.5, "pt"), # distance between legend and plot
    legend.text = element_text(vjust = 0.6),
    legend.spacing.x = unit(2, "pt"),
    legend.background = element_rect(fill = "white", color = NA),
    #legend.key.width = unit(10, "pt")
    axis.ticks.length = unit(0, "pt"),
    axis.ticks = element_blank()
  )
```

Drawing multiple sets of contour lines in different colors can be a powerful strategy for showing the distributions of several point clouds at once. However, this technique needs to be employed with care. It only works when the number of groups with distinct colors is small (two to three) and the groups are clearly separated. Otherwise, we may end up with a hairball of differently colored lines all crisscrossing each other and not showing any particular pattern at all.

To illustrate this potential problem, I will employ the diamonds dataset, which contains information for 53,940 diamonds, including their price, weight (carat), and cut. Figure \@ref(fig:diamonds-points) shows this dataset as a scatter plot. We see clear problems with overplotting. There are so many different-colored points on top of one another that it is impossible to discern anything beyond the overall broad outline of where diamonds fall on the price--carat spectrum.

(ref:diamonds-points) Price of diamonds versus their carat value, for 53,940 individual diamonds. Each diamond's cut is indicated by color. The plot is labeled as "bad" because the extensive overplotting makes it impossible to discern any patterns among the different diamond cuts. Data source: Hadley Wickham, ggplot2

```{r diamonds-points, fig.width = 5*6/4.2, fig.asp = 3/4, fig.cap = '(ref:diamonds-points)'}
p <- ggplot(diamonds, aes(carat, price, color = cut)) + 
  geom_point(size = .2, alpha = 1/5) +
  scale_x_continuous(
    limits = c(-1, 5.1)
  ) +
  scale_y_log10(
    name = "price (USD)",
    breaks = c(300, 1000, 3000, 10000),
    labels = c("$300", "$1,000", "$3,000", "$10,000")
  ) +
  scale_color_discrete_sequential(
    palette = "Inferno",
    nmax = 6,
    order = 1:5,
    rev = FALSE,
    breaks = c("Ideal", "Premium", "Very Good", "Good", "Fair"),
    labels = c("ideal", "premium", "very good", "good", "fair"),
    guide = guide_legend(
      override.aes = list(size = 2, alpha = 1)
    )
  ) +
  coord_cartesian(xlim = c(-.1, 3.2), ylim = c(240, 25000), expand = FALSE) + 
  theme_dviz_grid() +
  panel_border() +
  theme(
    plot.margin = margin(18, 7, 1, 1.5),
    legend.key.width = unit(6, "pt"),
    legend.spacing.y = unit(3, "pt"),
    legend.title = element_text(hjust = 0, margin = margin(0, 0, 0, 0)),
    legend.position = c(.97, .3),
    legend.justification = c(1, 0.5),
    legend.box.margin = margin(7, 7, 7, 7),
    legend.box.background = element_rect(fill = "white", color = NA),
    axis.ticks.length = unit(0, "pt")
  )

stamp_bad(p)
```

We could try to draw colored contour lines for the different qualities of cut, as in Figure \@ref(fig:blue-jays-contour-by-sex). However, in the diamonds dataset, we have five distinct colors and the groups strongly overlap. Therefore, the contour plot (Figure \@ref(fig:diamonds-contour-colors)) is not much better than the original scatter plot (Figure \@ref(fig:diamonds-points)).

(ref:diamonds-contour-colors) Price of diamonds versus their carat value. As Figure \@ref(fig:diamonds-points), but now individual points have been replaced by contour lines. The resulting plot is still labeled "bad", because the contour lines all lie on top of each other. Neither the point distribution for individual cuts nor the overall point distribution can be discerned. Data source: Hadley Wickham, ggplot2


```{r diamonds-contour-colors, fig.width = 5*6/4.2, fig.asp = 3/4, fig.cap = '(ref:diamonds-contour-colors)'}
p <- ggplot(diamonds, aes(carat, price, color = cut)) + 
  geom_density2d(size = .35, binwidth = 0.8) +
  scale_x_continuous(
    limits = c(-1, 5.1)
  ) +
  scale_y_log10(
    name = "price (USD)",
    breaks = c(300, 1000, 3000, 10000),
    labels = c("$300", "$1,000", "$3,000", "$10,000")
  ) +
  scale_color_discrete_sequential(
    palette = "Inferno",
    nmax = 6,
    order = 1:5,
    rev = FALSE,
    breaks = c("Ideal", "Premium", "Very Good", "Good", "Fair"),
    labels = c("ideal", "premium", "very good", "good", "fair"),
    guide = guide_legend(
      override.aes = list(size = 0.5)
    )
  ) +
  coord_cartesian(xlim = c(-.1, 2.3), ylim = c(240, 25000), expand = FALSE) + 
  theme_dviz_grid() +
  panel_border() +
  theme(
    plot.margin = margin(18, 7, 1, 1.5),
    legend.spacing.y = unit(3, "pt"),
    legend.title = element_text(hjust = 0, margin = margin(0, 0, 0, 0)),
    legend.position = c(.97, .3),
    legend.justification = c(1, 0.5),
    legend.box.margin = margin(7, 7, 7, 7),
    legend.box.background = element_rect(fill = "white", color = NA),
    axis.ticks.length = unit(0, "pt")
  )

stamp_bad(p)
```

What helps here is to draw the contour lines for each cut quality in its own plot panel (Figure \@ref(fig:diamonds-contour-facets)). The purpose of drawing them all in one panel might be to enable visual comparison between the groups, but Figure \@ref(fig:diamonds-contour-colors) is so busy that a comparison isn't possible.  Instead, in Figure \@ref(fig:diamonds-contour-facets), the background grid enables us to make  comparisons across cut qualities, by paying attention to where exactly the contour lines fall relative to the grid lines. (A similar effect could have been achieved by plotting partially transparent individual points instead of contour lines in each panel.)

(ref:diamonds-contour-facets) Price of diamonds versus their carat value. Here, we have taken the density contours from Figure \@ref(fig:diamonds-contour-colors) and drawn them separately for each cut. We can now see that better cuts (very good, premium, ideal) tend to have lower carat values than the poorer cuts (fair, good) but command a higher price per carat. Data source: Hadley Wickham, ggplot2


```{r diamonds-contour-facets, fig.width = 5.5*6/4.2, fig.asp = 3/4, fig.cap = "(ref:diamonds-contour-facets)"}
ggplot(diamonds, aes(carat, price)) + 
  stat_density_2d(aes(fill = ..level..), geom = "polygon", color = darken("#0072B2", .2), size = .3, binwidth = 0.8) +
  #geom_density2d(color = darken("#0072B2", .2), size = .3, binwidth = 0.8) +
  scale_fill_gradient(low = desaturate(lighten("#0072B2", .9), .6), high = desaturate(lighten("#0072B2", .6), .6), guide = "none") +
  scale_x_continuous(
    limits = c(-1, 5.1)
  ) +
  scale_y_log10(
    name = "price (USD)",
    breaks = c(300, 1000, 3000, 10000),
    labels = c("$300", "$1,000", "$3,000", "$10,000")
  ) +
  coord_cartesian(xlim = c(-.1, 2.3), ylim = c(200, 25000), expand = FALSE) + 
  facet_wrap(~cut, scales = "free_x", labeller = labeller(cut = tolower)) +
  theme_dviz_grid() +
  panel_border() +
  theme(
    legend.title = element_text(hjust = 0.5),
    legend.position = c(.95, .05),
    legend.justification = c(1, 0),
    axis.ticks.length = unit(0, "pt")
  )
```

We can make out two main trends. First, the better cuts (very good, premium, ideal) tend to have lower carat values than the poorer cuts (fair, good). Recall that carat is a measure of diamond weight (1 carat = 0.2 gram). Better cuts tend to result (on average) in lighter diamonds because more material needs to be removed to create them. Second, at the same carat value, better cuts tend to command higher prices. To see this pattern, look for example at the price distribution for 0.5 carat. The distribution is shifted upwards for better cuts, and in particular it is substantially higher for diamonds with ideal cut than for diamonds with fair or good cut.

<!--chapter:end:overlapping_points.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")
```

# Common pitfalls of color use {#color-pitfalls}

Color can be an incredibly effective tool to enhance data visualizations. At the same time, poor color choices can ruin an otherwise excellent visualization. Color needs to be applied to serve a purpose, it must be clear, and it must not distract.

## Encoding too much or irrelevant information

One common mistake is trying to give color a job that is too big for it to handle, by encoding too many different items in different colors. As an example, consider Figure \@ref(fig:popgrowth-vs-popsize-colored). It shows population growth versus population size for all 50 U.S. states and the District of Columbia. I have attempted to identify each state by giving it its own color. However, the result is not very useful. Even though we can guess which state is which by looking at the colored points in the plot and in the legend, it takes a lot of effort to go back and forth between the two to try to match them up. There are simply too many different colors, and many of them are quite similar to each other. Even if with a lot of effort we can figure out exactly which state is which, this visualization defeats the purpose of coloring. We should use color to enhance figures and make them easier to read, not to obscure the data by creating visual puzzles.

(ref:popgrowth-vs-popsize-colored) Population growth from 2000 to 2010 versus population size in 2000, for all 50 U.S. states and the District of Columbia. Every state is marked in a different color. Because there are so many states, it is very difficult to match the colors in the legend to the dots in the scatter plot. Data source: U.S. Census Bureau

```{r popgrowth-vs-popsize-colored, fig.width = 6, fig.asp = 2*0.618, fig.cap = '(ref:popgrowth-vs-popsize-colored)'}

popgrowth_df <- left_join(US_census, US_regions) %>%
    group_by(region, division, state) %>%
    summarize(pop2000 = sum(pop2000, na.rm = TRUE),
              pop2010 = sum(pop2010, na.rm = TRUE),
              popgrowth = (pop2010-pop2000)/pop2000,
              area = sum(area)) %>%
    arrange(popgrowth) %>%
    ungroup() %>%
    mutate(state = factor(state, levels = state),
           region = factor(region, levels = c("West", "South", "Midwest", "Northeast")))


colors <- c(rainbow_hcl(8, l = 35, c = 25, start = 0, end = 315),
            rainbow_hcl(8, l = 45, c = 34, start = -10, end = 305),
            rainbow_hcl(9, l = 55, c = 42, start = -20, end = 300),
            rainbow_hcl(9, l = 65, c = 50, start = -30, end = 290),
            rainbow_hcl(9, l = 75, c = 55, start = -40, end = 280),
            rainbow_hcl(8, l = 85, c = 32, start = -50, end = 265))

#colors <- sample(colors, 51)

p_base <- ggplot(popgrowth_df, aes(x = pop2000, y = popgrowth, color = as.character(state))) +
  geom_point(size = 4) +
  scale_x_log10(labels = label_log10) +
  scale_y_continuous(labels = scales::percent_format(accuracy = 1)) +
  scale_color_manual(values = colors, name = "state") +
  xlab("population size in 2000") +
  ylab("population growth\n2000 to 2010   ") +
  theme_dviz_grid() +
  theme(
    legend.text = element_text(size = 10),
    legend.title = element_text(size = 12),
    legend.justification = "center",
    legend.box.margin = margin(14, 0, 28, 0),
    legend.spacing.y = unit(2, "pt"),
    plot.margin = margin(14, 14, 14, 1.5)
  )

p_comb <- plot_grid(p_base + theme(legend.position = "none"), get_legend(p_base), ncol = 1)
stamp_bad(p_comb)
```

As a rule of thumb, qualitative color scales work best when there are three to five different categories that need to be colored. Once we reach eight to ten different categories or more, the task of matching colors to categories becomes too burdensome to be useful, even if the colors remain sufficiently different to be distinguishable in principle. For the dataset of Figure \@ref(fig:popgrowth-vs-popsize-colored), it is probably best to use color only to indicate the geographic region of each state and to identify individual states by direct labeling, i.e., by placing appropriate text labels adjacent to the data points (Figure \@ref(fig:popgrowth-vs-popsize-bw)). Even though we cannot label every individual state without making the figure too crowded, direct labeling is the right choice for this figure. In general, for figures such as this one, we don't need to label every single data point. It is sufficient to label a representative subset, for example a set of states we specifically want to call out in the text that will accompany the figure. We always have the option to also provide the underlying data as a table if we want to make sure the reader has access to it in its entirety.

(ref:popgrowth-vs-popsize-bw) Population growth from 2000 to 2010 versus population size in 2000. In contrast to Figure \@ref(fig:popgrowth-vs-popsize-colored), I have now colored states by region and have directly labeled a subset of states. The majority of states have been left unlabeled to keep the figure from overcrowding. Data source: U.S. Census Bureau

```{r popgrowth-vs-popsize-bw, fig.width = 5.5*6/4.2, fig.cap = '(ref:popgrowth-vs-popsize-bw)'}
library(ggrepel)

set.seed(7586)
region_colors <- c("#E69F00", "#56B4E9", "#009E73", "#F0E442")

labeled_states <- c(
  "Alaska", "Arizona", "California", "Florida", "Wisconsin", 
  "Louisiana", "Nevada", "Michigan", "Montana", "New Mexico",
  "Pennsylvania", "New York", "Oregon", "Rhode Island",
  "Tennessee", "Texas", "Utah", "Vermont"
)

df_repel <- select(popgrowth_df, x = pop2000, y = popgrowth, state) %>%
  mutate(label = ifelse(state %in% labeled_states, as.character(state), ""))

ggplot(popgrowth_df, aes(x = pop2000, y = popgrowth, color = region, fill = region)) +
  geom_text_repel(
    data = df_repel,
    aes(x, y, label = label),
    segment.alpha = 0.5, point.padding = 0.25,
    box.padding = .8,
    force = 1,
    min.segment.length = 0.1,
    family = dviz_font_family,
    size = 11/.pt, inherit.aes = FALSE
  ) +
  geom_point(size = 3.5, color = "white") +
  geom_point(size = 2.5, shape = 21) +
  scale_x_log10(labels = label_log10) +
  scale_y_continuous(labels = scales::percent_format(accuracy = 1)) +
  scale_fill_manual(values = region_colors) +
  scale_color_manual(values = darken(region_colors, .3)) +
  xlab("population size in 2000") +
  ylab("population growth, 2000 to 2010") +
  theme_dviz_grid()
```


```{block type='rmdtip', echo=TRUE}
Use direct labeling instead of colors when you need to distinguish between more than about eight categorical items.
```

A second common problem is coloring for the sake of coloring, without having a clear purpose for the colors. As an example, consider Figure \@ref(fig:popgrowth-US-rainbow), which is a variation of Figure \@ref(fig:popgrowth-US). However, now instead of coloring the bars by geographic regions, I have given each bar its own color, so that in aggregate the bars create a rainbow effect. This may look like an interesting visual effect, but it is not creating any new insight into the data or making the figure easier to read.

(ref:popgrowth-US-rainbow) Population growth in the U.S. from 2000 to 2010. The rainbow coloring of states serves no purpose and is distracting. Furthermore, the colors are overly saturated. Data source: U.S. Census Bureau

```{r popgrowth-US-rainbow, fig.width = 6, fig.asp = 1.2, fig.cap = '(ref:popgrowth-US-rainbow)'}
popgrowth_bars_rainbow <- ggplot(popgrowth_df, aes(x = state, y = 100*popgrowth, fill = state)) + 
  geom_col() + 
  scale_y_continuous(
    limits = c(-.6, 37.5), expand = c(0, 0),
    labels = scales::percent_format(accuracy = 1, scale = 1),
    name = "population growth, 2000 to 2010"
  ) +
  scale_fill_hue(c = 140, l = 55) +
  coord_flip() + 
  theme_dviz_vgrid(12, rel_small = 1) +
  theme(axis.title.y = element_blank(),
        axis.line.y = element_blank(),
        axis.ticks.length = unit(0, "pt"),
        axis.text.y = element_text(size = 10),
        legend.position = "none",
        plot.margin = margin(18, 1.5, 3, 1.5))

stamp_ugly(popgrowth_bars_rainbow)
```

Besides the gratuitous use of different colors, Figure \@ref(fig:popgrowth-US-rainbow) has a second color-related problem: The chosen colors are too saturated and intense. This color intensity makes the figure difficult to look at. For example, it is difficult to read the names of the states without having our eyes drawn to the large, strongly colored areas right next to the state names. Similarly, it is difficult to compare the endpoints of the bars to the underlying grid lines.

```{block type='rmdtip', echo=TRUE}
Avoid large filled areas of overly saturated colors. They make it difficult for your reader to carefully inspect your figure.
```


## Using non-monotonic color scales to encode data values

In Chapter \@ref(color-basics), I listed two critical conditions for designing sequential color scales that can represent data values: The colors need to clearly indicate which data values are larger or smaller than which other ones, and the differences between colors need to visualize the corresponding differences between data values. Unfortunately, several existing color scales---including very popular ones---violate one or both of these conditions. The most popular such scale is the rainbow scale (Figure \@ref(fig:rainbow-desaturated)). It runs through all possible colors in the color spectrum. This means the scale is effectively circular; the colors at the beginning and the end are nearly the same (dark red). If these two colors end up next to each other in a plot, we do not instinctively perceive them as representing data values that are maximally apart. In addition, the scale is highly non-monotonic. It has regions where colors change very slowly and others when colors change rapidly. This lack of monotonicity becomes particularly apparent if we look at the color scale in grayscale (Figure \@ref(fig:rainbow-desaturated)). The scale goes from medium dark to light to very dark and back to medium dark, and there are large stretches where lightness changes very little followed by relatively narrow stretches with large changes in lightness. 

(ref:rainbow-desaturated) The rainbow colorscale is highly non-monotonic. This becomes clearly visible by converting the colors to gray values. From left to right, the scale goes from moderately dark to light to very dark and back to moderately dark. In addition, the changes in lightness are very non-uniform. The lightest part of the scale (corresponding to the colors yellow, light green, and cyan) takes up almost a third of the entire scale while the darkest part (corresponding to dark blue) is concentrated in a narrow region of the scale.

```{r rainbow-desaturated, fig.width=5*6/4.2, fig.asp=2*.14, fig.cap = '(ref:rainbow-desaturated)'}
p1 <- gg_color_gradient(
  plot_margin = margin(17.5, 1, 0, 1),
  ymargin = 0.05
) + 
  scale_fill_gradientn(colors = rainbow(10))

p2 <- gg_color_gradient(
  plot_margin = margin(17.5, 1, 0, 1),
  ymargin = 0.05
) + 
  scale_fill_gradientn(colors = desaturate(rainbow(10)))

plot_grid(
  p1, NULL, p2,
  ncol = 1,
  rel_heights = c(0.9, 0.1, 0.9),
  labels = c("rainbow scale", "", "rainbow converted to grayscale"),
  label_x = 0, label_y = 0.96,
  label_size = 14
)
```

In a visualization of actual data, the rainbow scale tends to obscure data features and/or highlight arbitrary aspects of the data (Figure \@ref(fig:map-Texas-rainbow)). As an aside, the colors in the rainbow scale are also overly saturated. Looking at Figure \@ref(fig:map-Texas-rainbow) for any extended period of time can be quite uncomfortable.


(ref:map-Texas-rainbow) Percentage of people identifying as white in Texas counties. The rainbow color scale is not an appropriate scale to visualize continuous data values, because it tends to place emphasis on arbitrary features of the data. Here, it emphasizes counties in which approximately 75% of the population identify as white. Data source: 2010 Decennial U.S. Census

```{r map-Texas-rainbow, fig.width = 6, fig.asp = 0.75, fig.cap = '(ref:map-Texas-rainbow)'}
library(sf)

# EPSG:3083
# NAD83 / Texas Centric Albers Equal Area
# http://spatialreference.org/ref/epsg/3083/
texas_crs <- "+proj=aea +lat_1=27.5 +lat_2=35 +lat_0=18 +lon_0=-100 +x_0=1500000 +y_0=6000000 +ellps=GRS80 +datum=NAD83 +units=m +no_defs"

# -110, -93.5 transformed using texas_crs
texas_xlim <- c(558298.7, 2112587)

# we reuse the geometries from texas_income
texas_race %>% st_sf() %>%
  st_transform(crs = texas_crs) %>%
  filter(variable == "White") %>%
  ggplot(aes(fill = pct)) +
    geom_sf(color = "white") +
    coord_sf(xlim = texas_xlim, datum = NA) + 
    theme_dviz_map() +
    scale_fill_gradientn(
      colors = rainbow(10),
      limits = c(0, 100),
      breaks = 25*(0:4),
      labels = c("0% ", "25%", "50%", "75%", " 100%"),
      name = "percent identifying as white",
      guide = guide_colorbar(
        direction = "horizontal",
        label.position = "bottom",
        title.position = "top",
        ticks = FALSE,
        barwidth = grid::unit(3.0, "in"),
        barheight = grid::unit(0.2, "in")
      )
    ) +
    theme(
      legend.title.align = 0.5,
      legend.text.align = 0.5,
      legend.justification = c(0, 0),
      legend.position = c(0, 0.1)
    ) -> texas_rainbow

stamp_bad(texas_rainbow)
```


## Not designing for color-vision deficiency

Whenever we are choosing colors for a visualization, we need to keep in mind that a good proportion of our readers may have some form of color-vision deficiency (i.e., are colorblind). These readers may not be able to distinguish colors that look clearly different to most other people. People with impaired color vision are not literally unable to see any colors, however. Instead, they will typically have difficulty to distinguish certain types of colors, for example red and green (red--green color-vision deficiency) or blue and green (blue--yellow color-vision deficiency). The technical terms for these deficiencies are deuteranomaly/deuteranopia and protanomaly/protanopia for the red--green variant (where people have difficulty perceiving either green or red, respectively) and tritanomaly/tritanopia for the blue--yellow variant (where people have difficulty perceiving blue). The terms ending in "anomaly" refer to some impairment in the perception of the respective color, and the terms ending in "anopia" refer to complete absence of perception of that color. Approximately 8% of males and 0.5% of females suffer from some sort of color-vision deficiency, and deuteranomaly is the most common form whereas tritanomaly is relatively rare.

As discussed in Chapter \@ref(color-basics), there are three fundamental types of color scales used in data visualization: sequential scales, diverging scales, and qualitative scales. Of these three, sequential scales will generally not cause any problems for people with color-vision deficiency (cvd), since a properly designed sequential scale should present a continuous gradient from dark to light colors. Figure \@ref(fig:heat-cvd-sim) shows the Heat scale from Figure \@ref(fig:sequential-scales) in simulated versions of deuteranomaly, protanomaly, and tritanomaly. While none of these cvd-simulated scales look like the original, they all present a clear gradient from dark to light and they all work well to convey the magnitude of a data value.

(ref:heat-cvd-sim) Color-vision deficiency (cvd) simulation of the sequential color scale Heat, which runs from dark red to light yellow. From left to right and top to bottom, we see the original scale and the scale as seen under deuteranomaly, protanomaly, and tritanomaly simulations. Even though the specific colors look different under the three types of cvd, in each case we can see a clear gradient from dark to light. Therefore, this color scale is safe to use for cvd.

```{r heat-cvd-sim, fig.width = 5*6/4.2, fig.asp = 2*.14, fig.cap = '(ref:heat-cvd-sim)'}
grad_heat <- gg_color_gradient(
  plot_margin = margin(17.5, 1, 0, 1),
  ymargin = 0.05
) + 
  scale_fill_continuous_sequential("Heat", rev = FALSE)

cvd_sim(grad_heat, label_y = 0.96)
```

Things become more complicated for diverging scales, because popular color contrasts can become indistinguishable under cvd. In particular, the colors red and green provide about the strongest contrast for people with normal color vision but become nearly indistinguishable for deutans (people with deuteranomaly) or protans (people with protanomaly) (Figure \@ref(fig:red-green-cvd-sim)). Similarly, blue-green contrasts are visible for deutans and protans but become indistinguishable for tritans (people with tritanomaly) (Figure \@ref(fig:blue-green-cvd-sim)).

(ref:red-green-cvd-sim) A red--green contrast becomes indistinguishable under red--green cvd (deuteranomaly or protanomaly).

```{r red-green-cvd-sim, fig.width = 5*6/4.2, fig.asp = 2*.14, fig.cap = '(ref:red-green-cvd-sim)'}
cols <- scales::colour_ramp(c("#FF1B1B", "#F9F1CE", high = "#057905"))(seq(0, 1, .25))

grad_red_green <- gg_color_swatches(
  n = 5,
  plot_margin = margin(17.5, 1, 0, 1),
  xmargin = 0.1,
  ymargin = 0.05
) + 
  scale_fill_manual(values = cols)

cvd_sim(grad_red_green, label_y = 0.96)
```

(ref:blue-green-cvd-sim) A blue--green contrast becomes indistinguishable under blue--yellow cvd (tritanomaly).

```{r blue-green-cvd-sim, fig.width = 5*6/4.2, fig.asp = 2*.14, fig.cap = '(ref:blue-green-cvd-sim)'}
cols <- scales::colour_ramp(c("#284F9B", "grey90", high = "#056D05"))(seq(0, 1, .25))

grad_red_green <- gg_color_swatches(
  n = 5,
  plot_margin = margin(17.5, 1, 0, 1),
  xmargin = 0.1,
  ymargin = 0.05
) + 
  scale_fill_manual(values = cols)

cvd_sim(grad_red_green, label_y = 0.96)
```

With these examples, it might seem that it is nearly impossible to find two contrasting colors that are safe under all forms of cvd. However, the situation is not that dire. It is often possible to make slight modifications to the colors such that they have the desired character while also being safe for cvd. For example, the ColorBrewer PiYG (pink to yellow-green) scale from Figure \@ref(fig:diverging-scales) looks red--green to people with normal color vision yet remains distinguishable for people with cvd (Figure \@ref(fig:PiYG-cvd-sim)).

(ref:PiYG-cvd-sim) The ColorBrewer PiYG (pink to yellow-green) scale from Figure \@ref(fig:diverging-scales) looks like a red--green contrast to people with regular color vision but works for all forms of color-vision deficiency. It works because the reddish color is actually pink (a mix of red and blue) while the greenish color also contains yellow. The difference in the blue component between the two colors can be picked up even by deutans or protans, and the difference in the red component can be picked up by tritans.

```{r PiYG-cvd-sim, fig.width = 5*6/4.2, fig.asp = 2*.14, fig.cap = '(ref:PiYG-cvd-sim)'}
grad_red_green <- gg_color_swatches(
  n = 5,
  plot_margin = margin(17.5, 1, 0, 1),
  xmargin = 0.1,
  ymargin = 0.05
) + 
  scale_fill_brewer(type = "div", palette = "PiYG")

cvd_sim(grad_red_green, label_y = 0.96)
```

Things are most complicated for qualitative scales, because there we need many different colors and they all need to be distinguishable from each other under all forms of cvd. My preferred qualitative color scale, which I use extensively throughout this book, was developed specifically to address this challenge (Figure \@ref(fig:palette-Okabe-Ito)). By providing eight different colors, the palette works for nearly any scenario with discrete colors. As discussed at the beginning of this chapter, you should probably not color-code more than eight different items in a plot anyways.

(ref:palette-Okabe-Ito) Qualitative color palette for all color-vision deficiencies [@Okabe-Ito-CUD]. The alphanumeric codes represent the colors in RGB space, encoded as hexadecimals. In many plot libraries and image-manipulation programs, you can just enter these codes directly. If your software does not take hexadecimals directly, you can also use the values in Table \@ref(tab:color-codes).

```{r palette-Okabe-Ito, fig.width=5.5*6/4.2, fig.asp=.14, fig.cap = '(ref:palette-Okabe-Ito)'}
# from: http://jfly.iam.u-tokyo.ac.jp/color/
cbPalette <- c("#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7", "#000000")

palette_plot(cbPalette, label_size = 5, label_family = dviz_font_family) +
  theme(plot.margin = margin(3, 1, 3, 1))
```


Table: (\#tab:color-codes) Colorblind-friendly color scale, developed by @Okabe-Ito-CUD. 

Name            Hex code     Hue     C, M, Y, K (%)   R, G, B (0-255)   R, G, B (%)
--------------  ------------ ------- ---------------- ----------------- ------------
        orange       #E69F00     41° 0, 50, 100, 0    230, 159, 0       90, 60, 0
      sky blue       #56B4E9    202° 80, 0, 0, 0      86, 180, 233      35, 70, 90
  bluish green       #009E73    164° 97, 0, 75, 0     0, 158, 115       0, 60, 50
        yellow       #F0E442     56° 10, 5, 90, 0     240, 228, 66      95, 90, 25
          blue       #0072B2    202° 100, 50, 0, 0    0, 114, 178       0, 45, 70
     vermilion       #D55E00     27° 0, 80, 100, 0    213, 94, 0        80, 40, 0
reddish purple       #CC79A7    326° 10, 70, 0, 0     204, 121, 167     80, 60, 70
         black       #000000      -  0, 0, 0, 100     0, 0, 0           0, 0, 0
 
While there are several good, cvd-safe color scales readily available, we need to recognize that they are no magic bullets. It is very possible to use a cvd-safe scale and yet produce a figure a person with cvd cannot decipher. One critical parameter is the size of the colored graphical elements. Colors are much easier to distinguish when they are applied to large areas than to small ones or thin lines [@Stone_et_al_2014]. And this effect is exacerbated under cvd (Figure \@ref(fig:colors-thin-lines)). In addition to the various color-design considerations discussed in this chapter and in Chapter \@ref(color-basics), I recommend to view color figures under cvd simulations to get a sense of what they may look like for a person with cvd. There are several online services and desktop apps available that allow users to run arbitrary figures through a cvd simulation.

(ref:colors-thin-lines) Colored elements become difficult to distinguish at small sizes. The top left panel (labeled "original") shows four rectangles, four thick lines, four thin lines, and four groups of points, all colored in the same four colors. We can see that the colors become more difficult to distinguish the smaller or thinner the visual elements are. This problem becomes exacerbated in the cvd simulations, where the colors are already more difficult to distinguish even for the large graphical elements.

```{r colors-thin-lines, fig.width = 5*6/4.2, fig.asp = 0.8, fig.cap = '(ref:colors-thin-lines)'}
tiles_df <- data.frame(
  x = c(1, 2, 1, 2),
  y = c(1.75, 1.75, 1.25, 1.25),
  type = c("A", "B", "C", "D")
)

segments_df <- data.frame(
  x0 = rep(0.55, 4),
  x1 = rep(2.45, 4),
  y0 = seq(.9, .6, -.1),
  y1 = seq(.9, .6, -.1),
  type = c("A", "B", "C", "D")
)

points_df <- data.frame(
  x = rep(1.58 + .28*(0:3), 4),
  y = rep(seq(.4, .1, -.1), each = 4),
  size = rep(c(3, 2, 1, .5), 4),
  type = rep(c("A", "B", "C", "D"), each = 4)
)

p <- ggplot() + 
  geom_tile(data = tiles_df, aes(x, y, fill = type), width = 0.9, height = 0.45) +
  geom_segment(
    data = segments_df,
    aes(x = x0, xend = x1, y = y0, yend = y1, color = type),
    size = 1.5
  ) +
  geom_segment(
    data = segments_df,
    aes(x = x0, xend = x0 + .9, y = y0 - .5, yend = y1 - .5, color = type),
    size = .5
  ) +
  geom_point(data = points_df, aes(x, y, color = type, size = size)) +
  scale_fill_OkabeIto(order = c(1, 4, 2, 3)) +
  scale_color_OkabeIto(order = c(1, 4, 2, 3)) +
  scale_size_identity() +
  coord_cartesian(xlim = c(0.55, 2.45), ylim = c(0.05, 2.05), expand = FALSE) +
  theme_nothing() +
  theme(plot.margin = margin(14, 1, 0, 1))

cvd_sim(p, label_y = 0.97)
```

```{block type='rmdtip', echo=TRUE}
To make sure your figures work for people with cvd, don't just rely on specific color scales. Instead, test your figures in a cvd simulator.
```


```{r eval = FALSE, include = FALSE} 
## Notes, entered as R comment so as to not appear in the printed output

#Some useful posts on color-blind friendly palettes:
#
#- [Color Universal Design by Okabe and Ito](http://jfly.iam.u-tokyo.ac.jp/color/)
#- [Gradient-based color palettes](https://blog.graphiq.com/finding-the-right-color-palettes-for-data-visualizations-fcd4e707a283)
#- [Avoid equidistant HSV colors](https://www.vis4.net/blog/posts/avoid-equidistant-hsv-colors/)
```

<!--chapter:end:pitfalls_of_color_use.Rmd-->

# Preface {-}

If you are a scientist, an analyst, a consultant, or anybody else who has to prepare technical documents or reports, one of the most important skills you need to have is the ability to make compelling data visualizations, generally in the form of figures. Figures will typically carry the weight of your arguments. They need to be clear, attractive, and convincing. The difference between good and bad figures can be the difference between a highly influential or an obscure paper, a grant or contract won or lost, a job interview gone well or poorly. And yet, there are surprisingly few resources to teach you how to make compelling data visualizations. Few colleges offer courses on this topic, and there are not that many books on this topic either. (Some exist, of course.) Tutorials for plotting software typically focus on how to achieve specific visual effects rather than explaining why certain choices are preferred and others not. In your day-to-day work, you are simply expected to know how to make good figures, and if you're lucky you have a patient adviser who teaches you a few tricks as you're writing your first scientific papers.

In the context of writing, experienced editors talk about "ear", the ability to hear (internally, as you read a piece of prose) whether the writing is any good. I think that when it comes to figures and other visualizations, we similarly need "eye", the ability to look at a figure and see whether it is balanced, clear, and compelling. And just as is the case with writing, the ability to see whether a figure works or not can be learned. Having eye means primarily that you are aware of a larger collection of simple rules and principles of good visualization, and that you pay attention to little details that other people might not.

In my experience, again just as in writing, you don't develop eye by reading a book over the weekend. It is a lifelong process, and concepts that are too complex or too subtle for you today may make much more sense five years from now. I can say for myself that I continue to evolve in my understanding of figure preparation. I routinely try to expose myself to new approaches, and I pay attention to the visual and design choices others make in their figures. I'm also open to change my mind. I might today consider a given figure great, but next month I might find a reason to criticize it. So with this in mind, please don't take anything I say as gospel. Think critically about my reasoning for certain choices and decide whether you want to adopt them or not.

While the materials in this book are presented in a logical progression, most chapters can stand on their own, and there is no need to read the book cover to cover. Feel free to skip around, to pick out a specific section that you're interested in at the moment, or one that covers a specific design choice you're pondering. In fact, I think you will get the most out of this book if you don't read it all at once, but rather read it piecemeal over longer stretches of time, try to apply just a few concepts from the book in your figure-making, and come back to read about other concepts or re-read concepts you learned about a while back. You may find that the same chapter tells you different things if you re-read it after a few months of time have passed.

Even though nearly all of the figures in this book were made with R and ggplot2, I do not see this as an R book. I am talking about general principles of figure preparation. The software used to make the figures is incidental. You can use any plotting software you want to generate the kinds of figures I'm showing here. However, ggplot2 and similar packages make many of the techniques I'm using much simpler than other plotting libraries. Importantly, because this is not an R book, I do not discuss code or programming techniques anywhere in this book. I want you to focus on the concepts and the figures, not on the code. If you are curious how any of the figures were made, you can check out the book's source code at its GitHub repository, https://github.com/clauswilke/dataviz.


## Thoughts on graphing software and figure-preparation pipelines {-}

I have over two decades of experience preparing figures for scientific publications and have made thousands of figures. If there is one constant over these two decades, it's the change in figure preparation pipelines. Every few years, a new plotting library is developed or a new paradigm arises, and large groups of scientists switch over to the hot new toolkit. I have made figures using gnuplot, Xfig, Mathematica, Matlab, matplotlib in python, base R, ggplot2 in R, and possibly others I can't currently remember. My current preferred approach is ggplot2 in R, but I don't expect that I'll continue using it until I retire.

This constant change in software platforms is one of the key reasons why this book is not a programming book and why I have left out all code examples. I want this book to be useful to you regardless of which software you use, and I want it to remain valuable even once everybody has moved on from ggplot2 and uses the next new thing. I realize that this choice may be frustrating to some ggplot2 users who would like to know how I made a given figure. To them I say, read the source code of the book. It is available. Also, in the future I may release a supplementary document focused just on the code.

One thing I have learned over the years is that automation is your friend. I think figures should be autogenerated as part of the data analysis pipeline (which should also be automated), and they should come out of the pipeline ready to be sent to the printer, no manual post-processing needed. I see a lot of trainees autogenerate rough drafts of their figures, which they then import into Illustrator for sprucing up. There are several reasons why this is a bad idea. First, the moment you manually edit a figure, your final figure becomes irreproducible. A third party cannot generate the exact same figure you did. While this may not matter much if all you did was change the font of the axis labels, the lines are blurry, and it's easy to cross over into territory where things are less clear cut. As an example, let's say you want to manually replace cryptic labels with more readable ones. A third party may not be able to verify that the label replacement was appropriate. Second, if you add a lot of manual post-processing to your figure-preparation pipeline then you will be more reluctant to make any changes or redo your work. Thus, you may ignore reasonable requests for change made by collaborators or colleagues, or you may be tempted to re-use an old figure even though you actually regenerated all the data. These are not made-up examples. I've seen all of them play out with real people and real papers. Third, you may yourself forget what exactly you did to prepare a given figure, or you may not be able to generate a future figure on new data that exactly visually matches your earlier figure.

For all the above reasons, interactive plot programs are a bad idea. They inherently force you to manually prepare your figures. In fact, it's probably better to auto-generate a figure draft and spruce it up in Illustrator than make the entire figure by hand in some interactive plot program. Please be aware that Excel is an interactive plot program as well and is not recommended for figure preparation (or data analysis).

One critical component in a book on data visualization is feasibility of the proposed visualizations. It's nice to invent some elegant new way of visualization, but if nobody can easily generate figures using this visualization then there isn't much use to it. For example, when Tufte first proposed sparklines nobody had an easy way of making them. While we need visionaries who move the world forward by pushing the envelope of what's possible, I envision this book to be practical and directly applicable to working data scientists preparing figures for their publications. Therefore, the visualizations I propose in the subsequent chapters can be generated with a few lines of R code via ggplot2 and readily available extension packages. In fact, nearly every figure in this book, with the exception of a few figures in Chapters \@ref(no-3d), \@ref(image-file-formats), and \@ref(choosing-visualization-software), was autogenerated exactly as shown. 

## Acknowledgments {-}

This project would not have been possible without the fantastic work the RStudio team has put into turning the R universe into a first-rate publishing platform. In particular, I have to thank Hadley Wickham for creating **ggplot2**, the plotting software that was used to make all the figures throughout this book. I would also like to thank Yihui Xie for creating R Markdown and for writing the **knitr** and **bookdown** packages. I don't think I would have started this project without these tools ready to go. Writing R Markdown files is fun, and it's easy to collect material and gain momentum. Special thanks go to Achim Zeileis and  Reto Stauffer for **colorspace**, Thomas Lin Pedersen for **ggforce** and **gganimate**, Kamil Slowikowski for **ggrepel**, Edzer Pebesma for **sf**, and Claire McWhite for her work on **colorspace** and **colorblindr** to simulate color-vision deficiency in assembled R figures. 

Several people have provided helpful feedback on draft versions of this book. Most importantly, Mike Loukides, my editor at O'Reilly, and Steve Haroz have both read and commented on every chapter. I also received helpful comments from Carl Bergstrom, Jessica Hullman, Matthew Kay, Edzer Pebesma, Tristan Mahr, Jon Schwabish, and Hadley Wickham. Len Kiefer's blog and Kieran Healy's book and blog postings have provided numerous inspirations for figures to make and datasets to use. A number of people pointed out minor issues or typos, including Thiago Arrais, Malcolm Barrett, Jessica Burnett, Jon Calder, Antônio Pedro Camargo, Daren Card, Kim Cressman, Akos Hajdu, Andrew Kinsman, Will Koehrsen, Alex Lalejini, John Leadley, Katrin Leinweber, Mikel Madina, Claire McWhite, S'busiso Mkhondwane, Jose Nazario, Steve Putman, Maëlle Salmon, Christian Schudoma, James Scott-Brown, Enrico Spinielli, Wouter van der Bijl, and Ron Yurko.

I would also more broadly like to thank all the other contributors to the tidyverse and the R community in general. There truly is an R package for any visualization challenge one may encounter. All these packages have been developed by an extensive community of thousands of data scientists and statisticians, and many of them have in some form contributed to the making of this book.


<!--chapter:end:preface.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(tidyr)
library(lubridate)
library(ggridges)
library(ggforce)
library(treemapify)
```


# (PART\*) Part II: Principles of figure design {-}

# The principle of proportional ink {#proportional-ink}

In many different visualization scenarios, we represent data values by the extent of a graphical element. For example, in a bar plot, we draw bars that begin at 0 and end at the data value they represent. In this case, the data value is not only encoded in the end point of the bar but also in the height or length of the bar. If we drew a bar that started at a different value than 0, then the length of the bar and the bar endpoint would convey contradicting information. Such figures are internally inconsistent, because they show two different values with the same graphical element. Contrast this to a scenario where we visualize the data value with a dot. In this case, the value is only encoded in the location of the dot but not in the size or shape of the dot.

Similar issues will arise whenever we use graphical elements such as bars, rectangles, shaded areas of arbitrary shape, or any other elements that have a clear visual extent which can be either consistent or inconsistent with the data value shown. In all these cases, we need to make sure that there is no inconsistency. This concept has been termed by Bergstrom and West as the *principle of proportional ink* [@BergstromWest2016]:

> **The principle of proportional ink:** The sizes of shaded areas in a visualization need to be proportional to the data values they represent.

(It is common practice to use the word "ink" to refer to any part of a visualization that deviates from the background color. This includes lines, points, shared areas, and text. In this chapter, however, we are talking primarily about shaded areas.) Violations of this principle are quite common, in particular in the popular press and in the world of finance.

## Visualizations along linear axes

We first consider the most common scenario, visualization of amounts along a linear scale. Figure \@ref(fig:hawaii-income-bars-bad) shows the median income in the five counties that make up the state of Hawaii. It is a typical figure one might encounter in a newspaper article. A quick glance at the figure suggests that the county of Hawaii is incredibly poor while the county of Honolulu is much richer than the other counties. However, Figure \@ref(fig:hawaii-income-bars-bad) is quite misleading, because all bars begin at \$50,000 median income. Thus, while the endpoint of each bar correctly represents the actual median income in each county, the bar height represents the extent to which median incomes exceed $50,000, an arbitrary number. And human perception is such that the bar height is the key quantity we perceive when looking at this figure, not the location of the bar endpoint relative to the *y* axis.

(ref:hawaii-income-bars-bad) Median income in the five counties of the state of Hawaii. This figure is misleading, because the *y* axis scale starts at \$50,000 instead of \$0. As a result, the bar heights are not proportional to the values shown, and the income differential between the county of Hawaii and the other four counties appears much bigger than it actually is. Data source: 2015 Five-Year American Community Survey.

```{r hawaii-income-bars-bad, fig.cap = '(ref:hawaii-income-bars-bad)'}
p_income_base <- ggplot(filter(hawaii_income, year == 2015), aes(x = reorder(county, desc(median_income)), y = median_income)) +
  geom_col(fill = "#56B4E9") +
  xlab("county") +
  theme_dviz_hgrid() +
  theme(
    axis.ticks.x = element_blank(),
    plot.margin = margin(3, 7, 3, 1.5)
  )

p_income_bad <- p_income_base + 
  coord_cartesian(xlim = c(0.5, 5.55), ylim = c(50000, 75000), expand = FALSE) +
  scale_y_continuous(
    name = "median income (USD)", 
    breaks = 10000*(5:7),
    labels = function(x) paste0("$", scales::comma(x))
  )

stamp_bad(p_income_bad)
```

An appropriate visualization of these data makes for a less exciting story (Figure \@ref(fig:hawaii-income-bars-good)). While there are differences in median income between the counties, they are nowhere near as big as Figure \@ref(fig:hawaii-income-bars-bad) suggested. Overall, the median incomes in the different counties are somewhat comparable.

(ref:hawaii-income-bars-good) Median income in the five counties of the state of Hawaii. Here, the *y* axis scale starts at \$0 and therefore the relative magnitudes of the median incomes in the five counties are accurately shown. Data source: 2015 Five-Year American Community Survey.

```{r hawaii-income-bars-good, fig.cap = '(ref:hawaii-income-bars-good)'}
p_income_good <- p_income_base + 
  coord_cartesian(xlim = c(0.5, 5.55), ylim = c(0, 78000), expand = FALSE, clip = "off") +
  scale_y_continuous(
    name = "median income (USD)", 
    breaks = 20000*(0:3),
    labels = function(x) paste0("$", scales::comma(x))
  ) +
  theme(axis.line.x = element_blank())

p_income_good
```

```{block type='rmdtip', echo=TRUE}
Bars on a linear scale should always start at 0.
```

Similar visualization problems frequently arise in the visualization of time series, such as those of stock prices. Figure \@ref(fig:fb-stock-drop-bad) suggests a massive collapse in the stock price of Facebook occurred around Nov. 1, 2016. In reality, the price decline was moderate relative to the total price of the stock (Figure \@ref(fig:fb-stock-drop-good)). The *y*-axis range in Figure \@ref(fig:fb-stock-drop-bad) would be questionable even without the shading underneath the curve. But with the shading, the figure becomes particularly problematic. The shading emphasizes the distance from the location of the *x* axis to the specific *y* values shown, and thus it creates the visual impression that the height of the shaded area at a given day represents the stock price of that day. Instead, it only represents the difference in stock price from the baseline, which is $110 in Figure \@ref(fig:fb-stock-drop-bad).

(ref:fb-stock-drop-bad) Stock price of Facebook (FB) from Oct. 22, 2016 to Jan. 21, 2017. This figure seems to imply that the Facebook stock price collapsed around Nov. 1, 2016. However, this is misleading, because the *y* axis starts at $110 instead of $0.

```{r fb-stock-drop-bad, fig.cap = '(ref:fb-stock-drop-bad)'}
df_fb_drop <- filter(tech_stocks, ticker == "FB", date >= ymd("2016-10-22") & date < ymd("2017-01-22"))

fb_drop_bad <- ggplot(df_fb_drop, aes(x=date, height=price - 110, y = 110)) +
  geom_ridgeline(alpha = 0.7) +
  scale_x_date(name = NULL, #name = "day",
               breaks = ymd(c("2016-11-01", "2016-12-01", "2017-01-01")),
               labels = c("Nov 1, 2016", "Dec 1, 2016", "Jan 1, 2017"),
               expand=c(0, 0)) + 
  scale_y_continuous(name="stock price (USD)",
                     limits = c(110, 135),
                     expand=c(0, 0)) + 
  theme_dviz_open() +
  background_grid(major = 'y', minor = 'none') +
  theme(plot.margin = margin(14, 7, 3, 1.5))

stamp_bad(fb_drop_bad)
```

(ref:fb-stock-drop-good) Stock price of Facebook (FB) from Oct. 22, 2016 to Jan. 21, 2017. By showing the stock price on a *y* scale from $0 to $150, this figure more accurately relays the magnitude of the FB price drop around Nov. 1, 2016.

```{r fb-stock-drop-good, fig.cap = '(ref:fb-stock-drop-good)'}
fb_drop_good <- ggplot(df_fb_drop, aes(x=date, height=price, y = 0)) +
  geom_ridgeline(alpha = 0.7) +
  scale_x_date(name = NULL, #name = "day",
               breaks = ymd(c("2016-11-01", "2016-12-01", "2017-01-01")),
               labels = c("Nov 1, 2016", "Dec 1, 2016", "Jan 1, 2017"),
               expand=c(0,0)) + 
  scale_y_continuous(name="stock price (USD)",
                     limits = c(0, 150),
                     expand=c(0,0)) +
  theme_dviz_open() +
  background_grid(major = 'y', minor = 'none') +
  theme(plot.margin = margin(14, 7, 3, 1.5))

stamp_phantom(fb_drop_good)
```

The examples of Figures \@ref(fig:hawaii-income-bars-good) and Figure \@ref(fig:fb-stock-drop-good) could suggest that bars and shaded areas are not useful to represent small changes over time or differences between conditions, since we always have to draw the whole bar or area starting from 0. However, this is not the case. It is perfectly valid to use bars or shaded areas to show differences between conditions, as long as we make it explicit which differences we are showing. For example, we can use bars to visualize the change in median income in Hawaiian counties from 2010 to 2015 (Figure \@ref(fig:hawaii-income-change)). For all counties except Kalawao, this change amounts to less than $5000. (Kalawao is an unusual county, with fewer than 100 inhabitants, and it can experience large swings in median income from a small number of people moving into or out of the county.) And for Hawaii County, the change is negative, i.e., the median income in 2015 was lower than it was in 2010. We represent negative values by drawing bars that go in the opposite direction, i.e., that extend from 0 down rather than up.

(ref:hawaii-income-change) Change in median income in Hawaiian counties from 2010 to 2015. Data source: 2010 and 2015 Five-Year American Community Surveys.

```{r hawaii-income-change, fig.cap = '(ref:hawaii-income-change)'}
hawaii_income_diff <- select(hawaii_income, county, year, median_income) %>%
  spread(year, median_income) %>%
  mutate(income_diff = `2015` - `2010`,
         income_ratio = `2015` / `2010`)

ggplot(hawaii_income_diff, aes(x = reorder(county, desc(filter(hawaii_income, year == 2015)$median_income)),
                               y = income_diff)) + 
  geom_col(fill = "#56B4E9") +
  xlab("county") +
  scale_y_continuous(
    name = "5-year change in median income (USD)", 
    limits = c(-5000, 25000),
    expand = c(0, 0),
    labels = c("-$5,000", "$0", "$5,000", "$10,000", "$15,000", "$20,000", "$25,000")
  ) +
  theme_dviz_hgrid() + 
  theme(axis.ticks.x = element_blank(),
        plot.margin = margin(7, 7, 3, 1.5))
```

Similarly, we can draw the change in Facebook stock price over time as the difference from its temporary high point on Oct. 22, 2016 (Figure \@ref(fig:fb-stock-drop-inverse)). By shading an area that represents the distance from the high point, we are accurately representing the absolute magnitude of the price drop without making any implicit statement about the magnitude of the price drop relative to the total stock price.

(ref:fb-stock-drop-inverse) Loss in Facebook (FB) stock price relative to the price of Oct. 22, 2016. Between Nov. 1, 2016 and Jan. 1, 2017, the price remained approximately \$15 lower than it was at its high point on Oct. 22, 2016. But then the price started to recover in Jan. 2017.

```{r fb-stock-drop-inverse, fig.cap = '(ref:fb-stock-drop-inverse)'}
df_fb_drop2 <- filter(tech_stocks, 
                      ticker == "FB", date >= ymd("2016-10-22") & date < ymd("2017-01-22")) %>%
  mutate(price_drop = price - max(price))

fb_drop_inverse <- ggplot(df_fb_drop2, aes(x=date, height=price_drop, y = 0)) +
  geom_ridgeline(alpha = 0.7, min_height = -50) +
  scale_x_date(name = NULL, #name = "day",
               breaks = ymd(c("2016-11-01", "2016-12-01", "2017-01-01")),
               labels = c("Nov 1, 2016", "Dec 1, 2016", "Jan 1, 2017"),
               expand=c(0, 0)) + 
  scale_y_continuous(name="price loss (USD)",
                     limits = c(-25, 5),
                     expand=c(0, 0)) + 
  theme_dviz_open() +
  background_grid(major = 'y', minor = 'none') +
  theme(plot.margin = margin(7, 7, 3, 1.5))

stamp_phantom(fb_drop_inverse)
```

## Visualizations along logarithmic axes

When we are visualizing data along a linear scale, the areas of bars, rectangles, or other shapes are automatically proportional to the data values. The same is not true if we are using a logarithmic scale, because data values are not linearly spaced along the axis. Therefore, one could argue that, for example, bar graphs on a log scale are inherently flawed. On the flip side, the area of each bar will be proportional to the logarithm of the data value, and thus bar graphs on a log scale satisfy the principle of proportional ink in log-transformed coordinates. In practice, I think neither of these two arguments can resolve whether log-scale bar graphs are appropriate. Instead, the relevant question is whether we want to visualize amounts or ratios.

In Chapter \@ref(coordinate-systems-axes), I have explained that a log scale is the natural scale to visualize ratios, because a unit step along a log scale corresponds to multiplication with or division by a constant factor. In practice, however, log scales are often used not specifically to visualize ratios but rather just because the numbers shown vary over many orders of magnitude. As an example, consider the gross domestic products (GDPs) of countries in Oceania. In 2007, these varied from less than a billion U.S. dollars (USD) to over 300 billion USD (Figure \@ref(fig:oceania-gdp-logbars)). Visualizing these numbers on a linear scale would not work, because the two countries with the largest GDPs (New Zealand and Australia) would dominate the figure.

(ref:oceania-gdp-logbars) GDP in 2007 of countries in Oceania. The lengths of the bars do not accurately reflect the data values shown, since bars start at the arbitrary value of 0.3 billion USD. Data source: Gapminder.

```{r oceania-gdp-logbars, fig.width = 6, fig.asp = 0.5, fig.cap = '(ref:oceania-gdp-logbars)'}
library(gapminder)

df_oceania <- filter(gapminder_unfiltered, year == 2007, continent == "Oceania") %>%
  mutate(GDP = pop*gdpPercap) %>%
  arrange(desc(GDP))

oc_bad <- ggplot(df_oceania, aes(x = reorder(country, -GDP), y = log10(GDP))) + 
  geom_col(fill = "#56B4E9") + 
  scale_y_continuous(breaks = log10(c(3.1e8, 1e9, 3.e9, 1e10, 3.e10, 1e11, 3.e11, 1e12)),
                     labels = c("0.3", "1.0", "3.0", "10", "30", "100", "300", "1000"),
                     name = "GDP (billion USD)") +
  scale_x_discrete(name = NULL) +
  coord_flip(ylim = log10(c(3.1e8, 9.9e11)), expand = FALSE) +
  theme_dviz_vgrid(12, rel_small = 1) +
  theme(axis.ticks.y = element_blank(),
        plot.margin = margin(12, 6, 3, 1.5))

stamp_bad(oc_bad)
```

However, the visualization with bars on a log scale (Figure \@ref(fig:oceania-gdp-logbars)) does not work either. The bars start at an arbitrary value of 0.3 billion USD, and at a minimum the figure suffers from the same problem of Figure \@ref(fig:hawaii-income-bars-bad), that the bar lengths are not representative of the data values. The added difficulty with a log scale, though, is that we cannot simply let the bars start at 0. In Figure \@ref(fig:oceania-gdp-logbars), the value 0 would lie infinitely far to the left. Therefore, we could make our bars arbitrary long by pushing their origin further and further way, see e.g. Figure \@ref(fig:oceania-gdp-logbars-long). This problem always arises when we try to visualize amounts (which is what the GDP values are) on a log scale.

(ref:oceania-gdp-logbars-long) GDP in 2007 of countries in Oceania. The lengths of the bars do not accurately reflect the data values shown, since bars start at the arbitrary value of 10<sup>-9</sup> billion USD. Data source: Gapminder.

```{r oceania-gdp-logbars-long, fig.width = 6, fig.asp = 0.5, fig.cap = '(ref:oceania-gdp-logbars-long)'}
oc_bad2 <- ggplot(df_oceania, aes(x = reorder(country, -GDP), y = log10(GDP))) + 
  geom_col(fill = "#56B4E9") + 
  scale_y_continuous(breaks = 2*(0:6),
                     labels = function(x) label_log10(10^x/1e9),
                     name = "GDP (billion USD)") +
  scale_x_discrete(name = NULL) +
  coord_flip(ylim = log10(c(1, 9.9e11)), expand = FALSE) +
  theme_dviz_vgrid(12, rel_small = 1) +
  theme(axis.ticks.y = element_blank(),
        plot.margin = margin(12, 6, 3, 1.5))

stamp_bad(oc_bad2)
```

For the data of Figure \@ref(fig:oceania-gdp-logbars), I think bars are inappropriate. Instead, we can simply place a dot at the appropriate location along the scale for each country's GDP and avoid the issue of bar lengths altogether (Figure \@ref(fig:oceania-gdp-dots)). Importantly, by placing the country names right next to the dots rather than along the *y* axis, we avoid generating the visual perception of a magnitude conveyed by the distance from the country name to the dot.

(ref:oceania-gdp-dots) GDP in 2007 of countries in Oceania. Data source: Gapminder.

```{r oceania-gdp-dots, fig.width = 6, fig.asp = 0.5, fig.cap = '(ref:oceania-gdp-dots)'}
ggplot(df_oceania, aes(x = reorder(country, -GDP), y = log10(GDP))) + 
  geom_point(size = 3.5, color = "#0072B2") + 
  geom_label(aes(label = country, y = log10(GDP) - .08), hjust = 1, size = 12/.pt,
             family = dviz_font_family,
             fill = "white", alpha = 0.5, label.padding = grid::unit(2, "pt"),
             label.r = grid::unit(0, "pt"), label.size = 0) +
  scale_y_continuous(breaks = log10(c(3e8, 1e9, 3.e9, 1e10, 3.e10, 1e11, 3.e11, 1e12)),
                     labels = c("0.3", "1.0", "3.0", "10", "30", "100", "300", "1000"),
                     name = "              GDP (billion USD)",
                     limits = log10(c(3e7, 9.9e11)),
                     expand = c(0, 0)) +
  scale_x_discrete(name = NULL, breaks = NULL) +
  coord_flip() +
  theme_dviz_vgrid(12, rel_small = 1) +
  theme(plot.margin = margin(12, 6, 3, 1.5))
```

If we want to visualize ratios rather than amounts, however, bars on a log scale are a perfectly good option. In fact, they are preferable over bars on a linear scale in that case. As an example, let's visualize the GDP values of countries in Oceania relative to the GDP of Papua New Guinea. The resulting figure does a good job highlighting the key relationships between the GDPs of the various countries (Figure \@ref(fig:oceania-gdp-relative)). We can see that New Zealand has over eight times the GDP of Papua New Guinea and Australia over 64 times, while Tonga and the Federated States of Micronesia have less than one-sixteenth of the GDP of Papua New Guinea. French Polynesia and New Caledonia are close but have a slightly smaller GDPs than Papua New Guinea does.

(ref:oceania-gdp-relative) GDP in 2007 of countries in Oceania, relative to the GDP of Papua New Guinea. Data source: Gapminder.

```{r oceania-gdp-relative, fig.width = 6, fig.asp = 0.5, fig.cap = '(ref:oceania-gdp-relative)'}
GDP_PNG <- filter(df_oceania, country == "Papua New Guinea")$GDP

df_oceania_ratios <- mutate(df_oceania, gdp_ratio = GDP/GDP_PNG) %>%
  filter(country != "Papua New Guinea")

ggplot(df_oceania_ratios, aes(x = reorder(country, gdp_ratio), y = gdp_ratio)) + 
  geom_col(fill = "#56B4E9") + 
  scale_y_log10(breaks = c(1/16, 1/8, 1/4, 1/2, 1, 2, 4, 8, 16, 32, 64),
                labels = c("1/16", "1/8", "1/4", "1/2", "1", "2", "4", "8", "16", "32", "64"),
                name = "GDP relative to Papua New Guinea",
                limits = c(.055, 70),
                expand = c(0, 0)) +
  scale_x_discrete(name = NULL) +
  coord_flip() +
  theme_dviz_vgrid(12, rel_small = 1) +
  theme(axis.line = element_blank(),
        axis.ticks.length = grid::unit(0, "pt"),
        axis.ticks.y = element_blank(),
        plot.margin = margin(12, 6, 3, 1.5))
```

Figure \@ref(fig:oceania-gdp-relative) also highlights that the natural midpoint of a log scale is 1, with bars representing numbers above 1 going in one direction and bars representing numbers below one going in the other direction. Bars on a log scale represent ratios and must always start at 1, and bars on a linear scale represent amounts and must always start at 0.

```{block type='rmdtip', echo=TRUE}
When bars are drawn on a log scale, they represent ratios and need to be drawn starting from 1, not 0.
```

## Direct area visualizations

All preceding examples visualized data along one linear dimension, so that the data value was encoded both by area and by location along the *x* or *y* axis. In these cases, we can consider the area encoding as incidental and secondary to the location encoding of the data value. Other visualization approaches, however, represent the data value primarily or directly by area, without a corresponding location mapping. The most common one is the pie chart (Figure \@ref(fig:RI-pop-pie)). Even though technically the data values are mapped onto angles, which are represented by location along a circular axis, in practice we are typically not judging the angles of a pie chart. Instead, the dominant visual property we notice is the size of the areas of each pie wedge.

(ref:RI-pop-pie) Number of inhabitants in Rhode Island counties, shown as a pie chart. Both the angle and the area of each pie wedge are proportional to the number of inhabitants in the respective county. Data source: 2010 Decennial U.S. Census.

```{r RI-pop-pie, fig.cap = '(ref:RI-pop-pie)'}
RI_pop <- US_census %>% 
  filter(state == "Rhode Island") %>%
  extract(name, "county", regex = "(.+) County") %>%
  select(county, pop2010) %>%
  mutate(
    label_pop = scales::comma(signif(pop2010, 3)),
    label_comb = paste(county, label_pop, sep = "\n")
  ) %>%
  arrange(desc(pop2010))

RI_pop$county = factor(RI_pop$county, levels = RI_pop$county)

RI_pie <- RI_pop %>%
  mutate(
    total = sum(pop2010),
    end_angle = 2*pi*cumsum(pop2010)/total,      # ending angle for each pie slice
    start_angle = lag(end_angle, default = 0),   # starting angle for each pie slice
    mid_angle = 0.5*(start_angle + end_angle),   # middle of each pie slice, for the text label
    hjust = ifelse(mid_angle>pi, 1, 0),
    vjust = ifelse(mid_angle<pi/2 | mid_angle>3*pi/2, 0, 1)
  )

rpie = 1

RI_pie$rlabel = c(.3, .4, .4, 1.02, 1.02) * rpie
RI_pie$size = c(16, 14, 14, 12, 12)/.pt

plot_RI_pies <- ggplot(RI_pie) + 
  geom_arc_bar(
    aes(
      x0 = 0, y0 = 0, r0 = 0, r = rpie,
      start = start_angle, end = end_angle, fill = county
    ),
    color = "white", size = 0.75
  ) +
  geom_text(
    aes(
      x = rlabel*sin(mid_angle),
      y = rlabel*cos(mid_angle),
      label = label_pop,
      hjust = hjust, vjust = vjust,
      size = size
    ),
    family = dviz_font_family
  ) +
  coord_fixed() +
  scale_x_continuous(limits = c(-1.1, 1.1), expand = c(0, 0), name = NULL, breaks = NULL, labels = NULL) +
  scale_y_continuous(limits = c(-1.1, 1.1), expand = c(0, 0),name = NULL, breaks = NULL, labels = NULL) +
  #scale_fill_brewer(type = "qual", palette = "Pastel1") +
  scale_fill_OkabeIto(darken = -.3) +
  scale_size_identity() +
  guides(fill = guide_legend(override.aes = list(size = 1.))) +
  theme_dviz_open() +
  theme(
    axis.line.x = element_blank(),
    axis.title.x = element_blank(),
    axis.ticks.y = element_blank(),
    legend.title.align = 0.5,
    legend.key.size = grid::unit(25, "pt"),
    legend.spacing.x = grid::unit(2, "pt"),
    legend.spacing.y = grid::unit(2, "pt"),
    plot.margin = margin(7, 7, 3, 1.5)
  )

plot_RI_pies
```

Because the area of each pie wedge is proportional to its angle which is proportional to the data value the wedge represents, pie charts satisfy the principle of proportional ink. However, we perceive the area in a pie chart differently from the same area in a bar plot. The fundamental reason is that human perception primarily judges distances and not areas. Thus, if a data value is encoded entirely as a distance, as is the case with the length of a bar, we perceive it more accurately than when the data value is encoded through a combination of two or more distances that jointly create an area. To see this difference, compare Figure \@ref(fig:RI-pop-pie) to Figure \@ref(fig:RI-pop-bars), which shows the same data as bars. The difference in the number of inhabitants between Providence County and the other counties appears larger in Figure  \@ref(fig:RI-pop-bars) than in Figure \@ref(fig:RI-pop-pie).

(ref:RI-pop-bars) Number of inhabitants in Rhode Island counties, shown as bars. The length of each bar is proportional to the number of inhabitants in the respective county. Data source: 2010 Decennial U.S. Census.

```{r RI-pop-bars, fig.width = 5.5, fig.cap = '(ref:RI-pop-bars)'}
ggplot(RI_pop, aes(x = factor(county, levels = rev(county)), y = pop2010, fill = county)) + 
  geom_col() +
  scale_fill_OkabeIto(darken = -.3, guide = "none") +
  scale_y_continuous(
    expand = c(0, 0),
    breaks = c(0, 2e5, 4e5, 6e5),
    labels = c("0", "200,000", "400,000", "600,000"),
    name = "number of inhabitants"
  ) +
  scale_x_discrete(name = "county") +
  coord_flip() +
  theme_dviz_vgrid() +
  theme(
    #axis.line = element_blank(),
    #axis.ticks.length = grid::unit(0, "pt"),
    axis.ticks.y = element_blank(),
    plot.margin = margin(3, 14, 3, 1.5)
  )

```

The problem that human perception is better at judging distances than at judging areas also arises in treemaps (Figure \@ref(fig:RI-pop-treemap)), which can be thought of as a square versions of pie charts. Again, in comparison to Figure \@ref(fig:RI-pop-bars), the differences in the number of inhabitants among the counties appears less pronounced in Figure \@ref(fig:RI-pop-treemap).

(ref:RI-pop-treemap) Number of inhabitants in Rhode Island counties, shown as a treemap. The area of each rectangle is proportional to the number of inhabitants in the respective county.  Data source: 2010 Decennial U.S. Census.

```{r RI-pop-treemap, fig.width = 4.5, fig.asp = .75, fig.cap = '(ref:RI-pop-treemap)'}
# fix label for Bristol
RI_pop$label_comb[5] <- paste(RI_pop$county[5], RI_pop$label_pop[5])
RI_pop$label_comb = factor(RI_pop$label_comb, levels = RI_pop$label_comb)

p <- ggplot(RI_pop, aes(area = pop2010, fill = county, label = label_comb)) + 
  geom_treemap(
    color = "white", size = .75*.pt,
    fixed = FALSE
  ) + 
  geom_treemap_text(
    family = dviz_font_family,
    colour = "black", place = "centre",
    grow = FALSE,
    fixed = FALSE
  ) +
  scale_fill_OkabeIto(darken = -.3, guide = "none") +
  coord_cartesian(clip = "off")

p
```

<!--chapter:end:proportional_ink.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(lubridate)
library(ggridges)
```


# Redundant coding {#redundant-coding}

In Chapter \@ref(color-pitfalls), we have seen that color cannot always convey information as effectively as we might wish. If we have many different items we want to identify, doing so by color may not work. It will be difficult to match the colors in the plot to the colors in the legend (Figure \@ref(fig:popgrowth-vs-popsize-colored)). And even if we only need to distinguish two to three different items, color may fail if the colored items are very small (Figure \@ref(fig:colors-thin-lines)) and/or the colors look similar for people suffering from color-vision deficiency (Figures \@ref(fig:red-green-cvd-sim) and \@ref(fig:blue-green-cvd-sim)). The general solution in all these scenarios is to use color to enhance the visual appearance of the figure without relying entirely on color to convey key information. I refer to this design principle as *redundant coding*, because it prompts us to encode data redundantly, using multiple different aesthetic dimensions.

## Designing legends with redundant coding 

Scatter plots of several groups of data are frequently designed such that the points representing different groups differ only in their color. As an example, consider Figure \@ref(fig:iris-scatter-one-shape), which shows the sepal width versus the sepal length of three different *Iris* species. (Sepals are the outer leafs of flowers in flowering plants.) The points representing the different species differ in their colors, but otherwise all points look exactly the same. Even though this figure contains only three distinct groups of points, it is difficult to read even for people with normal color vision. The problem arises because the data points for the two species *Iris virginica* and *Iris versicolor* intermingle, and their two respective colors, green and blue, are not particularly distinct from each other.

(ref:iris-scatter-one-shape) Sepal width versus sepal length for three different iris species (*Iris setosa*, *Iris virginica*, and *Iris versicolor*). Each point represents the measurements for one plant sample. A small amount of jitter has been applied to all point positions to prevent overplotting. The figure is labeled "bad" because the *virginica* points in green and the *versicolor* points in blue are difficult to distinguish from each other.

```{r iris-scatter-one-shape, fig.cap = '(ref:iris-scatter-one-shape)'}

breaks = c("setosa", "virginica", "versicolor")
labels = paste0("Iris ", breaks)

iris_scatter_base <- ggplot(iris, aes(x = Sepal.Length, y = Sepal.Width, fill = Species, color = Species)) + 
    scale_color_manual(
      values = darken(c("#E69F00", "#56B4E9", "#009E73"), 0.3),
      breaks = breaks,
      labels = labels,
      name = NULL
    ) +
    scale_fill_manual(
      values = c("#E69F0080", "#56B4E980", "#009E7380"),
      breaks = breaks,
      labels = labels,
      name = NULL
    ) +
    scale_x_continuous(
      limits = c(3.95, 8.2), expand = c(0, 0),
      labels = c("4.0", "5.0", "6.0", "7.0", "8.0"),
      name = "sepal length"
    ) +
    scale_y_continuous(
      limits = c(1.9, 4.6), expand = c(0, 0),
      name = "sepal width"
    )

iris_scatter <- iris_scatter_base +
  geom_point(
    size=2.5, shape=21, stroke = 0.5,
    position = position_jitter(
      width = 0.01 * diff(range(iris$Sepal.Length)),
      height = 0.01 * diff(range(iris$Sepal.Width)),
      seed = 3942
    )
  ) +
  theme_dviz_grid() +
  theme(
    legend.title.align = 0.5,
    legend.text = element_text(face = "italic"),
    legend.spacing.y = unit(3.5, "pt"),
    plot.margin = margin(7, 7, 3, 1.5)
  )

stamp_bad(iris_scatter)
```

Surprisingly, the green and blue points look more distinct for people with red--green color-vision-deficiency (deuteranomaly or protanomaly) than for people with normal color vision (compare Figure \@ref(fig:iris-scatter-one-shape-cvd), top row, to Figure \@ref(fig:iris-scatter-one-shape)). On the other hand, for people with blue--yellow deficiency (tritanomaly) the blue and green points look very similar (Figure \@ref(fig:iris-scatter-one-shape-cvd), bottom left). And if we print out the figure in gray-scale (i.e., we *desaturate* the figure), we cannot distinguish any of the iris species (Figure \@ref(fig:iris-scatter-one-shape-cvd), bottom right).

(ref:iris-scatter-one-shape-cvd) Color-vision-deficiency simulation of Figure \@ref(fig:iris-scatter-one-shape).

```{r iris-scatter-one-shape-cvd, fig.width = 5.5*6/4.2, fig.asp = 0.66, fig.cap = '(ref:iris-scatter-one-shape-cvd)'}
iris_scatter_small <- iris_scatter_base +
  geom_point(
    size=.655*2.5, shape=21, stroke = .655*0.5,
    position = position_jitter(
      width = 0.01 * diff(range(iris$Sepal.Length)),
      height = 0.01 * diff(range(iris$Sepal.Width)),
      seed = 3942
    )
  ) +
  theme_dviz_grid(
    .655*14,
    line_size = .85*.5 # make line size a little bigger than mathematically correct
  ) + 
  theme(
    legend.title.align = 0.5,
    legend.text = element_text(face = "italic"),
    legend.spacing.y = grid::unit(.655*3, "pt"),
    plot.margin = margin(18, 1, 1, 1)
  )

cvd_sim2(iris_scatter_small, label_size = 14, label_y = .98, scale = .95)
```

There are two simple improvements we can make to Figure \@ref(fig:iris-scatter-one-shape) to alleviate these issues. First, we can swap the colors used for *Iris setosa* and *Iris versicolor*, so that the blue is no longer directly next to the green (Figure \@ref(fig:iris-scatter-three-shapes)). Second, we can use three different symbol shapes, so that the points all look different. With these two changes, both the original version of the figure (Figure \@ref(fig:iris-scatter-three-shapes)) and the versions under color-vision-deficiency and in grayscale (Figure \@ref(fig:iris-scatter-three-shapes-cvd)) become legible.

(ref:iris-scatter-three-shapes) Sepal width versus sepal length for three different iris species. Compared to Figure \@ref(fig:iris-scatter-one-shape), we have swapped the colors for *Iris setosa* and *Iris versicolor* and we have given each iris species its own point shape.

```{r iris-scatter-three-shapes, fig.cap = '(ref:iris-scatter-three-shapes)'}
iris_scatter2_base <- ggplot(
  iris, aes(x = Sepal.Length, y = Sepal.Width, shape = Species, fill = Species, color = Species)
) +     
    scale_shape_manual(
      values = c(21, 22, 23),
      breaks = breaks,
      labels = labels,
      name = NULL
    ) +
    scale_color_manual(
      values = darken(c("#56B4E9", "#E69F00", "#009E73"), 0.3),
      breaks = breaks,
      labels = labels,
      name = NULL
    ) +
    scale_fill_manual(
      values = c("#56B4E980", "#E69F0080", "#009E7380"),
      breaks = breaks,
      labels = labels,
      name = NULL
    ) +
    scale_x_continuous(
      limits = c(3.95, 8.2), expand = c(0, 0),
      labels = c("4.0", "5.0", "6.0", "7.0", "8.0"),
      name = "sepal length"
    ) +
    scale_y_continuous(
      limits = c(1.9, 4.6), expand = c(0, 0),
      name = "sepal width"
    )

iris_scatter2 <- iris_scatter2_base +
  geom_point(
    size=2.5, stroke = 0.5,
    position = position_jitter(
      width = 0.01 * diff(range(iris$Sepal.Length)),
      height = 0.01 * diff(range(iris$Sepal.Width)),
      seed = 3942)
  ) +
  theme_dviz_grid() +
  theme(
    legend.title.align = 0.5,
    legend.text = element_text(face = "italic"),
    legend.spacing.y = unit(3.5, "pt"),
    plot.margin = margin(7, 7, 3, 1.5)
  )

iris_scatter2
```


(ref:iris-scatter-three-shapes-cvd) Color-vision-deficiency simulation of Figure \@ref(fig:iris-scatter-three-shapes). Because of the use of different point shapes, even the fully desaturated gray-scale version of the figure is legible.

```{r iris-scatter-three-shapes-cvd, fig.width = 5.5*6/4.2, fig.asp = 0.66, fig.cap = '(ref:iris-scatter-three-shapes-cvd)'}
iris_scatter2_small <- iris_scatter2_base +
  geom_point(
    size=.655*2.5, stroke = .655*0.5,
    position = position_jitter(
      width = 0.01 * diff(range(iris$Sepal.Length)),
      height = 0.01 * diff(range(iris$Sepal.Width)),
      seed = 3942)
  ) +
  theme_dviz_grid(
   .655*14,
    line_size = .85*.5 # make line size a little bigger than mathematically correct
  ) + 
  theme(
    legend.title.align = 0.5,
    legend.text = element_text(face = "italic"),
    legend.spacing.y = grid::unit(.655*3, "pt"),
    plot.margin = margin(18, 1, 1, 1)
  )

cvd_sim2(iris_scatter2_small, label_size = 14, label_y = .98, scale = .95)
```

Changing the point shape is a simple strategy for scatter plots but it doesn't necessarily work for other types of plots. In line plots, we could change the line type (solid, dashed, dotted, etc., see also Figure \@ref(fig:common-aesthetics)), but using dashed or dotted lines often yields sub-optimal results. In particular, dashed or dotted lines usually don't look good unless they are perfectly straight or only gently curved, and in either case they create visual noise. Also, it frequently requires significant mental effort to match different types of dash or dot--dash patterns from the plot to the legend. So what do we do with a visualization such as Figure \@ref(fig:tech-stocks-bad-legend), which uses lines to show the change in stock price over time for four different major tech companies?

(ref:tech-stocks-bad-legend) Stock price over time for four major tech companies. The stock price for each company has been normalized to equal 100 in June 2012. This figure is labeled as "bad" because it takes considerable mental energy to match the company names in the legend to the data curves. Data source: Yahoo Finance

```{r tech-stocks-bad-legend, fig.cap = '(ref:tech-stocks-bad-legend)'}
price_plot_base <- ggplot(tech_stocks, aes(x = date, y = price_indexed, color = ticker)) +
  geom_line(size = 0.66, na.rm = TRUE) +
  scale_color_manual(
    values = c("#000000", "#E69F00", "#56B4E9", "#009E73"),
    name = "",
    breaks = c("GOOG", "AAPL", "FB", "MSFT"),
    labels = c("Alphabet", "Apple", "Facebook", "Microsoft")
  ) +
  scale_x_date(
    name = "year",
    limits = c(ymd("2012-06-01"), ymd("2017-05-31")),
    expand = c(0,0)
  ) + 
  scale_y_continuous(
    name = "stock price, indexed",
    limits = c(0, 560),
    expand = c(0,0)
  )

stamp_bad(
  price_plot_base + 
    theme_dviz_hgrid() + 
    theme(plot.margin = margin(3, 7, 3, 1.5))
)
```


The figure contains four lines representing the stock prices of the four different companies. The lines are color coded using a colorblind-friendly color scale. So it should be relatively straightforward to associate each line with the corresponding company. Yet it is not. The problem here is that the data lines have a clear visual order. The yellow line, representing Facebook, is clearly the highest line, and the black line, representing Apple, is clearly the lowest, with Alphabet and Microsoft in between, in that order. Yet the order of the four companies in the legend is Alphabet, Apple, Facebook, Microsoft (alphabetic order). Thus, the perceived order of the data lines differs from the order of the companies in the legend, and it takes a surprising amount of mental effort to match data lines with company names.

This problem arises commonly with plotting software that autogenerates legends. The plotting software has no concept of the visual order the viewer will perceive. Instead, the software sorts the legend by some other order, most commonly alphabetical. We can fix this problem by manually reordering the entries in the legend so they match the perceived ordering in the data (Figure \@ref(fig:tech-stocks-good-legend)). The result is a figure that makes it much easier to match the legend to the data.

(ref:tech-stocks-good-legend) Stock price over time for four major tech companies. The stock price for each company has been normalized to equal 100 in June 2012. Data source: Yahoo Finance

```{r tech-stocks-good-legend, fig.cap = '(ref:tech-stocks-good-legend)'}
price_plot_base_good <- ggplot(tech_stocks, aes(x = date, y = price_indexed, color = ticker)) +
  scale_color_manual(
    values = c("#000000", "#E69F00", "#56B4E9", "#009E73"),
    name = "",
    breaks = c("FB", "GOOG", "MSFT", "AAPL"),
    labels = c("Facebook", "Alphabet", "Microsoft", "Apple")
  ) +
  scale_x_date(
    name = "year",
    limits = c(ymd("2012-06-01"), ymd("2017-05-31")),
    expand = c(0,0)
  ) + 
  scale_y_continuous(
    name = "stock price, indexed",
    limits = c(0, 560),
    expand = c(0,0)
  )

price_plot_base_good +
  geom_line(size = 0.66, na.rm = TRUE) +
  theme_dviz_hgrid() + 
  theme(plot.margin = margin(3, 7, 3, 1.5))
```

```{block type='rmdtip', echo=TRUE}
If there is a clear visual ordering in your data, make sure to match it in the legend.
```


Matching the legend order to the data order is always helpful, but the benefits are particularly obvious under color-vision deficiency simulation (Figure \@ref(fig:tech-stocks-good-legend-cvd)). For example, it helps in the tritanomaly version of the figure, where the blue and the green become difficult to distinguish (Figure \@ref(fig:tech-stocks-good-legend-cvd), bottom left). It also helps in the grayscale version (Figure \@ref(fig:tech-stocks-good-legend-cvd), bottom right). Even though the two colors for Facebook and Alphabet have virtually the same gray value, we can see that Microsoft and Apple are represented by darker colors and take the bottom two spots. Therefore, we correctly assume that the highest line corresponds to Facebook and the second-highest line to Alphabet.

(ref:tech-stocks-good-legend-cvd) Color-vision-deficiency simulation of Figure \@ref(fig:tech-stocks-good-legend).

```{r tech-stocks-good-legend-cvd, fig.width = 5.5*6/4.2, fig.asp = 0.66, fig.cap = '(ref:tech-stocks-good-legend-cvd)'}
price_plot_good_small <-
  price_plot_base_good + 
  geom_line(
    size = .85*0.66, # make line size a little bigger than mathematically correct
    na.rm = TRUE
  ) +
  theme_dviz_hgrid(
    .655*14,
    line_size = .85*.5 # make line size a little bigger than mathematically correct
  ) + 
  theme(plot.margin = margin(18, 1, 1, 1))

cvd_sim2(price_plot_good_small, label_size = 14, label_y = .98, scale = .95)
```


## Designing figures without legends

Even though legend legibility can be improved by encoding data redundantly, in multiple aesthetics, legends always put an extra mental burden on the reader. In reading a legend, the reader needs to pick up information in one part of the visualization and then transfer it over to a different part. We can typically make our readers' lives easier if we eliminate the legend altogether. Eliminating the legend does not mean, however, that we simply not provide one and instead write sentences such as "The yellow dots represent *Iris versicolor*" in the figure caption. Eliminating the legend means that we design the figure in such a way that it is immediately obvious what the various graphical elements represent, even if no explicit legend is present.

The general strategy we can employ is called *direct labeling*, whereby we place appropriate text labels or other visual elements that serve as guideposts to the rest of the figure. We have previously encountered direct labeling in Chapter \@ref(color-pitfalls) (Figure \@ref(fig:popgrowth-vs-popsize-bw)), as an alternative to drawing a legend with over 50 distinct colors. To apply the direct labeling concept to the stock-price figure, we place the name of each company right next to the end of its respective data line (Figure \@ref(fig:tech-stocks-good-no-legend)). 

(ref:tech-stocks-good-no-legend) Stock price over time for four major tech companies. The stock price for each company has been normalized to equal 100 in June 2012. Data source: Yahoo Finance

```{r tech-stocks-good-no-legend, fig.cap = '(ref:tech-stocks-good-no-legend)'}
price_plot <- price_plot_base_good + 
  geom_line(size = 0.66, na.rm = TRUE) +
  theme_dviz_hgrid()

yann <- axis_canvas(price_plot, axis = "y") +
  geom_text(
    data = filter(tech_stocks, date == "2017-06-02"),
    aes(y = price_indexed, label = paste0(" ", company)),
    family = dviz_font_family,
    x = 0, hjust = 0, size = 12/.pt
  )

price_plot_ann <- insert_yaxis_grob(
  price_plot + theme(legend.position = "none"),
  yann,
  width = grid::unit(0.3, "null")
)
ggdraw(price_plot_ann)
```


```{block type='rmdtip', echo=TRUE}
Whenever possible, design your figures so they don't need a legend.
```

We can also apply the direct labeling concept to the iris data from the beginning of this chapter, specifically Figure \@ref(fig:iris-scatter-three-shapes). Because it is a scatter plot of many points that separate into three different groups, we need to direct label the groups rather than the individual points. One solution is to draw ellipses that enclose the majority of the points and then label the ellipses (Figure \@ref(fig:iris-scatter-with-ellipses)).

(ref:iris-scatter-with-ellipses) Sepal width versus sepal length for three different iris species. I have removed the background grid from this figure because otherwise the figure was becoming too busy.

```{r iris-scatter-with-ellipses, fig.width = 4.6, fig.asp = 0.8, fig.cap = '(ref:iris-scatter-with-ellipses)'}
label_df <- data.frame(
  Species = c("setosa", "virginica", "versicolor"),
  label = c("Iris setosa", "Iris virginica", "Iris versicolor"),
  Sepal.Width = c(4.2, 3.76, 2.08),
  Sepal.Length = c(5.7, 7, 5.1),
  hjust = c(0, 0.5, 0),
  vjust = c(0, 0.5, 1))

iris_scatter3 <- ggplot(iris, 
      aes(
        x = Sepal.Length,
        y = Sepal.Width,
        color = Species
      )
    ) + 
    geom_point(
      aes(shape = Species, fill = Species),
      size = 2.5,
      position = position_jitter(
        width = 0.01 * diff(range(iris$Sepal.Length)),
        height = 0.01 * diff(range(iris$Sepal.Width)),
        seed = 3942)
    ) +
    stat_ellipse(size = 0.5) +
    geom_text(
      data = label_df,
      aes(
        x = Sepal.Length, y = Sepal.Width, label = label, color = Species,
        hjust = hjust, vjust = vjust
      ),
      family = dviz_font_family,
      size = 14/.pt,
      fontface = "italic",
      inherit.aes = FALSE
    ) +
    scale_shape_manual(
      values = c(21, 22, 23),
      breaks = breaks,
      name = NULL
    ) +
    scale_fill_manual(
      values = c("#56B4E980", "#E69F0080", "#009E7380"),
      breaks = breaks,
      name = NULL
    ) +
    scale_color_manual(
      values = darken(c("#56B4E9", "#E69F00", "#009E73"), 0.3),
      breaks = breaks,
      name = NULL
    ) +
    guides(fill = "none", color = "none", shape = "none") +
    scale_x_continuous(
      limits = c(3.95, 8.2), expand = c(0, 0),
      labels = c("4.0", "5.0", "6.0", "7.0", "8.0"),
      name = "sepal length"
    ) +
    scale_y_continuous(
      limits = c(1.9, 4.6), expand = c(0, 0),
      name = "sepal width"
    ) +
    theme_dviz_open()

iris_scatter3
```

For density plots, we can similarly direct-label the curves rather than providing a color-coded legend (Figure \@ref(fig:iris-densities-direct-label)). In both Figures \@ref(fig:iris-scatter-with-ellipses) and \@ref(fig:iris-densities-direct-label), I have colored the text labels in the same colors as the data. Colored labels can greatly enhance the direct labeling effect, but they can also turn out very poorly. If the text labels are printed in a color that is too light, then the labels become difficult to read. And, because text consists of very thin lines, colored text often appears to be lighter than an adjacent filled area of the same color. I generally circumvent these issues by using two different shades of each color, a light one for filled areas and a dark one for lines, outlines, and text. If you carefully inspect Figure \@ref(fig:iris-scatter-with-ellipses) or \@ref(fig:iris-densities-direct-label), you will see how each data point or shaded area is filled with a light color and has an outline drawn in a darker color of the same hue. And the text labels are drawn in the same darker colors. 

(ref:iris-densities-direct-label) Density estimates of the sepal lengths of three different iris species. Each density estimate is directly labeled with the respective species name.

```{r iris-densities-direct-label, fig.cap = '(ref:iris-densities-direct-label)'}
# compute densities for sepal lengths
iris_dens <- group_by(iris, Species) %>%
  do(ggplot2:::compute_density(.$Sepal.Length, NULL)) %>%
  rename(Sepal.Length = x)

# get the maximum values
iris_max <- filter(iris_dens, density == max(density)) %>%
  ungroup() %>%
  mutate(
    hjust = c(0, 0.4, 0),
    vjust = c(1, 0, 1),
    nudge_x = c(0.11, 0, 0.24),
    nudge_y = c(-0.02, 0.02, -0.02),
    label = paste0("Iris ", Species)
  )

iris_p <- ggplot(iris_dens, aes(x = Sepal.Length, y = density, fill = Species, color = Species)) + 
  geom_density_line(stat = "identity") +
  geom_text(
    data = iris_max,
    aes(
      label = label, hjust = hjust, vjust = vjust, color = Species,
      x = Sepal.Length + nudge_x, 
      y = density + nudge_y
    ), 
    family = dviz_font_family,
    size = 14/.pt,
    inherit.aes = FALSE,
    fontface = "italic"
  ) +
  scale_color_manual(
    values = darken(c("#56B4E9", "#E69F00", "#009E73"), 0.3),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_fill_manual(
    values = c("#56B4E950", "#E69F0050", "#009E7350"),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_x_continuous(expand = c(0, 0), name = "sepal length") +
  scale_y_continuous(limits = c(0, 1.5), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(6.5, 1.5, 3.5, 1.5)
  )
  
iris_p
```

We can also use density plots such as the one in Figure \@ref(fig:iris-densities-direct-label) as a legend replacement, by placing the density plots into the margins of a scatter plot (Figure \@ref(fig:iris-scatter-dens)). This allows us to direct-label the marginal density plots rather than the central scatter plot and hence results in a figure that is somewhat less cluttered than Figure \@ref(fig:iris-scatter-with-ellipses) with directly-labeled ellipses.

(ref:iris-scatter-dens) Sepal width versus sepal length for three different iris species, with marginal density estimates of each variable for each species.

```{r iris-scatter-dens, fig.width = 5*6/4.2, fig.asp=0.85, fig.cap = '(ref:iris-scatter-dens)'}
# compute densities for sepal lengths
iris_dens2 <- group_by(iris, Species) %>%
  do(ggplot2:::compute_density(.$Sepal.Width, NULL)) %>%
  rename(Sepal.Width = x)

dens_limit <- max(iris_dens$density, iris_dens2$density) * 1.05 # upper limit of density curves

# we need different hjust and nudge values here
iris_max <- 
  iris_max %>%
  mutate(
    hjust = c(1, 0.4, 0),
    vjust = c(1, 0, 1),
    nudge_x = c(-0.18, 0, 0.47),
    nudge_y = c(-0.01, 0.06, 0.03),
    label = paste0("Iris ", Species)
  )

xdens <- axis_canvas(iris_scatter2, axis = "x") +
  geom_density_line(
    data=iris_dens,
    aes(x = Sepal.Length, y = density, fill = Species, color = Species),
    stat = "identity", size = .2
  ) +
  geom_text(
    data = iris_max,
    aes(
      label = label, hjust = hjust, vjust = vjust, color = Species,
      x = Sepal.Length + nudge_x, 
      y = density + nudge_y
    ),
    family = dviz_font_family, 
    ize = 12/.pt, 
    #color = "black", inherit.aes = FALSE,
    fontface = "italic"
  ) +
  scale_color_manual(
    values = darken(c("#56B4E9", "#E69F00", "#009E73"), 0.3),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_fill_manual(
    values = c("#56B4E950", "#E69F0050", "#009E7350"),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_y_continuous(limits = c(0, dens_limit), expand = c(0, 0))

ydens <- axis_canvas(iris_scatter2, axis = "y", coord_flip = TRUE) +
  geom_density_line(
    data = iris_dens2,
    aes(x = Sepal.Width, y = density, fill = Species, color = Species),
    stat = "identity", size = .2
  )  +
  scale_color_manual(
    values = darken(c("#56B4E9", "#E69F00", "#009E73"), 0.3),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_fill_manual(
    values = c("#56B4E950", "#E69F0050", "#009E7350"),
    breaks = c("virginica", "versicolor", "setosa"),
    guide = "none"
  ) +
  scale_y_continuous(limits = c(0, dens_limit), expand = c(0, 0)) +
  coord_flip()

p1 <- insert_xaxis_grob(
  iris_scatter2 + theme(legend.position = "none"),
  xdens,
  grid::unit(3*14, "pt"), position = "top"
)
p2 <- insert_yaxis_grob(p1, ydens, grid::unit(3*14, "pt"), position = "right")

ggdraw(p2)
```

And finally, whenever we encode a single variable in multiple aesthetics, we don't normally want multiple separate legends for the different aesthetics. Instead, there should be only a single legend-like visual element that conveys all mappings at once. In the case where we map the same variable onto a position along a major axis and onto color, this implies that the reference color bar should run along and be integrated into the same axis. Figure \@ref(fig:temp-ridgeline-colorbar) shows a case where we map temperature to both a position along the *x* axis and onto color, and where we therefore have integrated the color legend into the *x* axis.

(ref:temp-ridgeline-colorbar) Temperatures in Lincoln, Nebraska, in 2016. This figure is a variation of Figure \@ref(fig:temp-ridgeline). Temperature is now shown both by location along the *x* axis and by color, and a color bar along the *x* axis visualizes the scale that converts temperatures into colors.

```{r temp-ridgeline-colorbar, fig.asp = 0.652, fig.cap = '(ref:temp-ridgeline-colorbar)'}
bandwidth <- 3.4

lincoln_base <- ggplot(lincoln_weather, aes(x = `Mean Temperature [F]`, y = `Month`, fill = ..x..)) +
  geom_density_ridges_gradient(
    scale = 3, rel_min_height = 0.01, bandwidth = bandwidth,
    color = "black", size = 0.25
  ) +
  scale_x_continuous(
    name = "mean temperature (°F)",
    expand = c(0, 0), breaks = c(0, 25, 50, 75), labels = NULL
  ) +
  scale_y_discrete(name = NULL, expand = c(0, .2, 0, 2.6)) +
  scale_fill_continuous_sequential(
    palette = "Heat",
    l1 = 20, l2 = 100, c2 = 0,
    rev = FALSE
  ) +
  guides(fill = "none") +
  theme_dviz_grid() +
  theme(
    axis.text.y = element_text(vjust = 0),
    plot.margin = margin(3, 7, 3, 1.5)
  )

# x axis labels
temps <- data.frame(temp = c(0, 25, 50, 75))

# calculate corrected color ranges
# stat_joy uses the +/- 3*bandwidth calculation internally
tmin <- min(lincoln_weather$`Mean Temperature [F]`) - 3*bandwidth
tmax <- max(lincoln_weather$`Mean Temperature [F]`) + 3*bandwidth

xax <- axis_canvas(lincoln_base, axis = "x", ylim = c(0, 2)) +
  geom_ridgeline_gradient(
    data = data.frame(temp = seq(tmin, tmax, length.out = 100)),
    aes(x = temp, y = 1.1, height = .9, fill = temp),
    color = "transparent"
  ) +
  geom_text(
    data = temps, aes(x = temp, label = temp),
    color = "black", family = dviz_font_family,
    y = 0.9, hjust = 0.5, vjust = 1, size = 14/.pt
  ) +
  scale_fill_continuous_sequential(
    palette = "Heat",
    l1 = 20, l2 = 100, c2 = 0,
    rev = FALSE
  )

lincoln_final <- insert_xaxis_grob(lincoln_base, xax, position = "bottom", height = unit(0.1, "null"))

ggdraw(lincoln_final)
```

<!--chapter:end:redundant_coding.Rmd-->

`r if (knitr:::is_html_output()) '# References {-}'`

<!--chapter:end:references.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")
```

# Use larger axis labels {#small-axis-labels}

If you take away only one single lesson from this book, make it this one: Pay attention to your axis labels, axis tick labels, and other assorted plot annotations. Chances are they are too small. In my experience, nearly all plot libraries and graphing softwares have poor defaults. If you use the default values, you're almost certainly making a poor choice.

For example, consider Figure \@ref(fig:Aus-athletes-small). I see figures like this all the time. The axis labels, axis tick labels, and legend labels are all incredibly small. We can barely see them, and we may have to zoom into the page to read the annotations in the legend.

(ref:Aus-athletes-small) Percent body fat versus height in professional male Australian athletes. (Each point represents one athlete.) This figure suffers from the common affliction that the text elements are way too small and are barely legible. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-small, fig.width = 6, fig.cap='(ref:Aus-athletes-small)'}
male_Aus <- filter(Aus_athletes, sex == "m") %>%
    filter(sport %in% c("basketball", "field", "swimming", "track (400m)",
                        "track (sprint)", "water polo")) %>%
    mutate(sport = case_when(sport == "track (400m)" ~ "track",
                             sport == "track (sprint)" ~ "track",
                             TRUE ~ sport))

male_Aus$sport <- factor(male_Aus$sport,
                         levels = c("field", "water polo", "basketball", "swimming", "track"))

p <- ggplot(male_Aus, aes(x = height, y = pcBfat, color = sport, fill = sport, shape = sport)) +
    scale_shape_manual(values = 21:25) +
    scale_color_OkabeIto(order = c(2, 1, 3, 4, 5), darken = 0.3) +
    scale_fill_OkabeIto(order = c(2, 1, 3, 4, 5), darken = 0.1, alpha = 0.7) +
    ylab("% body fat") +
    xlab("height (cm)")

stamp_bad(p + geom_point(size = 1.5) + 
            theme_dviz_grid(7) + 
            theme(plot.margin = margin(7, 7, 3, 1.5)))
```

A somewhat better version of this figure is shown as Figure \@ref(fig:Aus-athletes-ugly). I think the fonts are still too small, and that's why I have labeled the figure as ugly. However, we are moving in the right direction. This figure might be passable under some circumstances. My main criticism here is not so much that the labels aren't legible as that the figure is not balanced; the text elements are too small compared to the rest of the figure.

(ref:Aus-athletes-ugly) Percent body fat versus height in male athletes. This figure is an improvement over Figure \@ref(fig:Aus-athletes-small), but the text elements remain too small and the figure is not balanced. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-ugly, fig.width = 6, fig.cap='(ref:Aus-athletes-ugly)'}
stamp_ugly(p + geom_point(size = 1.5) +
             theme_dviz_grid(10) + theme(plot.margin = margin(7, 7, 3, 1.5)))
```


The next figure uses the default settings I'm applying throughout this book. I think it is well balanced, the text is clearly visible, and it fits with the overall size of the figure. 

(ref:Aus-athletes-good) Percent body fat versus height in male athletes. All figure elements are appropriately scaled. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-good, fig.width = 6, fig.cap='(ref:Aus-athletes-good)'}
p + geom_point(size = 2.25) +
  theme_dviz_grid() + theme(plot.margin = margin(7, 7, 3, 1.5))
```

Importantly, we can overdo it and make the labels too big (Figure \@ref(fig:Aus-athletes-big-ugly)). Sometimes we need big labels, for example if the figure is meant to be reduced in size, but the various elements of the figure (in particular, label text and plot symbols) need to fit together. In Figure \@ref(fig:Aus-athletes-big-ugly), the points used to visualize the data are too small relative to the text.  Once we fix this issue, the figure becomes acceptable again (Figure \@ref(fig:Aus-athletes-big-good)).

(ref:Aus-athletes-big-ugly) Percent body fat versus height in male athletes. The text elements are fairly large, and their size may be appropriate if the figure is meant to be reproduced at a very small scale. However, the figure overall is not balanced; the points are too small relative to the text elements. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-big-ugly, fig.width = 6, fig.cap='(ref:Aus-athletes-big-ugly)'}
stamp_ugly(p + geom_point(size = 2.25) +
             theme_dviz_grid(19) + theme(plot.margin = margin(7, 7, 3, 1.5)))
```


(ref:Aus-athletes-big-good) Percent body fat versus height in male athletes. All figure elements are sized such that the figure is balanced and can be reproduced at a small scale. Data source: @Telford-Cunningham-1991

```{r Aus-athletes-big-good, fig.width=6, fig.cap='(ref:Aus-athletes-big-good)'}
p + geom_point(size = 3.5) +
  theme_dviz_grid(19) + theme(plot.margin = margin(7, 7, 3, 1.5))
```

You may look at Figure \@ref(fig:Aus-athletes-big-good) and find everything too big. However, keep in mind that it is meant to be scaled down. Scale the figure down so that it is only an inch or two in width, and it looks just fine. In fact, at that scaling this is the only figure in this chapter that looks good.

```{block type='rmdtip', echo=TRUE}
Always look at scaled-down versions of your figures to make sure the axis labels are appropriately sized.
```

I think there is a simple psychological reason for why we routinely make figures whose axis labels are too small, and it relates to large, high-resolution computer monitors. We routinely preview figures on the computer screen, and often we do so while the figure takes up a large amount of space on the screen. In this viewing mode, even comparatively small text seems perfectly fine and legible, and large text can seem awkward and overpowering. In fact, if you take the first figure from this chapter and magnify it to the point where it fills your entire screen, you will likely think that it looks just fine. The solution is to always make sure that you look at your figures at a realistic print size. You can either zoom out so they are only three to five inches in width on your screen, or you can go to the other side of your room and check whether the figure still looks good from a substantial distance.

<!--chapter:end:small_axis_labels.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")
library(tidyverse)
library(lubridate)
library(patchwork)
library(ggforce)
library(ggrepel)
library(ggridges)
library(ggthemes)
library(ggmap)
library(gridExtra)
library(treemapify)
library(hexbin)
library(magick)
library(plot3D)
library(geofacet)
library(ggspatial)
library(rgeos)
library(maptools)
library(sf)
library(statebins)
library(maps)
library(tidybayes)
library(broom)
library(mgcv)
library(emmeans)
library(ungeviz)
library(gganimate)
library(RColorBrewer)
library(gapminder)
library(nycflights13)
```

# Technical notes {-}

The entire book was written in R Markdown, using the **bookdown**, **rmarkdown**, and **knitr** packages. All figures were made with **ggplot2**, with the help of add-on packages **cowplot**, **egg**, **ggforce**, **ggrepel**, **ggridges**, **sf**, and **treemapify**. Color manipulations were done with the **colorspace** and **colorblindr** packages. For many of these packages, the current development version is required to compile all parts of the book.

The source code for the book is available here: https://github.com/clauswilke/dataviz. The book also requires a supporting R package, **dviz.supp**, whose code is available here: https://github.com/clauswilke/dviz.supp.

The book was last compiled using the following environment:
```{r}
sessionInfo()
```

<!--chapter:end:technical_notes.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(lubridate)
library(forcats)
library(tidyr)
library(ggtextures)
```

# Telling a story and making a point {#telling-a-story}

Most data visualization is done for the purpose of communication. We have an insight about a dataset, and we have a potential audience, and we would like to convey our insight to our audience. To communicate our insight successfully, we will have to present the audience with a clear and exciting story. The need for a story may seem disturbing to scientists and engineers, who may equate it with making things up, putting a spin on things, or overselling results. However, this perspective misses the important role that stories play in reasoning and memory. We get excited when we hear a good story, and we get bored when the story is bad or when there is none. Moreover, any communication creates a story in the audience's minds. If we don't provide a clear story ourselves, then our audience will make one up. In the best-case scenario, the story they make up is reasonably close to our own view of the material presented. However, it can be and often is much worse. The made-up story could be "this is boring," "the author is wrong," or "the author is incompetent."

Your goal in telling a story should be to use facts and logical reasoning to get your audience interested and excited. Let me tell you a story about the theoretical physicist Stephen Hawking. He was diagnosed with motor neuron disease at age 21---one year into his PhD---and was given two years to live. Hawking did not accept this predicament and started pouring all his energy into doing science. Hawking ended up living to be 76, became one of the most influential physicists of his time, and did all of his seminal work while being severely disabled. I'd argue that this is a compelling story. It's also entirely fact-based and true.

## What is a story?

Before we can discuss strategies for turning visualizations into stories, we need to understand what a story actually is. A story is a set of observations, facts, or events, true or invented, that are presented in a specific order such that they create an emotional reaction in the audience. The emotional reaction is created through the build-up of tension at the beginning of the story followed by some type of resolution towards the end of the story. We refer to the flow from tension to resolution also as the story arc, and every good story has a clear, identifiable arc.

Experienced writers know that there are standard patterns for storytelling that  resonate with how humans think. For example, we can tell a story using the Opening--Challenge--Action--Resolution format. In fact, this is the format I used for the Hawking story in the previous subsection. I opened the story by introducing the topic, the physicist Stephen Hawking. Next I presented the challenge, the diagnosis of motor neuron disease at age 21. Then came the action, his fierce dedication to science. Finally I presented the resolution, that Hawking led a long and successful life and ended up becoming one of the most influential physicists of his time. Other story formats are also commonly used. Newspaper articles frequently follow the Lead--Development--Resolution format or, even shorter, just Lead--Development, where the lead gives away the main point up front and the subsequent material provides further details. If we wanted to tell the Hawking story in this format, we might start out with a sentence such as "The influential physicist Stephen Hawking, who revolutionized our understanding of black holes and of cosmology, outlived his doctors' prognosis by 53 years and did all of his most influential work while being severely disabled." This is the lead. In the development, we could follow up with a more in-depth description of Hawking's life, illness, and devotion to science. Yet another format is Action--Background--Development--Climax--Ending, which develops the story a little more rapidly than Opening--Challenge--Action--Resolution but not as rapidly as Lead--Development. In this format, we might open with a sentence such as "The young Stephen Hawking, facing a debilitating disability and the prospect of an early death, decided to pour all his efforts into his science, determined to make his mark while he still could." The purpose of this format is to draw in the audience and to create an emotional connection early on, but without immediately giving away the final resolution.

My goal in this chapter is not to describe these standard forms of story telling in more detail. There are excellent resources that cover this material. For scientists and analysts, I particularly recommend @Schimel2011. Instead, I want to discuss how we can bring data visualizations into the story arc. Most importantly, we need to realize that a single (static) visualization will rarely tell an entire story. A visualization may illustrate the opening, the challenge, the action, or the resolution, but it is unlikely to convey all these parts of the story at once. To tell a complete story, we will usually need multiple visualizations. For example, when giving a presentation, we may first show some background or motivational material, then a figure that creates a challenge, and eventually some other figure that provides the resolution. Likewise, in a research paper, we may present a sequence of figures that jointly create a convincing story arc. It is, however, also possible to condense an entire story arc into a single figure. Such a figure must contain a challenge and a resolution at the same time, and it is comparable to a story arc that starts with a lead.

To provide a concrete example of incorporating figures into stories, I will now tell a story on the basis of two figures. The first creates the challenge and the second serves as the resolution. The context of my story is the growth of preprints in the biological sciences (see also Chapter \@ref(time-series)). Preprints are manuscripts in draft form that scientists share with their colleagues before formal peer review and official publication. Scientists have been sharing manuscript drafts for as long as scientific manuscripts have existed. However, in the early 1990s, with the advent of the internet, physicists realized that it was much more efficient to store and distribute manuscript drafts in a central repository. They invented the preprint server, a web server where scientists can upload, download, and search for manuscript drafts.

The preprint server physicists developed and still use today is called arXiv.org. Shortly after it was established, arXiv.org started to branch out and become popular in related quantitative fields, including mathematics, astronomy, computer science, statistics, quantitative finance, and quantitative biology. Here, I am interested in the preprint submissions to the quantitative biology (q-bio) section of arXiv.org. The number of submissions per month grew exponentially from 2007 to late 2013, but then the growth suddenly stopped (Figure \@ref(fig:q-bio-monthly-growth)). Something must have happened in late 2013 that radically changed the landscape in preprint submissions for quantitative biology. What caused this drastic change in submission growth?

(ref:q-bio-monthly-growth) Growth in monthly submissions to the quantitative biology (q-bio) section of the preprint server arXiv.org. A sharp transition in the rate of growth can be seen around 2014. While growth was rapid up to 2014, almost no growth occurred from 2014 to 2018. Note that the *y* axis is logarithmic, so a linear increase in *y* corresponds to exponential growth in preprint submissions. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r q-bio-monthly-growth, fig.cap = '(ref:q-bio-monthly-growth)'}
preprint_growth %>% filter(archive %in% c("bioRxiv", "arXiv q-bio")) %>%
filter(count > 0) -> df_preprints

df_final <- filter(df_preprints, date == max(date))
df_final_qbio <- filter(df_final, archive == "arXiv q-bio")

ggplot(
  filter(df_preprints, archive == "arXiv q-bio"),
  aes(date, count, color = archive)
) + 
  geom_line(size = 0.75) +
  scale_y_log10(
    limits = c(40, 400),
    breaks = c(50, 100, 200, 300),
    expand = c(0, 0),
    name = "preprints / month",
    sec.axis = dup_axis(
      breaks = df_final_qbio$count,
      labels = df_final_qbio$archive,
      name = NULL
    )
  ) + 
  scale_x_date(name = "year", expand = c(0, 0)) +
  scale_color_manual(values = c("#D55E00", "#0072B2"), guide = "none") +
  theme_dviz_grid() +
  theme(
    axis.ticks.y.right = element_blank(),
    axis.text.y.right = element_text(size = 14, margin = margin(0, 0, 0, 0))
  )
```

I will argue that late 2013 marks the point in time when preprints took off in biology, and ironically this caused the q-bio archive to slow its growth. In November 2013, the biology-specific preprint server bioRxiv was launched by Cold Spring Harbor Laboratory (CSHL) Press. CSHL Press is a publisher that is highly respected among biologists. The backing of CSHL Press helped tremendously with the acceptance of preprints in general and bioRxiv in particular among biologists. The same biologists that would have been quite suspicious of arXiv.org were much more comfortable with bioRxiv. As a result bioRxiv quickly gained acceptance among biologists, to a degree that arXiv had never managed. In fact, soon after its launch, bioRxiv started experiencing rapid, exponential growth in monthly submissions, and the slowdown in q-bio submissions exactly coincides with the start of this exponential growth in bioRxiv (Figure \@ref(fig:q-bio-bioRxiv-monthly-growth)). It appears to be the case that many quantitative biologists who otherwise might have deposited a preprint with q-bio decided to deposit it with bioRxiv instead.

(ref:q-bio-bioRxiv-monthly-growth) The leveling off of submission growth to q-bio coincided with the introduction of the bioRxiv server. Shown are the growth in monthly submissions to the q-bio section of the general-purpose preprint server arxiv.org and to the dedicated biology preprint server bioRxiv. The bioRxiv server went live in November 2013, and its submission rate has grown exponentially since. It seems likely that many scientists who otherwise would have submitted preprints to q-bio chose to submit to bioRxiv instead. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r q-bio-bioRxiv-monthly-growth, fig.cap = '(ref:q-bio-bioRxiv-monthly-growth)'}
ggplot(df_preprints, aes(date, count, color = archive)) + 
  geom_line(size = 0.75) +
  scale_y_log10(
    limits = c(29, 1600),
    breaks = c(30, 100, 300, 1000),
    expand = c(0, 0),
    name = "preprints / month",
    sec.axis = dup_axis(
      breaks = df_final$count,
      labels = df_final$archive,
      name = NULL
    )
  ) + 
  scale_x_date(name = "year", expand = c(0, 0)) +
  scale_color_manual(values = c("#D55E00", "#0072B2"), guide = "none") +
  theme_dviz_grid() +
  theme(
    axis.ticks.y.right = element_blank(),
    axis.text.y.right = element_text(size = 14, margin = margin(0, 0, 0, 0))
  )
```

This is my story about preprints in biology. I purposefully told it with two figures, even though the first (Figure \@ref(fig:q-bio-monthly-growth)) is fully contained within the second (Figure \@ref(fig:q-bio-bioRxiv-monthly-growth)). I think this story has the strongest impact when broken into two pieces, and this is how I would present it in a talk. However, Figure \@ref(fig:q-bio-bioRxiv-monthly-growth) alone can be used to tell the entire story, and the single-figure version might be more suitable to a medium where the audience can be expected to have short attention span, such as in a social media post. 

## Make a figure for the generals

For the remainder of this chapter, I will discuss strategies for making individual figures and sets of figures that help your audience to connect with your story and remain engaged throughout your entire story arc. First, and most importantly, you need to show your audience figures they can actually understand. It is entirely possible to follow all the recommendations I have provided throughout this book and still prepare figures that confuse. When this happens, you may have fallen victim to two common misconceptions; first, that the audience can see your figures and immediately infer the points you are trying to make; second, that the audience can rapidly process complex visualizations and understand the key trends and relationships that are shown. Neither of these assumptions is true. We need to do everything we can to help our readers understand the meaning of our visualizations and see the same patterns in the data that we see. This usually means less is more. Simplify your figures as much as possible. Remove all features that are tangential to your story. Only the important points should remain. I refer to this concept as "making a figure for the generals."

For several years, I was in charge of a large research project funded by the U.S. Army. For our annual progress reports, I was instructed by the program managers to not include a lot of figures. And any figure I did include should show very clearly how our project was succeeding. A general, the program managers told me, should be able to look at each figure and immediately see how what we were doing was improving upon or exceeding prior capabilities. Yet when my colleagues who were part of this project sent me figures for the annual progress report, many of the figures did not meet this criterion. The figures usually were overly complex, were labeled in confusing, technical terms, or did not make any obvious point at all. Most scientists are not trained to make figures for the generals.


```{block type='rmdtip', echo=TRUE}
Never assume your audience can rapidly process complex visual displays.
```

Some might hear this story and conclude that the generals are not very smart or just not that into science. I think that's exactly the wrong take-home message. The generals are simply very busy. They can't spend 30 minutes trying to decipher a cryptic figure. When they give millions of dollars of taxpayer funds to scientists to do basic research, the least they can expect in return is a handful of clear demonstrations that something worthwhile and interesting was accomplished. This story should also not be misconstrued as being about military funding in particular. The generals are a metaphor for anybody you may want to reach with your visualization. It can be a scientific reviewer for your paper or grant proposal, it can be a newspaper editor, or it can be your supervisor or your supervisor's boss at the company you're working. If you want your story to come across, you need to make figures that are appropriate for all these generals.

The first thing that will get in the way of making a figure for the generals is, ironically, the ease with which modern visualization software allows us to make sophisticated data visualizations. With nearly limitless power of visualization, it becomes tempting to keep piling on more dimensions of data. And in fact, I see a trend in the world of data visualization to make the most complex, multi-faceted visualizations possible. These visualizations may look very impressive, but they are unlikely to convey a clear story. Consider Figure \@ref(fig:arrival-delay-vs-distance), which shows the arrival delays for all flights departing out of the New York City area in 2013. I suspect it will take you a while to process this figure.

(ref:arrival-delay-vs-distance) Mean arrival delay versus distance from New York City. Each point represents one destination, and the size of each point represents the number of flights from one of the three major New York City airports (Newark, JFK, or LaGuardia) to that destination in 2013. Negative delays imply that the flight arrived early. Solid lines represent the mean trends between arrival delay and distance. Delta has consistently lower arrival delays than other airlines, regardless of distance traveled. American has among the lowest delays, on average, for short distances, but has among the highest delays for longer distances traveled.  This figure is labeled as "bad" because it is overly complex. Most readers will find it confusing and will not intuitively grasp what it is the figure is showing. Data source: U.S. Dept. of Transportation, Bureau of Transportation Statistics.

```{r arrival-delay-vs-distance, fig.width = 5*6/4.2, fig.cap = '(ref:arrival-delay-vs-distance)'}
library(nycflights13)
carrier_names <- data.frame(carrier = c("9E", "AA", "B6", "DL", "EV", "MQ", "UA", "US", "WN", "--"),
                            name= c("Endeavor", "American", "JetBlue", "Delta", "ExpressJet", "Envoy", "United",
                                    "US Airways", "Southwest", "other"))

flights_clean <- filter(flights,
                !dest %in% c("HNL", "ANC") # remove Honolulu and Anchorage because they're so far
                ) %>%
  mutate(carrier = ifelse(carrier %in% c("DL", "AA"), carrier, "--")) %>%
  left_join(carrier_names) %>%
  select(name, distance, arr_delay, dest) %>%
  na.omit()

flights_clean$name <- factor(flights_clean$name, levels = c("American", "Delta", "other"))

delay <- flights_clean %>%
  group_by(name, dest) %>%
  summarise(count = n(),
            distance = mean(distance, na.rm = TRUE),
            arr_delay = mean(arr_delay, na.rm = TRUE))

p_delay_distance <- ggplot(delay, aes(x = distance, y = arr_delay, color = name, fill = name)) +
  geom_point(aes(size = count), alpha = .5, pch = 21) +
  geom_smooth(data = flights_clean, aes(x = distance, y = arr_delay, color = name),
              se = FALSE, inherit.aes = FALSE, size = 0.75,
              method = 'gam', formula = y ~ s(x, bs = "cs", k = 3), show.legend = FALSE) +
  scale_x_continuous(limits = c(0, 3050),
                     expand = c(0, 0),
                     name = "distance (miles)") +
  scale_y_continuous(name = "mean arrival delay (min.)") +
  scale_size(breaks = c(4000, 8000, 12000), name = "# of flights",
             range = c(2, 8)) +
  scale_color_OkabeIto(name = "airline", darken = .2, order = c(2, 7, 1)) +
  scale_fill_OkabeIto(name = "airline", order = c(2, 7, 1)) +
  guides(color = guide_legend(order = 1),
         fill = guide_legend(override.aes = list(size = 4, alpha = .7), order = 1),
         size = guide_legend(override.aes = list(fill = "gray70"), order = 2)) +
  theme_dviz_grid() +
  theme(plot.margin = margin(3.5, 14, 3.5, 1.5))

stamp_bad(p_delay_distance) 
```

I think the most important feature of Figure \@ref(fig:arrival-delay-vs-distance) is that American and Delta have the shortest arrival delays. This insight is much better conveyed in a simple bar graph (Figure \@ref(fig:mean-arrival-delay-nyc)). Therefore, Figure \@ref(fig:mean-arrival-delay-nyc) is the correct figure to show if the story is about arrival delays of airlines, even if making that graph doesn't challenge your data visualization skills. And if you're then wondering whether these airlines have small delays because they don't fly that much out of New York City, you could present a second bar graph highlighting that both American and Delta are major carriers in the New York City area (Figure \@ref(fig:number-of-flights-nyc)). Both of these two bar graphs discard the distance variable shown in Figure \@ref(fig:arrival-delay-vs-distance). This is Ok. We don't need to visualize data dimensions that are tangential to our story, even if we have them and even if we could make a figure that showed them. Simple and clear is better than complex and confusing.

(ref:mean-arrival-delay-nyc) Mean arrival delay for flights out of the New York City area in 2013, by airline. American and Delta have the lowest mean arrival delays of all airlines flying out of the New York City area. Data source: U.S. Dept. of Transportation, Bureau of Transportation Statistics.

```{r mean-arrival-delay-nyc, fig.cap = '(ref:mean-arrival-delay-nyc)'}
flights %>% mutate(carrier = ifelse(carrier %in% c("OO", "HA", "YV", "F9", "AS", "FL", "VX"), "--", carrier)) %>%
    left_join(carrier_names) %>%
    group_by(name) -> flights_grouped

flights_grouped %>%
  summarize(mean_delay = mean(arr_delay, na.rm = TRUE)) %>%
  na.omit() %>%
  mutate(highlight = ifelse(name %in% c("Delta", "American"), "yes", "no")) %>%
  ggplot(aes(x=reorder(name, desc(mean_delay)), y=mean_delay, fill = highlight)) + 
    scale_fill_manual(values = c("#B0B0B0D0", "#BD3828D0"), guide = "none") +
    scale_y_continuous(expand = c(0, 0), name = "mean arrival delay (min.)") +
    scale_x_discrete(name = NULL) +
    geom_col() + 
    coord_flip(clip = "off") +
    theme_dviz_vgrid(rel_small = 1) +
    theme(
      axis.line.y = element_blank(),
      axis.ticks.y = element_blank()
    )
```

(ref:number-of-flights-nyc) Number of flights out of the New York City area in 2013, by airline. Delta and American are fourth and fifths largest carrier by flights out of the New York City area.  Data source: U.S. Dept. of Transportation, Bureau of Transportation Statistics.

```{r number-of-flights-nyc, fig.cap = '(ref:number-of-flights-nyc)'}
flights_grouped %>%
  tally() %>%
  mutate(highlight = ifelse(name %in% c("Delta", "American"), "yes", "no")) %>%
  ggplot(aes(x=reorder(name, n), y=n, fill = highlight)) + 
    scale_fill_manual(values = c("#B0B0B0D0", "#BD3828D0"), guide = "none") +
    scale_y_continuous(expand = c(0, 0), name = "number of flights") +
    scale_x_discrete(name = NULL) +
    geom_col() + 
    coord_flip(clip = "off") +
    theme_dviz_vgrid(rel_small = 1) +
    theme(
      axis.line.y = element_blank(),
      axis.ticks.y = element_blank()
    )
```


```{block type='rmdtip', echo=TRUE}
When you're trying to show too much data at once you may end up not showing anything.
```


## Build up towards complex figures

Sometimes, however, we do want to show more complex figures that contain a large amount of information at once. In those cases, we can make things easier for our readers if we first show them a simplified version of the figure before we show the final one in its full complexity. The same approach is also highly recommended for presentations. Never jump straight to a highly complex figure; first show an easily digestible subset.

This recommendation is particularly relevant if the final figure is a small multiples plot (Chapter \@ref(multi-panel-figures)) showing a grid of subplots with similar structure. The full grid is much easier to digest if the audience has first seen a single subplot by itself. For example, Figure \@ref(fig:united-departures-weekdays) shows the aggregate numbers of United Airlines departures out of Newark Airport (EWR) in 2013, broken down by weekday. Once we have seen and digested this figure, seeing the same information for ten airlines and three airports at once is much easier to process (Figure \@ref(fig:all-departures-weekdays)).

(ref:united-departures-weekdays) United Airlines departures out of Newark Airport (EWR) in 2013, by weekday. Most weekdays show approximately the same number of departures, but there are fewer departures on weekends. Data source: U.S. Dept. of Transportation, Bureau of Transportation Statistics.

```{r united-departures-weekdays, fig.width = 3.5, fig.asp = 1, fig.cap = '(ref:united-departures-weekdays)'}
flights_grouped$name <- factor(
  flights_grouped$name,
  levels = c(
    "United", "ExpressJet", "JetBlue", "Delta", "American",
    "Endeavor", "Envoy", "US Airways", "Southwest", "other"
  )
)

filter(flights_grouped, name == "United" & origin == "EWR") %>%
  ggplot(aes(x = wday(time_hour, label = TRUE, week_start = 1))) + 
    geom_bar(fill = "#0072B2D0", color = "white", size = 1, width = .97) + 
    #facet_grid(origin ~ name) +
    ggtitle("United departures, EWR") +
    scale_y_continuous(expand = c(0, 0), limits = c(0, 7500)) +
    scale_x_discrete(
      labels = c("M", "T", "W", "T", "F", "S", "S"),
      expand = c(0, 0.05),
      name = "weekday"
    ) +
    coord_cartesian(clip = "off") +
    theme_dviz_hgrid(14, dviz_font_family_condensed) +
    theme(
      strip.text = element_text(
        family = dviz_font_family_bold_condensed,
        margin = margin(4, 4, 4, 4)
      ),
      axis.line.x = element_blank(),
      panel.spacing.x = grid::unit(6.5, "pt"),
      panel.spacing.y = grid::unit(6.5, "pt"),
      panel.grid.major = element_line(color = "gray80"),
      panel.background = element_rect(fill = "gray90"),
      plot.title = element_text(
        family = dviz_font_family_bold_condensed,
        hjust = 0.5, vjust = 0.5,
        margin = margin(4, 4, 4, 4)
      )
    )
```


(ref:all-departures-weekdays) Departures out of airports in the New York city area in 2013, broken down by airline, airport, and weekday. United Airlines and ExpressJet make up most of the departures out of Newark Airport (EWR),  JetBlue, Delta, American, and Endeavor make up most of the departures out of JFK, and Delta, American, Envoy, and US Airways make up most of the departures out of LaGuardia (LGA). Most but not all airlines have fewer departures on weekends than during the work week. Data source: U.S. Dept. of Transportation, Bureau of Transportation Statistics.
 
```{r all-departures-weekdays, fig.width = 7.5*6/4.2, fig.asp = .4, fig.cap = '(ref:all-departures-weekdays)'}
ggplot(flights_grouped, aes(x = wday(time_hour, label = TRUE, week_start = 1))) + 
  geom_bar(fill = "#0072B2D0", color = "white") + 
  facet_grid(origin ~ name) +
  scale_y_continuous(expand = c(0, 0), limits = c(0, 7500)) +
  scale_x_discrete(
    labels = c("M", "", "W", "", "F", "", "S"),
    expand = c(0, 0.05),
    name = "weekday"
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12, rel_small = 1, dviz_font_family_condensed) +
  theme(
    strip.text = element_text(
      family = dviz_font_family_bold_condensed,
      margin = margin(3, 3, 3, 3)
    ),
    axis.title = element_text(size = 14),
    axis.ticks = element_line(color = "gray80"),
    axis.line.x = element_blank(),
    panel.spacing.x = grid::unit(5, "pt"),
    panel.spacing.y = grid::unit(5, "pt"),
    panel.grid.major = element_line(color = "gray80"),
    panel.background = element_rect(fill = "gray90"),
    plot.margin = margin(5.5, 1, 5.5, 1)
  )
```

## Make your figures memorable

Simple and clean figures such as simple bar plots have the advantage that they avoid distractions, are easy to read, and let your audience focus on the most important points you want to bring across. However, the simplicity can come with a disadvantage: Figures can end up looking generic. They don't have any features that stand out and make them memorable. If I showed you ten bargraphs in quick succession you'd have a hard time keeping them apart and afterwards remembering what they showed. For example, if you take a quick look at Figure \@ref(fig:petownership-bar), you will notice the visual similarity to Figure \@ref(fig:number-of-flights-nyc), which I discussed earlier in this chapter. However, the two figures have nothing in common other than they are bar charts. Figure \@ref(fig:number-of-flights-nyc) showed the number of flights out of the New York City area by airline, whereas Figure \@ref(fig:petownership-bar) shows the most popular pets in U.S. households. Neither figure has any element that helps you intuitively perceive what topic the figure covers, and therefore neither figure is particularly memorable.

(ref:petownership-bar) Number of households having one or more of the most popular pets: dogs, cats, fish, or birds. This bar graph is perfectly clear but not necessarily particularly memorable. The "cats" column has been highlighted solely to create visual similarity with Figure \@ref(fig:number-of-flights-nyc). Data source: 2012 U.S. Pet Ownership & Demographics Sourcebook, American Veterinary Medical Association

```{r petownership-bar, fig.asp = .5, fig.cap = '(ref:petownership-bar)'}

# source: 2012 U.S. Pet Ownership & Demographics Sourcebook
# American Veterinary Medical Association
# https://www.avma.org/KB/Resources/Statistics/Pages/Market-research-statistics-US-pet-ownership.aspx

df <- read.table(text = "pet  households
dogs 43346000
cats 36117000
fish	7738000
birds 3671000
horses 1780000
rabbits	1408000
turtles	1320000
poultry	1020000
", header = TRUE)


df$pet <- factor(df$pet, levels = rev(df$pet))
df <- filter(df, households > 3500000) %>%
  mutate(highlight = ifelse(pet == "cats", "yes", "no"))

ggplot(df, aes(x = pet, y = households, fill = highlight)) +
  geom_col() +
  geom_label(
    aes(label = paste0(signif(households*1e-6, 2), "M")),
    hjust = 0,
    nudge_y = 0.1e6,
    size = 14/.pt,
    family = dviz_font_family,
    label.size = 0, # no label outline
    label.padding = unit(2, "pt"),
    fill = "#FFFFFF"
  ) +
  coord_flip(clip = "off") +
  scale_x_discrete(
    name = NULL
  ) +
  scale_y_continuous(
    limits = 1e6*c(0, 47),
    breaks = 1e7*(0:4),
    labels = c("0", paste0(10*(1:4), "M")),
    name = "households",
    #position = "right",
    expand = c(0, 0)
  ) +
  scale_fill_manual(values = c("#B0B0B0D0", "#BD3828D0"), guide = "none") +
  theme_dviz_vgrid(rel_small = 1) +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = grid::unit(0, "pt")
  )

```

Research on human perception shows that more visually complex and unique figures are more memorable [@Bateman_et_al_2010; @Borgo_et_al_2012]. However, visual uniqueness and complexity do not just affect memorability, as they may hinder a person's ability to get a quick overview of the information or make it difficult to distinguish small differences in values. At the extreme, a figure could be highly memorable but utterly confusing. Such a figure would not be a good data visualization, even if it works well as a stunning piece of art. At the other extreme, figures may be very clear but forgettable and boring, and those figures may not have the impact we might hope for either. In general, we want to strike a balance between the two extremes and make our figures both memorable and clear. (The intended audience matters as well, however. If a figure is intended for a technical scientific publication, we will generally worry less about memorability than if the figure is intended for a broadly read newspaper or blog.)

We can make a figure more memorable by adding visual elements that reflect features of the data, for example drawings or pictograms of the things or objects that the dataset is about. One approach that is commonly taken is to show the data values itself in the form of repeated images, such that each copy of an image corresponds to a defined amount of the represented variable. For example, we can replace the bars in Figure \@ref(fig:petownership-bar) with repeated images of a dog, a cat, a fish, and a bird, drawn to a scale such that each complete animal corresponds to five million households (Figure \@ref(fig:petownership-isotype)). Thus, visually, Figure \@ref(fig:petownership-isotype) still functions as a bar plot, but we now have added some visual complexity that makes the figure more memorable, and we have also shown the data using images that directly reflect what the data mean. After only a quick glance at the figure, you may be able to remember that there were many more dogs and cats than fish or birds. Importantly, in such visualizations, we want to use the images to represent the data, rather than using images simply to adorn the visualization or to annotate the axes. In psychological experiments, the latter choices tend to be distracting rather than helpful [@Haroz_et_al_2015].

(ref:petownership-isotype) Number of households having one or more of the most popular pets, shown as an isotype graph. Each complete animal represents 5 million households who have that kind of pet. Data source: 2012 U.S. Pet Ownership & Demographics Sourcebook, American Veterinary Medical Association

```{r petownership-isotype, fig.asp = .5, fig.cap = '(ref:petownership-isotype)'}

images <- list(
  dogs = "images/dog.png",
  cats = "images/cat.png",
  fish = "images/fish.png",
  birds = "images/bird.png"
)

ggplot(df, aes(x = pet, y = households, image = pet)) +
  geom_isotype_col(
    img_width = grid::unit(5000000, "native"), img_height = NULL,
    nrow = 1, ncol = NA,
    hjust = 0, vjust = 0.5
  ) +
  geom_label(
    aes(label = paste0(signif(households*1e-6, 2), "M")),
    hjust = 0,
    nudge_y = 0.1e6,
    size = 14/.pt,
    family = dviz_font_family,
    label.size = 0, # no label outline
    label.padding = unit(2, "pt"),
    fill = "#FFFFFF"
  ) +
  coord_flip(clip = "off") +
  scale_image_manual(values = images, guide = "none") +
  scale_x_discrete(
    name = NULL,
    expand = expand_scale(add = c(0, 0.6))
  ) +
  scale_y_continuous(
    limits = 1e6*c(0, 47),
    breaks = 1e7*(0:4),
    labels = c("0", paste0(10*(1:4), "M")),
    name = "households",
    #position = "right",
    expand = c(0, 0)
  ) +
  theme_dviz_vgrid(rel_small = 1) +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = grid::unit(0, "pt")
  )

```

Visualizations such as Figure \@ref(fig:petownership-isotype) are often called isotype plots. The word isotype was introduced as an acronym of International System Of TYpographic Picture Education, and strictly speaking it refers to logo-like simplified pictograms that represent objects, animals, plants, or people [@Haroz_et_al_2015]. However, I think it makes sense to use the term isotype plot more broadly to apply to any type of visualization where repeated copies of the same image are used to indicate the magnitude of a value. After all, the prefix "iso" means "the same" and "type" can mean a particular kind, class, or group.


## Be consistent but don't be repetitive

When discussing compound figures in Chapter \@ref(compound-figures), I mentioned that it is important to use a consistent visual language for the different parts of a larger figure. The same is true across figures. If we make three figures that are all part of one larger story, then we need to design those figures so they look like they belong together. Using a consistent visual language does not mean, however, that everything should look exactly the same. On the contrary. It is important that figures describing different analyses look visually distinct, so that your audience can easily recognize where one analysis ends and another one starts. This is best achieved by using different visualization approaches for different parts of the overarching story. If you have used a bar plot already, next use a scatterplot, or a boxplot, or a line plot. Otherwise, the different analyses will blur together in your audience's mind, and they will have a hard time distinguishing one part of the story from another. For example, if we re-design Figure \@ref(fig:athletes-composite-good) from Chapter \@ref(compound-figures) so it uses only bar plots, the result is noticeable less distinct and more confusing (Figure \@ref(fig:athletes-composite-repetitive)).

(ref:athletes-composite-repetitive) Physiology and body-composition of male and female athletes. Error bars indicate the standard error of the mean. This figure is overly repetitive. It shows the same data as Figure \@ref(fig:athletes-composite-good) and it uses a consistent visual language, but all sub-figures use the same type of visualization (bar plot). This makes it difficult for the reader to process that parts (a), (b), and (c) show entirely different results. Data source: @Telford-Cunningham-1991

```{r athletes-composite-repetitive, fig.width = 5*6/4.2, fig.asp = 0.75, fig.cap = '(ref:athletes-composite-repetitive)'}
male_sport <- unique(filter(Aus_athletes, sex=="m")$sport)
female_sport <- unique(filter(Aus_athletes, sex=="f")$sport)
both_sport <- male_sport[male_sport %in% female_sport]
athletes_df <- filter(Aus_athletes, sport %in% both_sport) %>%
  mutate(
    sport = case_when(
      sport == "track (400m)" ~ "track",
      sport == "track (sprint)" ~ "track",
      TRUE ~ sport
    ),
    sex = factor(sex, levels = c("f", "m"))
  )

cc_df <- athletes_df %>%
  select(sex, rcc, wcc) %>%
  gather(variable, value, -sex) %>%
  group_by(sex, variable) %>%
  summarize(
    mean = mean(value),
    se = sd(value)/sqrt(n())
  )

p1 <- ggplot(athletes_df, aes(x = sex, fill = sex)) + 
  geom_bar() +
  scale_y_continuous(limits = c(0, 95), expand = c(0, 0), name = "number") +
  scale_x_discrete(name = NULL, labels = c("female", "male")) +
  scale_fill_manual(values = c("#D55E00D0", "#0072B2D0"), guide = "none") +
  theme_dviz_hgrid(12, rel_small = 1) + 
  theme(
    #axis.line.x = element_blank(),
    axis.ticks.x = element_blank(),
    #axis.ticks.length = grid::unit(0, "pt"),
    plot.margin = margin(3, 0, 0, 0)
  )

p2 <- ggplot(cc_df, aes(x = sex, y = mean, fill = sex)) + 
  geom_col() +
  geom_linerange(aes(ymin = mean - se, ymax = mean + se)) +
  facet_wrap(
    ~variable, scales = "free_y",
    labeller = as_labeller(function(label) {
      ifelse(label == "wcc", "WBC count", "RBC count")
    })
  ) +
  scale_y_continuous(limits = c(0, 7.8), expand = c(0, 0), name = "mean count") +
  scale_x_discrete(name = NULL, labels = c("female", "male")) +
  scale_fill_manual(values = c("#D55E00D0", "#0072B2D0"), guide = "none") +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    #axis.line.x = element_blank(),
    axis.ticks.x = element_blank(),
    #axis.ticks.length = grid::unit(0, "pt"),
    plot.margin = margin(3, 0, 0, 0)
  )

p_row <- plot_grid(
  p1, NULL, p2,
  labels = c("a", "", "b"),
  align = 'h',
  axis = 'l',
  nrow = 1,
  rel_widths = c(0.7, 0.02, 1)
)

bf_df <- athletes_df %>%
  select(sex, sport, pcBfat) %>%
  group_by(sex, sport) %>%
  summarize(
    mean = mean(pcBfat),
    se = sd(pcBfat)/sqrt(n())
  )

p3 <- ggplot(bf_df, aes(x = sport, y = mean, fill = sex)) + 
  geom_col(position = "dodge") +
  geom_linerange(aes(ymin = mean - se, ymax = mean + se), position = position_dodge(width = .9)) +
  scale_y_continuous(limits = c(0, 24.7), expand = c(0, 0), name = "% body fat") +
  scale_x_discrete(name = NULL) +
  scale_fill_manual(
    values = c("#D55E00D0", "#0072B2D0"), name = NULL,
    labels = c("female   ", "male")
  ) +
  guides(fill = guide_legend(
    override.aes = list(color = "white", size = 2),
    direction = "horizontal")
  ) +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    #axis.line.x = element_blank(),
    axis.ticks.x = element_blank(),
    #axis.ticks.length = grid::unit(0, "pt"),
    legend.position = c(1., 0.9),
    legend.justification = "right"
  )

stamp_ugly(
  plot_grid(
    p_row, NULL, p3,
    ncol = 1,
    rel_heights = c(1, .04, 1),
    labels = c("", "", "c")
  ) +
  theme(plot.margin = margin(6, 6, 3, 1.5))
)
```


```{block type='rmdtip', echo=TRUE}
When preparing a presentation or report, aim to use a different type of visualization for each distinct analysis.
```

Sets of repetitive figures are often a consequence of multi-part stories where each part is based on the same type of raw data. In those scenarios, it can be tempting to use the same type of visualization for each part. However, in aggregate, these figures will not hold the audience's attention. As an example, let's consider a story about the Facebook stock, in two parts: (i) the Facebook stock price has increased rapidly from 2012 to 2017; (ii) the price increase has outpaced that of other large tech companies. You might want to visualize these two statements with two figures showing stock price over time, as demonstrated in Figure \@ref(fig:tech-stocks-repetitive). However, while Figure \@ref(fig:tech-stocks-repetitive)a serves a clear purpose and should remain as is, Figure \@ref(fig:tech-stocks-repetitive)b is at the same time repetitive and obscures the main point. We don't particularly care about the exact temporal evolution of the stock price of Alphabet, Apple, and Microsoft, we just want to highlight that it grew less than the stock price of Facebook.

(ref:tech-stocks-repetitive) Growth of Facebook stock price over a five-year interval and comparison with other tech stocks. (a) The Facebook stock price rose from around \$25/share in mid-2012 to \$150/share in mid-2017. (b) The prices of other large tech companies did not rise comparably over the same time period. Prices have been indexed to 100 on June 1, 2012 to allow for easy comparison. This figure is labeled as "ugly" because parts (a) and (b) are repetitive. Data source: Yahoo Finance

```{r tech-stocks-repetitive, fig.asp = 2*0.5, fig.cap = '(ref:tech-stocks-repetitive)'}

final_prices <- filter(tech_stocks, date == ymd("2017-05-31"))

fb_price_plot <- ggplot(
  filter(tech_stocks, ticker == "FB"),
  aes(x = date, y = price, color = ticker)
) +
  geom_line(size = 0.66, color = "#0072B2", na.rm = TRUE) +
  scale_x_date(
    name = "year",
    limits = c(ymd("2012-06-01"), ymd("2017-05-31")),
    expand = c(0,0)
  ) + 
  scale_y_continuous(
    name = "stock price (USD)",
    limits = c(0, 155),
    expand = c(0,0),
    sec.axis = dup_axis(
      breaks = filter(final_prices, ticker == "FB")$price,
      labels = "Facebook",
      name = NULL
    )
  ) +
  theme_dviz_hgrid() + 
  theme(axis.ticks.y.right = element_blank())

all_indexed_price_plot <- ggplot(tech_stocks, aes(x = date, y = price_indexed, color = ticker)) +
  geom_line(size = 0.66, na.rm = TRUE) +
  scale_color_manual(
    values = c("#000000", "#0072B2", "#E69F00", "#009E73"),
    guide = "none"
  ) +
  scale_x_date(
    name = "year",
    limits = c(ymd("2012-06-01"), ymd("2017-05-31")),
    expand = c(0,0)
  ) + 
  scale_y_continuous(
    name = "stock price, indexed",
    limits = c(0, 560),
    expand = c(0,0),
    sec.axis = dup_axis(
      breaks = final_prices$price_indexed,
      labels = final_prices$company,
      name = NULL
    )
  ) +
  theme_dviz_hgrid() + 
  theme(axis.ticks.y.right = element_blank())

stamp_ugly(
  plot_grid(
    fb_price_plot,
    all_indexed_price_plot,
    ncol = 1, align = "v", labels = "auto"
  ) + theme(plot.margin = margin(14, 7, 0, 1.5))
)
```

I would recommend to leave part (a) as is but replace part (b) with a bar plot showing percent increase (Figure \@ref(fig:tech-stocks-diverse)). Now we have two distinct figures that each make a unique, clear point and that work well in combination. Part (a) allows the reader to get familiar with the raw, underlying data and part (b) highlights the magnitude of the effect while removing any tangential information.

(ref:tech-stocks-diverse) Growth of Facebook stock price over a five-year interval and comparison with other tech stocks. (a) The Facebook stock price rose from around \$25/share in mid-2012 to \$150/share in mid-2017, an increase of almost 450%. (b) The prices of other large tech companies did not rise comparably over the same time period. Price increases ranged from 90% to 240%. Data source: Yahoo Finance

```{r tech-stocks-diverse, fig.asp = 2*0.5, fig.cap = '(ref:tech-stocks-diverse)'}

final_prices <- filter(tech_stocks, date == ymd("2017-05-31"))

perc_increase <- 
  final_prices %>%
  mutate(
    perc = 100*(price-index_price)/index_price,
    label = paste(as.character(round(perc)), "%", sep="")) %>%
  arrange(perc)

perc_increase$ticker <- factor(perc_increase$ticker, levels=perc_increase$ticker)

perc_plot <- ggplot(perc_increase, aes(x = ticker, y=perc, fill = ticker)) +
  geom_col(alpha = 0.8) +
  geom_text(
    aes(label=label), color="white", hjust=1.1, size=5,
    family = dviz_font_family
  ) +
  scale_fill_manual(
    values = c("gray50", "gray50", "gray50", "#0072B2"),
    guide = "none"
  ) +
  scale_y_continuous(
    #name = "percent increase\n(June 2012 to June 2017)",
    name = "percent increase",
    limits = c(0, 499),
    expand = c(0, 0)
  ) +
  scale_x_discrete(
    name = NULL,
    breaks = c("FB", "GOOG", "MSFT", "AAPL"),
    labels = c("Facebook", "Alphabet", "Microsoft", "Apple")
  ) +
  coord_flip() +
  theme_dviz_vgrid()

plot_grid(
  fb_price_plot,
  perc_plot,
  ncol = 1, labels = "auto"
) +
  theme(plot.margin = margin(14, 7, 0, 1.5))
```

Figure \@ref(fig:tech-stocks-diverse) highlights a general principle that I follow when preparing sets of figures to tell a story: I start with a figure that is as close as possible to showing the raw data, and in subsequent figures I show increasingly more derived quantities. Derived quantities (such as percent increases, averages, coefficients of fitted models, and so on) are useful to summarize key trends in large and complex datasets. However, because they are derived they are less intuitive, and if we show a derived quantity before we have shown the raw data our audience will find it difficult to follow. On the flip side, if we try to show all trends by showing raw data we will end up needing too many figures and/or being repetitive.

How many figures you should you use to tell your story? The answer depends on the publication venue. For a short blog post or tweet, make one figure. For scientific papers, I recommend between three and six figures. If I have many more than six figures for a scientific paper, then some of them need to be moved into an appendix or supplementary materials section. It is good to document all the evidence we have collected, but we must not wear out our audience by presenting excessive numbers of mostly similar-looking figures. In other contexts, a larger number of figures may be appropriate. However, in those contexts, we will usually be telling multiple stories, or an overarching story with subplots. For example, if I am asked to give an hour-long scientific presentation, I usually aim to tell three distinct stories. Similarly, a book or thesis will contain more than one story, and in fact may contain one story per chapter or section. In those scenarios, each distinct story-line or subplot should be presented with no more than three to six figures. In this book, you will find that I follow this principle at the level of sections within chapter. Each section is approximately self-contained and will typically show no more than six figures.


<!--chapter:end:telling_a_story.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")

library(ggridges)
library(lubridate)
library(ggrepel)
```

# Visualizing time series and other functions of an independent variable {#time-series}

The preceding chapter discussed scatter plots, where we plot one quantitative variable against another. A special case arises when one of the two variables can be thought of as time, because time imposes additional structure on the data. Now the data points have an inherent order; we can arrange the points in order of increasing time and define a predecessor and successor for each data point. We frequently want to visualize this temporal order and we do so with line graphs. Line graphs are not limited to time series, however. They are appropriate whenever one variable imposes an ordering on the data. This scenario arises also, for example, in a controlled experiment where a treatment variable is purposefully set to a range of different values. If we have multiple variables that depend on time, we can either draw separate line plots or we can draw a regular scatter plot and then draw lines to connect the neighboring points in time.


## Individual time series

As a first demonstration of a time series, we will consider the pattern of monthly preprint submissions in biology. Preprints are scientific articles that researchers post online before formal peer review and publication in a scientific journal. The preprint server bioRxiv, which was founded in November 2013 specifically for researchers working in the biological sciences, has seen substantial growth in monthly submissions since. We can visualize this growth by making a form of scatter plot (Chapter \@ref(visualizing-associations)) where we draw dots representing the number of submissions in each month (Figure \@ref(fig:biorxiv-dots)).

(ref:biorxiv-dots) Monthly submissions to the preprint server bioRxiv, from its inception in November 2014 until April 2018. Each dot represents the number of submissions in one month. There has been a steady increase in submission volume throughout the entire 4.5-year period. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r biorxiv-dots, fig.cap = '(ref:biorxiv-dots)'}
preprint_growth %>% filter(archive == "bioRxiv") %>%
  filter(count > 0) -> biorxiv_growth

ggplot(biorxiv_growth, aes(date, count)) + 
  #geom_point(color = "#0072B2") +
  geom_point(color = "white", fill = "#0072B2", shape = 21, size = 2) +
  scale_y_continuous(limits = c(0, 1600), expand = c(0, 0),
                name = "preprints / month") + 
  scale_x_date(name = "year") +
  theme_dviz_open() +
  theme(plot.margin = margin(7, 7, 3, 1.5))
```

There is an important difference however between Figure \@ref(fig:biorxiv-dots) and the scatter plots discussed in Chapter \@ref(visualizing-associations). In Figure \@ref(fig:biorxiv-dots), the dots are spaced evenly along the *x* axis, and there is a defined order among them. Each dot has exactly one left and one right neighbor (except the leftmost and rightmost points which have only one neighbor each). We can visually emphasize this order by connecting neighboring points with lines (Figure \@ref(fig:biorxiv-dots-line)). Such a plot is called a *line graph*.

(ref:biorxiv-dots-line) Monthly submissions to the preprint server bioRxiv, shown as dots connected by lines. The lines do not represent data but are only meant as a guide to the eye. By connecting the individual dots with lines, we emphasize that there is an order between the dots, each dot has exactly one neighbor that comes before and one that comes after. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r biorxiv-dots-line, fig.cap = '(ref:biorxiv-dots-line)'}
ggplot(biorxiv_growth, aes(date, count)) +
  geom_line(size = 0.5, color = "#0072B2") + 
  geom_point(color = "white", fill = "#0072B2", shape = 21, size = 2) +
  scale_y_continuous(limits = c(0, 1600), expand = c(0, 0),
                name = "preprints / month") + 
  scale_x_date(name = "year") +
  theme_dviz_open() +
  theme(plot.margin = margin(7, 7, 3, 1.5))
```

Some people object to drawing lines between points because the lines do not represent observed data. In particular, if there are only a few observations spaced far apart, had observations been made at intermediate times they would probably not have fallen exactly onto the lines shown. Thus, in a sense, the lines correspond to made-up data. Yet they may help with perception when the points are spaced far apart or are unevenly spaced. We can somewhat resolve this dilemma by pointing it out in the figure caption, for example by writing "lines are meant as a guide to the eye" (see caption of Figure \@ref(fig:biorxiv-dots-line)).

Using lines to represent time series is generally accepted practice, however, and frequently the dots are omitted altogether (Figure \@ref(fig:biorxiv-line)). Without dots, the figure places more emphasis on the overall trend in the data and less on individual observations. A figure without dots is also visually less busy. In general, the denser the time series, the less important it is to show individual observations with dots. For the preprint dataset shown here, I think omitting the dots is fine.

(ref:biorxiv-line) Monthly submissions to the preprint server bioRxiv, shown as a line graph without dots. Omitting the dots emphasizes the overall temporal trend while de-emphasizing individual observations at specific time points. It is particularly useful when the time points are spaced very densely. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r biorxiv-line, fig.cap = '(ref:biorxiv-line)'}
ggplot(biorxiv_growth, aes(date, count)) + geom_line(color = "#0072B2", size = 1) +
  scale_y_continuous(limits = c(0, 1600), expand = c(0, 0),
                name = "preprints / month") + 
  scale_x_date(name = "year") +
  theme_dviz_open() +
  theme(plot.margin = margin(7, 7, 3, 1.5))
```

We can also fill the area under the curve with a solid color (Figure \@ref(fig:biorxiv-line-area)). This choice further emphasizes the overarching trend in the data, because it visually separates the area above the curve from the area below. However, this visualization is only valid if the *y* axis starts at zero, so that the height of the shaded area at each time point represents the data value at that time point.

(ref:biorxiv-line-area) Monthly submissions to the preprint server bioRxiv, shown as a line graph with filled area underneath. By filling the area under the curve, we put even more emphasis on the overarching temporal trend than if we just draw a line (Figure \@ref(fig:biorxiv-line)). Data source: Jordan Anaya, http://www.prepubmed.org/

```{r biorxiv-line-area, fig.cap = '(ref:biorxiv-line-area)'}
ggplot(biorxiv_growth, aes(date, height = count, y = 0)) + 
  geom_ridgeline(color = "#0072B2", fill = "#0072B240", size = 0.75) +
  scale_y_continuous(limits = c(0, 1600), expand = c(0, 0),
                name = "preprints / month") + 
  scale_x_date(name = "year") +
  theme_dviz_open() +
  theme(plot.margin = margin(7, 7, 3, 1.5))
```


## Multiple time series and dose--response curves

We often have multiple time courses that we want to show at once. In this case, we have to be more careful in how we plot the data, because the figure can become confusing or difficult to read. For example, if we want to show the monthly submissions to multiple preprint servers, a scatter plot is not a good idea, because the individual time courses run into each other (Figure \@ref(fig:bio-preprints-dots)). Connecting the dots with lines alleviates this issue (Figure \@ref(fig:bio-preprints-lines)).

(ref:bio-preprints-dots) Monthly submissions to three preprint servers covering biomedical research: bioRxiv, the q-bio section of arXiv, and PeerJ Preprints. Each dot represents the number of submissions in one month to the respective preprint server. This figure is labeled "bad" because the three time courses visually interfere with each other and are difficult to read. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r bio-preprints-dots, fig.cap = '(ref:bio-preprints-dots)'}
preprint_growth %>% filter(archive %in% c("bioRxiv", "arXiv q-bio", "PeerJ Preprints")) %>%
  filter(count > 0) %>%
  mutate(archive = factor(archive, levels = c("bioRxiv", "arXiv q-bio", "PeerJ Preprints")))-> preprints

p <- ggplot(preprints, aes(date, count, color = archive, fill = archive, shape = archive)) + 
  geom_point(color = "white", size = 2) +
  scale_shape_manual(values = c(21, 22, 23),
                     name = NULL) + 
  scale_y_continuous(limits = c(0, 600), expand = c(0, 0),
                name = "preprints / month") + 
  scale_x_date(name = "year",
               limits = c(min(biorxiv_growth$date), ymd("2017-01-01"))) +
  scale_color_manual(values = c("#0072b2", "#D55E00", "#009e73"),
                     name = NULL) +
  scale_fill_manual(values = c("#0072b2", "#D55E00", "#009e73"),
                     name = NULL) +
  theme_dviz_open() +
  theme(legend.title.align = 0.5,
        legend.position = c(0.1, .9),
        legend.just = c(0, 1),
        plot.margin = margin(14, 7, 3, 1.5))

stamp_bad(p)
```

(ref:bio-preprints-lines) Monthly submissions to three preprint servers covering biomedical research. By connecting the dots in Figure \@ref(fig:bio-preprints-dots) with lines, we help the viewer follow each individual time course. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r bio-preprints-lines, fig.cap = '(ref:bio-preprints-lines)'}
ggplot(preprints, aes(date, count, color = archive, fill = archive, shape = archive)) + 
  geom_line(size = 0.75) + geom_point(color = "white", size = 2) +
  scale_y_continuous(limits = c(0, 600), expand = c(0, 0),
                name = "preprints / month") + 
  scale_x_date(name = "year",
               limits = c(min(biorxiv_growth$date), ymd("2017-01-01"))) +
  scale_color_manual(values = c("#0072b2", "#D55E00", "#009e73"),
                     name = NULL) +
  scale_fill_manual(values = c("#0072b2", "#D55E00", "#009e73"),
                     name = NULL) +
  scale_shape_manual(values = c(21, 22, 23),
                     name = NULL) + 
  theme_dviz_open() +
  theme(legend.title.align = 0.5,
        legend.position = c(0.1, .9),
        legend.just = c(0, 1),
        plot.margin = margin(14, 7, 3, 1.5))
```

Figure \@ref(fig:bio-preprints-lines) represents an acceptable visualization of the preprints dataset. However, the separate legend creates unnecessary cognitive load. We can reduce this cognitive load by labeling the lines directly (Figure \@ref(fig:bio-preprints-direct-label)). We have also eliminated the individual dots in this figure, for a result that is much more streamlined and easy to read than the original starting point, Figure \@ref(fig:bio-preprints-dots).

(ref:bio-preprints-direct-label) Monthly submissions to three preprint servers covering biomedical research. By direct labeling the lines instead of providing a legend, we have reduced the cognitive load required to read the figure. And the elimination of the legend removes the need for points of different shapes. Thus, we could streamline the figure further by eliminating the dots. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r bio-preprints-direct-label, fig.cap = '(ref:bio-preprints-direct-label)'}
preprints_final <- filter(preprints, date == ymd("2017-01-01"))

ggplot(preprints) +
  aes(date, count, color = archive, fill = archive, shape = archive) + 
  geom_line(size = 1) + 
  #geom_point(color = "white", size = 2) +
  scale_y_continuous(
    limits = c(0, 600), expand = c(0, 0),
    name = "preprints / month",
    sec.axis = dup_axis(
      breaks = preprints_final$count,
      labels = c("arXiv\nq-bio", "PeerJ\nPreprints", "bioRxiv"),
      name = NULL)
  ) + 
  scale_x_date(name = "year",
               limits = c(min(biorxiv_growth$date), ymd("2017-01-01")),
               expand = expand_scale(mult = c(0.02, 0))) +
  scale_color_manual(values = c("#0072b2", "#D55E00", "#009e73"),
                     name = NULL) +
  scale_fill_manual(values = c("#0072b2", "#D55E00", "#009e73"),
                     name = NULL) +
  scale_shape_manual(values = c(21, 22, 23),
                     name = NULL) + 
  coord_cartesian(clip = "off") +
  theme_dviz_open() +
  theme(legend.position = "none") +
  theme(axis.line.y.right = element_blank(),
        axis.ticks.y.right = element_blank(),
        axis.text.y.right = element_text(margin = margin(0, 0, 0, 0)),
        plot.margin = margin(14, 7, 3, 1.5))
```

Line graphs are not limited to time series. They are appropriate whenever the data points have a natural order that is reflected in the variable shown along the *x* axis, so that neighboring points can be connected with a line. This situation arises, for example, in dose--response curves, where we measure how changing some numerical parameter in an experiment (the dose) affects an outcome of interest (the response). Figure \@ref(fig:oats-yield) shows a classic experiment of this type, measuring oat yield in response to increasing amounts of fertilization. The line-graph visualization highlights how the dose--response curve has a similar shape for the three oat varieties considered but differs in the starting point in the absence of fertilization (i.e., some varieties have naturally higher yield than others).


(ref:oats-yield) Dose--response curve showing the mean yield of oats varieties after fertilization with manure. The manure serves as a source of nitrogen, and oat yields generally increase as more nitrogen is available, regardless of variety. Here, manure application is measured in cwt (hundredweight) per acre. The hundredweight is an old imperial unit equal to 112 lbs or 50.8 kg. Data source: @Yates1935

```{r oats-yield, fig.cap = '(ref:oats-yield)'}
MASS::oats %>% 
  # 1 long (UK) cwt == 112 lbs == 50.802345 kg
  mutate(N = 1*as.numeric(sub("cwt", "", N, fixed = TRUE))) %>%
  group_by(N, V) %>%
  summarize(mean = 20 * mean(Y)) %>% # factor 20 converts units to lbs/acre
  mutate(variety = ifelse(V == "Golden.rain", "Golden Rain", as.character(V))) ->
  oats_df

oats_df$variety <- factor(oats_df$variety, levels = c("Marvellous", "Golden Rain", "Victory"))
 
ggplot(oats_df,
       aes(N, mean, color = variety, shape = variety, fill = variety)) +
  geom_line(size = 0.75) + geom_point(color = "white", size = 2.5) +
  scale_y_continuous(name = "mean yield (lbs/acre)") + 
  scale_x_continuous(name = "manure treatment (cwt/acre)") +
  scale_shape_manual(values = c(21, 22, 23),
                     name = "oat variety") + 
  scale_color_manual(values = c("#0072b2", "#D55E00", "#009e73"),
                     name = "oat variety") +
  scale_fill_manual(values = c("#0072b2", "#D55E00", "#009e73"),
                     name = "oat variety") +
  coord_cartesian(clip = "off") +
  theme_dviz_open() +
  theme(legend.title.align = 0.5)

```

## Time series of two or more response variables {#time-series-connected-scatter}

In the preceding examples we dealt with time courses of only a single response variable (e.g., preprint submissions per month or oat yield). It is not unusual, however, to have more than one response variable. Such situations arise commonly in macroeconomics. For example, we may be interested in the change in house prices from the previous 12 months as it relates to the unemployment rate. We may expect that house prices rise when the unemployment rate is low and vice versa.

Given the tools from the preceding subsections, we can visualize such data as two separate line graphs stacked on top of each other (Figure \@ref(fig:house-price-unemploy)). This plot directly shows the two variables of interest, and it is straightforward to interpret. However, because the two variables are shown as separate line graphs, drawing comparisons between them can be cumbersome. If we want to identify temporal regions when both variables move in the same or in opposite directions, we need to switch back and forth between the two graphs and compare the relative slopes of the two curves.

(ref:house-price-unemploy) 12-month change in house prices (a) and  unemployment rate (b) over time, from Jan. 2001 through Dec. 2017. Data sources: Freddie Mac House Prices Index, U.S. Bureau of Labor Statistics.

```{r house-price-unemploy, fig.width = 5*6/4.2, fig.cap = '(ref:house-price-unemploy)'}
# prepare dataset already for next figure
CA_house_prices <- 
  filter(house_prices, state == "California", year(date) > 2000) %>%
  mutate(
    label = ifelse(
      date %in% c(ymd("2005-01-01"), ymd("2007-07-01"), 
                  ymd("2010-01-01"), ymd("2012-07-01"), ymd("2015-01-01")),
      format(date, "%b %Y"), ""),
    nudge_x = case_when(
      label == "Jan 2005" ~ -0.003,
      TRUE ~ 0.003
    ),
    nudge_y = case_when(
      label == "Jan 2005" ~ 0.01,
      label %in% c("Jul 2007", "Jul 2012") ~ 0.01,
      TRUE ~ -0.01
    ),
    hjust = case_when(
      label == "Jan 2005" ~ 1,
      TRUE ~ 0
    )
  )

p1 <- ggplot(CA_house_prices, aes(date, house_price_perc)) +
  geom_line(size = 1, color = "#0072b2") +
  scale_y_continuous(
    limits = c(-0.3, .32), expand = c(0, 0),
    breaks = c(-.3, -.15, 0, .15, .3),
    name = "12-month change\nin house prices", labels = scales::percent_format(accuracy = 1)
  ) + 
  scale_x_date(name = "", expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_grid() +
  theme(
    axis.line = element_blank(),
    plot.margin = margin(12, 1.5, 0, 1.5)
  )

p2 <- ggplot(CA_house_prices, aes(date, unemploy_perc/100)) +
  geom_line(size = 1, color = "#0072b2") +
  scale_y_continuous(
    limits = c(0.037, 0.143),
    name = "unemployment\nrate", labels = scales::percent_format(accuracy = 1),
    expand = c(0, 0)
  ) +
  scale_x_date(name = "year", expand = c(0, 0)) +
  theme_dviz_grid() +
  theme(
    axis.line = element_blank(),
    plot.margin = margin(6, 1.5, 3, 1.5)
  )
 
plot_grid(p1, p2, align = 'v', ncol = 1, labels = "auto") 
```

As an alternative to showing two separate line graphs, we can plot the two variables against each other, drawing a path that leads from the earliest time point to the latest (Figure \@ref(fig:house-price-path)). Such a visualization is called a *connected scatter plot*, because we are technically making a scatter plot of the two variables against each other and then are connecting neighboring points. Physicists and engineers often call this a *phase portrait*, because in their disciplines it is commonly used to represent movement in phase space. We have previously encountered connected scatter plots in Chapter \@ref(coordinate-systems-axes), where we plotted the daily temperature normals in Houston, TX, versus those in San Diego, CA (Figure \@ref(fig:temperature-normals-Houston-San-Diego)).

(ref:house-price-path) 12-month change in house prices versus unemployment rate, from Jan. 2001 through Dec. 2017, shown as a connected scatter plot. Darker shades represent more recent months. The anti-correlation seen in Figure \@ref(fig:house-price-unemploy) between the change in house prices and the unemployment rate causes the connected scatter plot to form two counter-clockwise circles. Data sources: Freddie Mac House Price Index, U.S. Bureau of Labor Statistics. Original figure concept: Len Kiefer

```{r house-price-path, fig.asp = 3/4, fig.cap = '(ref:house-price-path)'}

ggplot(CA_house_prices) +
  aes(unemploy_perc/100, house_price_perc, colour = as.numeric(date)) + 
  geom_path(size = 1, lineend = "round") +
  geom_text_repel(
    aes(label = label), point.padding = .2, color = "black",
    min.segment.length = 0, size = 12/.pt,
    hjust = CA_house_prices$hjust,
    nudge_x = CA_house_prices$nudge_x,
    nudge_y = CA_house_prices$nudge_y,
    direction = "y",
    family = dviz_font_family
  ) +
  scale_x_continuous(
    limits = c(0.037, 0.143),
    name = "unemployment rate", labels = scales::percent_format(accuracy = 1),
    expand = c(0, 0)
  ) +
  scale_y_continuous(
    limits = c(-0.315, .315), expand = c(0, 0),
    breaks = c(-.3, -.15, 0, .15, .3),
    name = "12-month change in house prices", labels = scales::percent_format(accuracy = 1)
  ) + 
  scale_colour_gradient(low = "#E7F0FF", high = "#035B8F") + #"#0072b2") +
  guides(colour = FALSE) +
  coord_cartesian(clip = "off") +
  theme_dviz_grid() +
  theme(
    axis.ticks.length = unit(0, "pt"),
    plot.margin = margin(21, 14, 3.5, 1.5))
```

In a connected scatter plot, lines going in the direction from the lower left to the upper right represent correlated movement between the two variables (as one variable grows, so does the other), and lines going in the perpendicular direction, from the upper left to the lower right, represent anti-correlated movement (as one variable grows, the other shrinks). If the two variables have a somewhat cyclic relationship, we will see circles or spirals in the connected scatter plot. In Figure \@ref(fig:house-price-path), we see one small circle from 2001 through 2005 and one large circle for the remainder of the time course.


When drawing a connected scatter plot, it is important that we indicate both the direction  and the temporal scale of the data. Without such hints, the plot can turn into meaningless scribble (Figure \@ref(fig:house-price-path-bad)). I am using here (in Figure \@ref(fig:house-price-path)) a gradual darkening of the color to indicate direction. Alternatively, one could draw arrows along the path.

(ref:house-price-path-bad) 12-month change in house prices versus unemployment rate, from Jan. 2001 through Dec. 2017. This figure is labeled "bad" because without the date markers and color shading of Figure \@ref(fig:house-price-path), we can see neither the direction nor the speed of change in the data. Data sources: Freddie Mac House Prices Index, U.S. Bureau of Labor Statistics.

```{r house-price-path-bad, fig.asp = 3/4, fig.cap = '(ref:house-price-path-bad)'}

p <- ggplot(CA_house_prices) +
  aes(unemploy_perc/100, house_price_perc) + 
  geom_path(size = 1, lineend = "round", color = "#0072b2") +
  scale_x_continuous(
    limits = c(0.037, 0.143),
    name = "unemployment rate", labels = scales::percent_format(accuracy = 1),
    expand = c(0, 0)
  ) +
  scale_y_continuous(
    limits = c(-0.315, .315), expand = c(0, 0),
    breaks = c(-.3, -.15, 0, .15, .3),
    name = "12-month change in house prices", labels = scales::percent_format(accuracy = 1)
  ) + 
  coord_cartesian(clip = "off") +
  theme_dviz_grid() +
  theme(
    axis.ticks.length = unit(0, "pt"),
    plot.margin = margin(21, 14, 3.5, 1.5))

stamp_bad(p)
```

Is it better to use a connected scatter plot or two separate line graphs? Separate line graphs tend to be easier to read, but once people are used to connected scatter plots they may be able to extract certain patterns (such as cyclical behavior with some irregularity) that can be difficult to spot in line graphs. In fact, to me the cyclical relationship between change in house prices and unemployment rate is hard to spot in Figure \@ref(fig:house-price-unemploy), but the counter-clockwise spiral in Figure \@ref(fig:house-price-path) clearly shows it. Research reports that readers are more likely to confuse order and direction in a connected scatter plot than in line graphs and less likely to report correlation [@Haroz_et_al_2016]. On the flip side, connected scatter plots seem to result in higher engagement, and thus such plots may be a effective tools to draw readers into a story [@Haroz_et_al_2016].

Even though connected scatter plots can show only two variables at a time, we can also use them to visualize higher-dimensional datasets. The trick is to apply dimension reduction first (see Chapter \@ref(visualizing-associations)). We can then draw a connected scatterplot in the dimension-reduced space. As an example of this approach, we will visualize a database of monthly observations of over 100 macroeconomic indicators, provided by the Federal Reserve Bank of St. Louis. We perform a principal components analysis (PCA) of all indicators and then draw a connected scatter plot of PC 2 versus PC 1 (Figure \@ref(fig:fred-md-PCA)a) and versus PC 3 (Figure \@ref(fig:fred-md-PCA)b).

(ref:fred-md-PCA) Visualizing a high-dimensional time series as a connected scatter plot in principal components space. The path indicates the joint movement of over 100 macroeconomic indicators from January 1990 to December 2017. Times of recession and recovery are indicated via color, and the end points of the three recessions (March 1991, November 2001, and June 2009) are also labeled. (a) PC 2 versus PC 1. (b) PC 2 versus PC 3. Data source: M. W. McCracken, St. Louis Fed

```{r fred-md-PCA, fig.width = 5*6/4.2, fig.asp = 2*0.618, fig.cap = '(ref:fred-md-PCA)'}
fred_md %>%
  select(-date, -sasdate) %>%
  scale() %>%
  prcomp() -> pca

pca_data <- data.frame(date = fred_md$date, pca$x) %>%
  mutate(
    type = ifelse(
      (ymd("1990-07-01") <= date & date < ymd("1991-03-01")) |
      (ymd("2001-03-01") <= date & date < ymd("2001-11-01")) |
      (ymd("2007-12-01") <= date & date < ymd("2009-06-01")),
      "recession",
      "recovery"
    )
  )

pca_labels <-
  mutate(pca_data,
    label = ifelse(
      date %in% c(ymd("1990-01-01"), ymd("1991-03-01"), ymd("2001-11-01"),
                  ymd("2009-06-01"), ymd("2017-12-01")),
      format(date, "%b %Y"), ""
    )
  ) %>%
  filter(label != "") %>%
  mutate(
    nudge_x = c(.2, -.2, -.2, -.2, .2),
    nudge_y = c(.2, -.2, -.2, -.2, .2),
    hjust = c(0, 1, 1, 1, 0),
    vjust = c(0, 1, 1, 1, 0),
    nudge_x2 = c(.2, .2, .2, -.2, .2),
    nudge_y2 = c(.2, -.2, -1, .2, .2),
    hjust2 = c(0, 0, .2, 1, 0),
    vjust2 = c(0, 1, 1, 1, 0)
  )

colors = darken(c("#D55E00", "#009E73"), c(0.1, 0.1))

p1 <- ggplot(filter(pca_data, date >= ymd("1990-01-01"))) +
  aes(x=PC1, y=PC2, color=type, alpha = date, group = 1) +
  geom_path(size = 1, lineend = "butt") +
  geom_text_repel(
    data = pca_labels,
    aes(label = label),
    alpha = 1,
    point.padding = .2, color = "black",
    min.segment.length = 0, size = 12/.pt,
    family = dviz_font_family,
    nudge_x = pca_labels$nudge_x,
    nudge_y = pca_labels$nudge_y,
    hjust = pca_labels$hjust,
    vjust = pca_labels$vjust
  ) +
  scale_color_manual(
    values = colors,
    name = NULL
  ) +
  scale_alpha_date(range = c(0.45, 1), guide = "none") +
  scale_x_continuous(limits = c(-2, 15.5), name = "PC 1") +
  scale_y_continuous(limits = c(-5, 5), name = "PC 2") +
  theme_dviz_grid(14) +
  theme(
    legend.position = c(1, 1),
    legend.justification = c(1, 1),
    legend.direction = "horizontal",
    legend.box.background = element_rect(color = NA, fill = "white"),
    legend.box.margin = margin(6, 0, 6, 0),
    plot.margin = margin(3, 1.5, 6, 1.5)
  )

p2 <- ggplot(filter(pca_data, date >= ymd("1990-01-01"))) +
  aes(x=PC3, y=PC2, color=type, alpha = date, group = 1) +
  geom_path(size = 1, lineend = "butt") +
  geom_text_repel(
    data = pca_labels,
    aes(label = label),
    alpha = 1,
    point.padding = .2, color = "black",
    min.segment.length = 0, size = 12/.pt,
    family = dviz_font_family,
    nudge_x = pca_labels$nudge_x2,
    nudge_y = pca_labels$nudge_y2,
    hjust = pca_labels$hjust2,
    vjust = pca_labels$vjust2
  ) +
  scale_color_manual(
    values = colors,
    name = NULL
  ) +
  scale_alpha_date(range = c(0.45, 1), guide = "none") +
  scale_x_continuous(limits = c(-6, 8.5), name = "PC 3") +
  scale_y_continuous(limits = c(-5, 5), name = "PC 2") +
  theme_dviz_grid(14) +
  theme(
    legend.position = c(1, 1),
    legend.justification = c(1, 1),
    legend.direction = "horizontal",
    legend.box.background = element_rect(color = NA, fill = "white"),
    legend.box.margin = margin(6, 0, 6, 0),
    plot.margin = margin(6, 1.5, 3, 1.5)
  )

plot_grid(p1, p2, labels = "auto", ncol = 1)
```

Notably, Figure \@ref(fig:fred-md-PCA)a looks almost like a regular line plot, with time running from left to right. This pattern is caused by a common feature of PCA: The first component often measures the overall size of the system. Here, PC 1 approximately measures the overall size of the economy, which rarely decreases over time.

By coloring the connected scatter plot by times of recession and recovery, we can see that recessions are associated with a drop in PC 2 whereas recoveries do not correspond to a clear feature in either PC 1 or PC 2 (Figure \@ref(fig:fred-md-PCA)a). The recoveries do, however, seem to correspond to a drop in PC 3 (Figure \@ref(fig:fred-md-PCA)b). Moreover, in the PC 2 versus PC 3 plot, we see that the line follows the shape of a clockwise spiral. This pattern emphasizes the cyclical nature of the economy, with recessions following recoveries and vice versa.

<!--chapter:end:time_series.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")
library(forcats)
```


# Visualizing amounts {#visualizing-amounts}


In many scenarios, we are interested in the magnitude of some set of numbers. For example, we might want to visualize the total sales volume of different brands of cars, or the total number of people living in different cities, or the age of Olympians performing different sports. In all these cases, we have a set of categories (e.g., brands of cars, cities, or sports) and a quantitative value for each category. I refer to these cases as visualizing amounts, because the main emphasis in these visualizations will be on the magnitude of the quantitative values. The standard visualization in this scenario is the bar plot, which comes in several variations, including simple bars as well as grouped and stacked bars. Alternatives to the bar plot are the dot plot and the heatmap.

## Bar plots

To motivate the concept of a bar plot, consider the total ticket sales for the most popular movies on a given weekend. Table \@ref(tab:boxoffice-gross) shows the top-five weekend gross ticket sales on the Christmas weekend of 2017. The movie "Star Wars: The Last Jedi" was by far the most popular movie on that weekend, outselling the fourth- and fifth-ranked movies "The Greatest Showman" and "Ferdinand" by almost a factor of 10.

```{r boxoffice-gross}
# source: Box Office Mojo
# URL: http://www.boxofficemojo.com/weekend/chart/?view=&yr=2017&wknd=51&p=.htm
# downloaded: 2018-02-11

boxoffice <- data.frame(rank = 1:5,
                        title = c("Star Wars: The Last Jedi", "Jumanji: Welcome to the Jungle", "Pitch Perfect 3", "The Greatest Showman", "Ferdinand"),
                        title_short = c("Star Wars", "Jumanji", "Pitch Perfect 3", "Greatest Showman", "Ferdinand"),
                        amount = c(71565498, 36169328, 19928525, 8805843, 7316746),
                        amount_text = c("$71,565,498", "$36,169,328", "$19,928,525", "$8,805,843", "$7,316,746"))

boxoffice_display <- boxoffice %>%
  mutate(A = " ", B = " ", C = " ") %>%
  select(A, rank, title, amount_text, B, C) %>%
  rename(` ` = A,
         Rank = rank,
         Title = title,
         `Weekend gross` = amount_text,
         ` ` = B,
         ` ` = C)

knitr::kable(
  boxoffice_display,
  caption = 'Highest grossing movies for the weekend of December 22-24, 2017. Data source: Box Office Mojo (http://www.boxofficemojo.com/). Used with permission', booktabs = TRUE,
  row.names = FALSE,
  align = c('c', 'c', 'l', 'r', 'c', 'c')#,
  #format = "html",
  #table.attr = "style = \"width: 75%\""
)
```

This kind of data is commonly visualized with vertical bars. For each movie, we draw a bar that starts at zero and extends all the way to the dollar value for that movie's weekend gross (Figure \@ref(fig:boxoffice-vertical)). This visualization is called a *bar plot* or *bar chart*.

(ref:boxoffice-vertical) Highest grossing movies for the weekend of December 22-24, 2017, displayed as a bar plot. Data source: Box Office Mojo (http://www.boxofficemojo.com/). Used with permission

```{r boxoffice-vertical, fig.width = 5*6/4.2, fig.asp = .5, fig.cap = '(ref:boxoffice-vertical)'}
boxoffice %>%
  ggplot(aes(x = fct_reorder(title_short, rank), y = amount)) +
    geom_col(fill = "#56B4E9", width = 0.6, alpha = 0.9) +
    scale_y_continuous(expand = c(0, 0),
                       breaks = c(0, 2e7, 4e7, 6e7),
                       labels = c("0", "20", "40", "60"),
                       name = "weekend gross (million USD)") +
    scale_x_discrete(name = NULL,
                     expand = c(0, 0.4)) +
    coord_cartesian(clip = "off") +
    theme_dviz_hgrid(12, rel_small = 1) +
    theme(
      #axis.ticks.length = grid::unit(0, "pt"),
      axis.line.x = element_blank(),
      axis.ticks.x = element_blank()
    )
```

One problem we commonly encounter with vertical bars is that the labels identifying each bar take up a lot of horizontal space. In fact, I had to make Figure \@ref(fig:boxoffice-vertical) fairly wide and space out the bars so that I could place the movie titles underneath. To save horizontal space, we could place the bars closer together and rotate the labels (Figure \@ref(fig:boxoffice-rot-axis-tick-labels)). However, I am not a big proponent of rotated labels. I find the resulting plots awkward and difficult to read. And, in my experience, whenever the labels are too long to place horizontally they also don't look good rotated.

(ref:boxoffice-rot-axis-tick-labels) Highest grossing movies for the weekend of December 22-24, 2017, displayed as a bar plot with rotated axis tick labels. Rotated axis tick labels tend to be difficult to read and require awkward space use underneath the plot. For these reasons, I generally consider plots with rotated tick labels to be ugly. Data source: Box Office Mojo (http://www.boxofficemojo.com/). Used with permission

```{r boxoffice-rot-axis-tick-labels, fig.asp = 0.85, fig.cap = '(ref:boxoffice-rot-axis-tick-labels)'}
boxoffice %>%
  ggplot(aes(x = fct_reorder(title_short, rank), y = amount)) +
    geom_col(fill = "#56B4E9", alpha = 0.9) +
    scale_y_continuous(expand = c(0, 0),
                       breaks = c(0, 2e7, 4e7, 6e7),
                       labels = c("0", "20", "40", "60"),
                       name = "weekend gross (million USD)") +
    scale_x_discrete(name = NULL) +
    coord_cartesian(clip = "off") +
    theme_dviz_hgrid(rel_small = 1) +
    theme(
      #axis.ticks.length = grid::unit(0, "pt"),
      axis.line.x = element_blank(),
      axis.ticks.x = element_blank(),
      axis.text.x = element_text(angle = 45, vjust = 1, hjust = 1),
      plot.margin = margin(3, 7, 3, 1.5)
    ) -> p_box_axrot

stamp_ugly(p_box_axrot)
```

The better solution for long labels is usually to swap the *x* and the *y* axis, so that the bars run horizontally (Figure \@ref(fig:boxoffice-horizontal)). After swapping the axes, we obtain a compact figure in which all visual elements, including all text, are horizontally oriented. As a result, the figure is much easier to read than Figure \@ref(fig:boxoffice-rot-axis-tick-labels) or even Figure \@ref(fig:boxoffice-vertical).

(ref:boxoffice-horizontal) Highest grossing movies for the weekend of December 22-24, 2017, displayed as a horizontal bar plot. Data source: Box Office Mojo (http://www.boxofficemojo.com/). Used with permission

```{r boxoffice-horizontal, fig.cap = '(ref:boxoffice-horizontal)'}
ggplot(boxoffice, aes(x = fct_reorder(title_short, desc(rank)), y = amount)) +
  geom_col(fill = "#56B4E9", alpha = 0.9) +
  scale_y_continuous(limits = c(0, 7.5e7),
                     expand = c(0, 0),
                     breaks = c(0, 2e7, 4e7, 6e7),
                     labels = c("0", "20", "40", "60"),
                     name = "weekend gross (million USD)") +
  scale_x_discrete(name = NULL,
                   expand = c(0, 0.5)) +
  coord_flip(clip = "off") +
  theme_dviz_vgrid(rel_small = 1) +
  theme(
    #axis.ticks.length = grid::unit(0, "pt"),
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank()
  )
```

Regardless of whether we place bars vertically or horizontally, we need to pay attention to the order in which the bars are arranged. I often see bar plots where the bars are arranged arbitrarily or by some criterion that is not meaningful in the context of the figure. Some plotting programs arrange bars by default in alphabetic order of the labels, and other, similarly arbitrary arrangements are possible (Figure \@ref(fig:boxoffice-horizontal-bad-order)). In general, the resulting figures are more confusing and less intuitive than figures where bars are arranged in order of their size.  


(ref:boxoffice-horizontal-bad-order) Highest grossing movies for the weekend of December 22-24, 2017, displayed as a horizontal bar plot. Here, the bars have been placed in descending order of the lengths of the movie titles. This arrangement of bars is arbitrary, it doesn't serve a meaningful purpose, and it makes the resulting figure much less intuitive than Figure \@ref(fig:boxoffice-horizontal). Data source: Box Office Mojo (http://www.boxofficemojo.com/). Used with permission

```{r boxoffice-horizontal-bad-order, fig.cap = '(ref:boxoffice-horizontal-bad-order)'}
p <- ggplot(boxoffice, aes(x = factor(title_short, levels = title_short[c(2, 1, 5, 3, 4)]),
                           y = amount)) +
  geom_col(fill = "#56B4E9", alpha = 0.9) +
  scale_y_continuous(limits = c(0, 7.5e7),
                     expand = c(0, 0),
                     breaks = c(0, 2e7, 4e7, 6e7),
                     labels = c("0", "20", "40", "60"),
                     name = "weekend gross (million USD)") +
  scale_x_discrete(name = NULL,
                   expand = c(0, 0.5)) +
  coord_flip(clip = "off") +
  theme_dviz_vgrid(rel_small = 1) +
  theme(
    #axis.ticks.length = grid::unit(0, "pt"),
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank()
  )

stamp_bad(p)
```

We should only rearrange bars, however, when there is no natural ordering to the categories the bars represent. Whenever there is a natural ordering (i.e., when our categorical variable is an ordered factor) we should retain that ordering in the visualization. For example, Figure \@ref(fig:income-by-age) shows the median annual income in the U.S. by age groups. In this case, the bars should be arranged in order of increasing age. Sorting by bar height while shuffling the age groups makes no sense (Figure \@ref(fig:income-by-age-sorted)).


(ref:income-by-age) 2016 median U.S. annual household income versus age group. The 45--54 year age group has the highest median income. Data source: United States Census Bureau

```{r income-by-age, fig.cap = '(ref:income-by-age)'}
income_by_age %>% filter(race == "all") %>%
  ggplot(aes(x = age, y = median_income)) +
    geom_col(fill = "#56B4E9", alpha = 0.9) +
    scale_y_continuous(expand = c(0, 0),
                       name = "median income (USD)",
                       breaks = c(0, 20000, 40000, 60000),
                       labels = c("$0", "$20,000", "$40,000", "$60,000")) +
    xlab("age (years)") +
    coord_cartesian(clip = "off") +
    theme_dviz_hgrid() +
    theme(
      #axis.ticks.length = grid::unit(0, "pt"),
      axis.ticks.x = element_blank(),
      axis.line = element_blank(),
      plot.margin = margin(3, 7, 3, 1.5)
    )
```

(ref:income-by-age-sorted) 2016 median U.S. annual household income versus age group, sorted by income. While this order of bars looks visually appealing, the order of the age groups is now confusing. Data source: United States Census Bureau

```{r income-by-age-sorted, fig.cap = '(ref:income-by-age-sorted)'}
income_by_age %>% filter(race == "all") %>%
  ggplot(aes(x = fct_reorder(age, desc(median_income)), y = median_income)) +
    geom_col(fill = "#56B4E9", alpha = 0.9) +
    scale_y_continuous(
      expand = c(0, 0),
      name = "median income (USD)",
      breaks = c(0, 20000, 40000, 60000),
      labels = c("$0", "$20,000", "$40,000", "$60,000")
    ) +
    coord_cartesian(clip = "off") +
    xlab("age (years)") +
    theme_dviz_hgrid() +
    theme(
      #axis.ticks.length = grid::unit(0, "pt"),
      axis.ticks.x = element_blank(),
      axis.line = element_blank(),
      plot.margin = margin(3, 7, 3, 1.5)
    ) -> p_income_sorted

stamp_bad(p_income_sorted)
```

```{block type='rmdtip', echo=TRUE}
Pay attention to the bar order. If the bars represent unordered categories, order them by ascending or descending data values.
```

## Grouped and stacked bars

All examples from the previous subsection showed how a quantitative amount varied with respect to one categorical variable. Frequently, however, we are interested in two categorical variables at the same time. For example, the U.S. Census Bureau provides median income levels broken down by both age and race. We can visualize this dataset with a *grouped bar plot* (Figure \@ref(fig:income-by-age-race-dodged)). In a grouped bar plot, we draw a group of bars at each position along the *x* axis, determined by one categorical variable, and then we draw bars within each group according to the other categorical variable.

(ref:income-by-age-race-dodged) 2016 median U.S. annual household income versus age group and race. Age groups are shown along the *x* axis, and for each age group there are four bars, corresponding to the median income of Asian, white, Hispanic, and black people, respectively. Data source: United States Census Bureau

```{r income-by-age-race-dodged, fig.width = 5.5*6/4.2, fig.asp = 0.5, fig.cap = '(ref:income-by-age-race-dodged)'}
income_by_age %>% filter(race %in% c("white", "asian", "black", "hispanic")) %>%
   mutate(race = fct_relevel(race, c("asian", "white", "hispanic", "black")),
          race = fct_recode(race, Asian = "asian", Hispanic = "hispanic"),
          age = fct_recode(age, "≥ 75" = "> 74")) -> income_df

# Take the darkest four colors from 5-class ColorBrewer palette "PuBu"
colors_four = RColorBrewer::brewer.pal(5, "PuBu")[5:2]

ggplot(income_df, aes(x = age, y = median_income, fill = race)) +
  geom_col(position = "dodge", alpha = 0.9) +
  scale_y_continuous(
    expand = c(0, 0),
    name = "median income (USD)",
    breaks = c(0, 20000, 40000, 60000, 80000, 100000),
    labels = c("$0", "$20,000", "$40,000", "$60,000", "$80,000", "$100,000")
  ) +
  scale_fill_manual(values = colors_four, name = NULL) +
  coord_cartesian(clip = "off") +
  xlab("age (years)") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank()
  ) -> p_income_race_dodged

#stamp_ugly(p_income_race_dodged)
p_income_race_dodged
```

Grouped bar plots show a lot of information at once and they can be confusing. In fact, even though I have not labeled Figure \@ref(fig:income-by-age-race-dodged) as bad or ugly, I find it difficult to read. In particular, it is difficult to compare median incomes across age groups for a given racial group. So this figure is only appropriate if we are primarily interested in the differences in income levels among racial groups, separately for specific age groups. If we care more about the overall pattern of income levels among racial groups, it may be preferable to show race along the *x* axis and show ages as distinct bars within each racial group (Figure \@ref(fig:income-by-race-age-dodged)).

(ref:income-by-race-age-dodged) 2016 median U.S. annual household income versus age group and race. In contrast to Figure \@ref(fig:income-by-age-race-dodged), now race is shown along the *x* axis, and for each race we show seven bars according to the seven age groups. Data source: United States Census Bureau

```{r income-by-race-age-dodged, fig.width = 5.5*6/4.2, fig.asp = 0.4, fig.cap = '(ref:income-by-race-age-dodged) '}
# Take the darkest seven colors from 8-class ColorBrewer palette "PuBu"
colors_seven = RColorBrewer::brewer.pal(8, "PuBu")[2:8]

ggplot(income_df, aes(x = race, y = median_income, fill = age)) +
  geom_col(position = "dodge", alpha = 0.9) +
  scale_y_continuous(
    expand = c(0, 0),
    name = "median income (USD)",
    breaks = c(0, 20000, 40000, 60000, 80000, 100000),
    labels = c("$0", "$20,000", "$40,000", "$60,000", "$80,000", "$100,000")
  ) +
  scale_fill_manual(values = colors_seven, name = "age (yrs)") +
  coord_cartesian(clip = "off") +
  xlab(label = NULL) +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank(),
    legend.title.align = 0.5
  ) -> p_income_age_dodged

p_income_age_dodged
```

Both Figures \@ref(fig:income-by-age-race-dodged) and \@ref(fig:income-by-race-age-dodged) encode one categorical variable by position along the *x* axis and the other by bar color. And in both cases, the encoding by position is easy to read while the encoding by bar color requires more mental effort, as we have to mentally match the colors of the bars against the colors in the legend. We can avoid this added mental effort by showing four separate regular bar plots rather than one grouped bar plot (Figure \@ref(fig:income-by-age-race-faceted)). Which of these various options we choose is ultimately a matter of taste. I would likely choose Figure \@ref(fig:income-by-age-race-faceted), because it circumvents the need for different bar colors.

(ref:income-by-age-race-faceted) 2016 median U.S. annual household income versus age group and race. Instead of displaying this data as a grouped bar plot, as in Figures \@ref(fig:income-by-age-race-dodged) and \@ref(fig:income-by-race-age-dodged), we now show the data as four separate regular bar plots. This choice has the advantage that we don't need to encode either categorical variable by bar color. Data source: United States Census Bureau

```{r income-by-age-race-faceted, fig.width = 5.5*6/4.2, fig.cap = '(ref:income-by-age-race-faceted)'}
income_df %>%
   mutate(age = fct_recode(age, "15–24" = "15 to 24", "25–34" = "25 to 34", "35–44" = "35 to 44",
                                "45–54" = "45 to 54", "55–64" = "55 to 64", "65–74" = "65 to 74")) -> income_age_abbrev_df

ggplot(income_age_abbrev_df, aes(x = age, y = median_income)) +
  geom_col(fill = "#56B4E9", alpha = 0.9) +
  scale_y_continuous(
    expand = c(0, 0),
    name = "median income (USD)",
    breaks = c(0, 20000, 40000, 60000, 80000, 100000),
    labels = c("$0", "$20,000", "$40,000", "$60,000", "$80,000", "$100,000")
  ) +
  coord_cartesian(clip = "off") +
  xlab(label = "age (years)") +
  facet_wrap(~race, scales = "free_x") +
  theme_dviz_hgrid(14) +
  theme(
    #axis.ticks.length = grid::unit(0, "pt"),
    axis.ticks.x = element_blank(),
    axis.line = element_blank(),
    strip.text = element_text(size = 14),
    panel.spacing.y = grid::unit(14, "pt")
  ) -> p_income_age_faceted

p_income_age_faceted
```

Instead of drawing groups of bars side-by-side, it is sometimes preferable to stack bars on top of each other. Stacking is useful when the sum of the amounts represented by the individual stacked bars is in itself a meaningful amount. So, while it would not make sense to stack the median income values of Figure \@ref(fig:income-by-age-race-dodged) (the sum of two median income values is not a meaningful value), it might make sense to stack the weekend gross values of Figure \@ref(fig:boxoffice-vertical) (the sum of the weekend gross values of two movies is the total gross for the two movies combined). Stacking is also appropriate when the individual bars represent counts. For example, in a dataset of people, we can either count men and women separately or we can count them together. If we stack a bar representing a count of women on top of a bar representing a count of men, then the combined bar height represents the total count of people regardless of gender.

I will demonstrate this principle using a dataset about the passengers of the transatlantic ocean liner Titanic, which sank on April 15, 1912. On board were approximately 1300 passengers, not counting crew. The passengers were traveling in one of three classes (1st, 2nd, or 3rd), and there were almost twice as many male as female passengers on the ship. To visualize the breakdown of passengers by class and gender, we can draw separate bars for each class and gender and stack the bars representing women on top of the bars representing men, separately for each class (Figure \@ref(fig:titanic-passengers-by-class-sex)). The combined bars represent the total number of passengers in each class.

(ref:titanic-passengers-by-class-sex) Numbers of female and male passengers on the Titanic traveling in 1st, 2nd, and 3rd class.

```{r titanic-passengers-by-class-sex, fig.width = 5.5, fig.cap = '(ref:titanic-passengers-by-class-sex)'}
titanic_groups <- titanic_all %>% filter(class != "*") %>% 
  select(class, sex) %>% 
  group_by(class, sex) %>% 
  tally() %>% arrange(class, desc(sex)) %>%
  mutate(sex = factor(sex, levels = c("female", "male"))) %>%
  group_by(class) %>%
  mutate(nlabel = cumsum(n) - n/2) %>%
  ungroup() %>%
  mutate(class = paste(class, "class"))

ggplot(titanic_groups, aes(x = class, y = n, fill = sex)) +
  geom_col(position = "stack", color = "white", size = 1, width = 1) +
  geom_text(
    aes(y = nlabel, label = n), color = "white", size = 14/.pt,
    family = dviz_font_family
  ) +
  scale_x_discrete(expand = c(0, 0), name = NULL) +
  scale_y_continuous(expand = c(0, 0), breaks = NULL, name = NULL) +
  scale_fill_manual(
    values = c("#D55E00", "#0072B2"),
    breaks = c("female", "male"),
    labels = c("female passengers   ", "male passengers"),
    name = NULL
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_grid() +
  theme(
    panel.grid.major = element_blank(),
    axis.ticks = element_blank(),
    axis.text = element_text(size = 14),
    legend.position = "bottom",
    legend.justification = "center",
    legend.background = element_rect(fill = "white"),
    legend.spacing.x = grid::unit(4.5, "pt"),
    legend.spacing.y = grid::unit(0, "cm"),
    legend.box.spacing = grid::unit(7, "pt")
  )
```

Figure \@ref(fig:titanic-passengers-by-class-sex) differs from the previous bar plots I have shown in that there is no explicit *y* axis. I have instead shown the actual numerical values that each bar represents. Whenever a plot is meant to display only a small number of different values, it makes sense to add the actual numbers to the plot. This substantially increases the amount of information conveyed by the plot without adding much visual noise, and it removes the need for an explicit *y* axis.


## Dot plots and heatmaps

Bars are not the only option for visualizing amounts. One important limitation of bars is that they need to start at zero, so that the bar length is proportional to the amount shown. For some datasets, this can be impractical or may obscure key features. In this case, we can indicate amounts by placing dots at the appropriate locations along the *x* or *y* axis.

Figure \@ref(fig:Americas-life-expect) demonstrates this visualization approach for a dataset of life expectancies in 25 countries in the Americas. The citizens of these countries have life expectancies between 60 and 81 years, and each individual life expectancy value is shown with a blue dot at the appropriate location along the *x* axis. By limiting the axis range to the interval from 60 to 81 years, the figure highlights the key features of this dataset: Canada has the highest life expectancy among all listed countries, and Bolivia and Haiti have much lower life expectancies than all other countries. If we had used bars instead of dots (Figure \@ref(fig:Americas-life-expect-bars)), we'd have made a much less compelling figure. Because the bars are so long in this figure, and they all have nearly the same length, the eye is drawn to the middle of the bars rather than to their end points, and the figure fails to convey its message.

(ref:Americas-life-expect) Life expectancies of countries in the Americas, for the year 2007. Data source: Gapminder project

```{r Americas-life-expect, fig.width = 6., fig.asp = .9, fig.cap = '(ref:Americas-life-expect)'}
library(gapminder)
df_Americas <- gapminder %>% filter(year == 2007, continent == "Americas")
ggplot(df_Americas, aes(x = lifeExp, y = fct_reorder(country, lifeExp))) +
  geom_point(color = "#0072B2", size = 3) +
  scale_x_continuous(
    name = "life expectancy (years)",
    limits = c(59.7, 81.5),
    expand = c(0, 0)
  ) +
  scale_y_discrete(name = NULL, expand = c(0, 0.5)) +
  theme_dviz_grid(12, rel_small = 1) +
  theme(
    #axis.ticks.length = grid::unit(0, "pt"),
    #axis.title = element_text(size = 12),
    plot.margin = margin(18, 6, 3, 1.5)
  )
```

(ref:Americas-life-expect-bars) Life expectancies of countries in the Americas, for the year 2007, shown as bars. This dataset is not suitable for being visualized with bars. The bars are too long and they draw attention away from the key feature of the data, the differences in life expectancy among the different countries. Data source: Gapminder project

```{r Americas-life-expect-bars, fig.width = 6., fig.asp = .9, fig.cap = '(ref:Americas-life-expect-bars)'}
life_bars <- ggplot(df_Americas, aes(y = lifeExp, x = fct_reorder(country, lifeExp))) +
  geom_col(fill = "#56B4E9", alpha = 0.9) +
  scale_y_continuous(
    name = "life expectancy (years)",
    limits = c(0, 85),
    expand = c(0, 0)
  ) +
  scale_x_discrete(name = NULL, expand = c(0, 0.5)) +
  coord_flip(clip = "off") +
  theme_dviz_vgrid(12, rel_small = 1) +
  theme(
    #axis.ticks.length = grid::unit(0, "pt"),
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank(),
    #axis.title = element_text(size = 12),
    plot.margin = margin(18, 6, 3, 1.5)
  )

stamp_bad(life_bars)
```

Regardless of whether we use bars or dots, however, we need to pay attention to the ordering of the data values. In Figures \@ref(fig:Americas-life-expect) and \@ref(fig:Americas-life-expect-bars), the countries are ordered in descending order of life expectancy. If we instead ordered them alphabetically, we'd end up with a disordered cloud of points that is confusing and fails to convey a clear message (Figure \@ref(fig:Americas-life-expect-bad)). 

(ref:Americas-life-expect-bad) Life expectancies of countries in the Americas, for the year 2007. Here, the countries are ordered alphabetically, which causes a dots to form a disordered cloud of points. This makes the figure difficult to read, and therefore it deserves to be labeled as "bad." Data source: Gapminder project

```{r Americas-life-expect-bad, fig.width = 6., fig.asp = .9, fig.cap = '(ref:Americas-life-expect-bad)'}
p <- ggplot(df_Americas, aes(x = lifeExp, y = fct_rev(country))) +
  geom_point(color = "#0072B2", size = 3) +
  scale_x_continuous(name = "life expectancy (years)",
                     limits = c(59.7, 81.5),
                     expand = c(0, 0)) +
  scale_y_discrete(name = NULL, expand = c(0, 0.5)) +
  theme_dviz_grid(12, rel_small = 1) +
  theme(#axis.ticks.length = grid::unit(0, "pt"),
        #axis.title = element_text(size = 12),
        plot.margin = margin(18, 6, 3, 1.5))

stamp_bad(p)
```

All examples so far have represented amounts by location along a position scale, either through the end point of a bar or the placement of a dot. For very large datasets, neither of these options may be appropriate, because the resulting figure would become too busy. We had already seen in Figure \@ref(fig:income-by-age-race-dodged) that just seven groups of four data values can result in a figure that is complex and not that easy to read. If we had 20 groups of 20 data values, a similar figure would likely be highly confusing.

As an alternative to mapping data values onto positions via bars or dots, we can map data values onto colors. Such a figure is called a *heatmap*. Figure \@ref(fig:internet-over-time) uses this approach to show the percentage of internet users over time in 20 countries and for 23 years, from 1994 to 2016. While this visualization makes it harder to determine the exact data values shown (e.g., what's the exact percentage of internet users in the United States in 2015?), it does an excellent job of highlighting broader trends. We can see clearly in which countries internet use began early and which it did not, and we can also see clearly which countries have high internet penetration in the final year covered by the dataset (2016).

(ref:internet-over-time) Internet adoption over time, for select countries. Color represents the percent of internet users for the respective country and year. Countries were ordered by percent internet users in 2016. Data source: World Bank

```{r internet-over-time, fig.width = 5.5*6/4.2, fig.cap = '(ref:internet-over-time)'}
country_list = c("United States", "China", "India", "Japan", "Algeria",
                 "Brazil", "Germany", "France", "United Kingdom", "Italy", "New Zealand",
                 "Canada", "Mexico", "Chile", "Argentina", "Norway", "South Africa", "Kenya",
                 "Israel", "Iceland")

internet_short <- filter(internet, country %in% country_list) %>%
  mutate(users = ifelse(is.na(users), 0, users))

internet_summary <- internet_short %>%
  group_by(country) %>%
  summarize(year1 = min(year[users > 0]),
            last = users[n()]) %>%
  arrange(last, desc(year1))

internet_short <- internet_short %>%
  mutate(country = factor(country, levels = internet_summary$country))

ggplot(filter(internet_short, year > 1993),
       aes(x = year, y = country, fill = users)) +
  geom_tile(color = "white", size = 0.25) +
  scale_fill_viridis_c(
    option = "A", begin = 0.05, end = 0.98,
    limits = c(0, 100),
    name = "internet users / 100 people",
    guide = guide_colorbar(
      direction = "horizontal",
      label.position = "bottom",
      title.position = "top",
      ticks = FALSE,
      barwidth = grid::unit(3.5, "in"),
      barheight = grid::unit(0.2, "in")
    )
  ) +
  scale_x_continuous(expand = c(0, 0), name = NULL) +
  scale_y_discrete(name = NULL, position = "right") +
  theme_dviz_open(12) +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = grid::unit(1, "pt"),
    legend.position = "top",
    legend.justification = "left",
    legend.title.align = 0.5,
    legend.title = element_text(size = 12*12/14)
  )
```

As is the case with all other visualization approaches discussed in this chapter, we need to pay attention to the ordering of the categorical data values when making heatmaps. In Figure \@ref(fig:internet-over-time), countries are ordered by the percentage of internet users in 2016. This ordering places the United Kingdom, Japan, Canada, and Germany above the United States, because all these countries have higher internet penetration in 2016 than the United States does, even though the United States saw significant internet use at an earlier time. Alternatively, we could order countries by how early they started to see significant internet usage. In Figure \@ref(fig:internet-over-time2), countries are ordered by the year in which internet usage first rose to above 20%. In this figure, the United States falls into the third position from the top, and it stands out for having relatively low internet usage in 2016 compared to how early internet usage started there. A similar pattern can be seen for Italy. Israel and France, by contrast, started relatively late but gained ground rapidly.

(ref:internet-over-time2) Internet adoption over time, for select countries. Countries were ordered by the year in which their internet usage first exceeded 20%. Data source: World Bank


```{r internet-over-time2, fig.width = 5.5*6/4.2, fig.cap = '(ref:internet-over-time2)'}
internet_summary <- internet_short %>%
  group_by(country) %>%
  summarize(year1 = min(year[users > 20]),
            last = users[n()]) %>%
  arrange(desc(year1), last)

internet_short <- internet_short %>%
  mutate(country = factor(country, levels = internet_summary$country))

ggplot(filter(internet_short, year > 1993),
       aes(x = year, y = country, fill = users)) +
  geom_tile(color = "white", size = 0.25) +
  scale_fill_viridis_c(
    option = "A", begin = 0.05, end = 0.98,
    limits = c(0, 100),
    name = "internet users / 100 people",
    guide = guide_colorbar(
      direction = "horizontal",
      label.position = "bottom",
      title.position = "top",
      ticks = FALSE,
      barwidth = grid::unit(3.5, "in"),
      barheight = grid::unit(0.2, "in")
    )
  ) +
  scale_x_continuous(expand = c(0, 0), name = NULL) +
  scale_y_discrete(name = NULL, position = "right") +
  theme_dviz_open(12) +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = grid::unit(1, "pt"),
    legend.position = "top",
    legend.justification = "left",
    legend.title.align = 0.5,
    legend.title = element_text(size = 12*12/14)
  )
```

Both Figures \@ref(fig:internet-over-time) and \@ref(fig:internet-over-time2) are valid representations of the data. Which one is preferred depends on the story we want to convey. If our story is about internet usage in 2016, then Figures \@ref(fig:internet-over-time) is probably the better choice. If, however, our story is about how early or late adoption of the internet relates to current-day usage, then Figure \@ref(fig:internet-over-time2) is preferable.

<!--chapter:end:visualizing_amounts.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")

library(dplyr)
library(tidyr)
```

# Visualizing associations among two or more quantitative variables {#visualizing-associations}

Many datasets contain two or more quantitative variables, and we may be interested in how these variables relate to each other. For example, we may have a dataset of quantitative measurements of different animals, such as the animals' height, weight, length, and daily energy demands. To plot the relationship of just two such variables, e.g. the height and weight, we will normally use a scatter plot. If we want to show more than two variables at once, we may opt for a bubble chart, a scatter plot matrix, or a correlogram. Finally, for very high-dimensional datasets, it may be useful to perform dimension reduction, for example in the form of principal components analysis.

## Scatter plots {#associations-scatterplots}

I will demonstrate the basic scatter plot and several variations thereof using a dataset of measurements performed on 123 blue jay birds. The dataset contains information such as the head length (measured from the tip of the bill to the back of the head), the skull size (head length minus bill length), and the body mass of each bird. We expect that there are relationships between these variables. For example, birds with longer bills would be expected to have larger skull sizes, and birds with higher body mass should have larger bills and skulls than birds with lower body mass.

To explore these relationships, I begin with a plot of head length against body mass (Figure \@ref(fig:blue-jays-scatter)). In this plot, head length is shown along the *y* axis, body mass along the *x* axis, and each bird is represented by one dot. (Note the terminology: We say that we plot the variable shown along the *y* axis against the variable shown along the *x* axis.) The dots form a dispersed cloud (hence the term *scatter plot*), yet undoubtedly there is a trend for birds with higher body mass to have longer heads. The bird with the longest head falls close to the maximum body mass observed, and the bird with the shortest head falls close to the minimum body mass observed.

(ref:blue-jays-scatter) Head length (measured from the tip of the bill to the back of the head, in mm) versus body mass (in gram), for 123 blue jays. Each dot corresponds to one bird. There is a moderate tendency for heavier birds to have longer heads. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-scatter, fig.width = 5, fig.asp = 3/4, fig.cap='(ref:blue-jays-scatter)'}
ggplot(blue_jays, aes(Mass, Head)) + 
  geom_point(pch = 21, fill = "gray25", color = "white", size = 2.5) +
  scale_x_continuous(name = "body mass (g)") +
  scale_y_continuous(name = "head length (mm)") +
  theme_dviz_grid()
```

The blue jay dataset contains both male and female birds, and we may want to know whether the overall relationship between head length and body mass holds up separately for each sex. To address this question, we can color the points in the scatter plot by the sex of the bird (Figure \@ref(fig:blue-jays-scatter-sex)). This figure reveals that the overall trend in head length and body mass is at least in part driven by the sex of the birds. At the same body mass, females tend to have shorter heads than males. At the same time, females tend to be lighter than males on average.

(ref:blue-jays-scatter-sex) Head length versus body mass for 123 blue jays. The birds' sex is indicated by color. At the same body mass, male birds tend to have longer heads (and specifically, longer bills) than female birds. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-scatter-sex, fig.width = 5, fig.asp = 3.2/4, fig.cap='(ref:blue-jays-scatter-sex)'}
ggplot(blue_jays, aes(Mass, Head, fill = KnownSex)) + 
  geom_point(pch = 21, color = "white", size = 2.5) +
  scale_x_continuous(name = "body mass (g)") +
  scale_y_continuous(name = "head length (mm)") +
  scale_fill_manual(
    values = c(F = "#D55E00", M = "#0072B2"),
    breaks = c("F", "M"),
    labels = c("female birds   ", "male birds"),
    name = NULL,
    guide = guide_legend(
      direction = "horizontal",
      override.aes = list(size = 3)
      )
    ) +
  theme_dviz_grid() +
  theme(
    #legend.position = c(1, 0.01),
    #legend.justification = c(1, 0),
    legend.position = "top",
    legend.justification = "right",
    legend.box.spacing = unit(3.5, "pt"), # distance between legend and plot
    legend.text = element_text(vjust = 0.6),
    legend.spacing.x = unit(2, "pt"),
    legend.background = element_rect(fill = "white", color = NA),
    legend.key.width = unit(10, "pt")
  )
```

Because the head length is defined as the distance from the tip of the bill to the back of the head, a larger head length could imply a longer bill, a larger skull, or both. We can disentangle bill length and skull size by looking at another variable in the dataset, the skull size, which is similar to the head length but excludes the bill. As we are already using the *x* position for body mass, the *y* position for head length, and the dot color for bird sex, we need another aesthetic to which we can map skull size. One option is to use the size of the dots, resulting in a visualization called a *bubble chart* (Figure \@ref(fig:blue-jays-scatter-bubbles)).


(ref:blue-jays-scatter-bubbles) Head length versus body mass for 123 blue jays. The birds' sex is indicated by color, and the birds' skull size by symbol size. Head-length measurements include the length of the bill while skull-size measurements do not. Head length and skull size tend to be correlated, but there are some birds with unusually long or short bills given their skull size. Data source: Keith Tarvin, Oberlin College


```{r blue-jays-scatter-bubbles, fig.width = 5.5*6/4.2, fig.asp = 3.2/8, fig.cap='(ref:blue-jays-scatter-bubbles)'}
blue_jays$sex <- ifelse(blue_jays$KnownSex == "F", "female birds", "male birds")
blue_jays$sex <- factor(blue_jays$sex, levels = c("female birds", "male birds"))
ggplot(blue_jays, aes(Mass, Head, size = Skull, fill = KnownSex)) + 
  geom_point(pch = 21, color = "white") +
  facet_wrap(~sex, ncol = 2, scales = "fixed") +
  scale_x_continuous(name = "body mass (g)") +
  scale_y_continuous(name = "head length (mm)", breaks = c(52, 54, 56, 58, 60)) +
  scale_fill_manual(values = c(F = "#D55E00", M = "#0072B2"), guide = "none") +
  scale_radius(
    name = "skull size (mm)",
    range = c(2, 7),
    limits = c(28, 34),
    breaks = c(28, 30, 32, 34),
    labels = c("28   ", "30   ", "32   ", "34"),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      title.hjust = 0.5,
      label.position = "right",
      override.aes = list(fill = "gray40")
    )
  ) +
  theme_dviz_grid(12) +
  theme(
    legend.position = c(1, 0.03),
    legend.justification = c(1, 0),
    legend.spacing.x = unit(2, "pt"),
    legend.spacing.y = unit(2, "pt"),
    legend.background = element_rect(fill = "white", color = NA),
    legend.key.width = unit(10, "pt"),
    #strip.text = element_text(size = 14, family = "Myriad Pro Semibold")
    strip.text = element_text(size = 12, margin = margin(2, 0, 2, 0)),
    strip.background  = element_rect(
      fill = "grey85", colour = "grey85",
      linetype = 1, size = 0.25
    )
  )
```

Bubble charts have the disadvantage that they show the same types of variables, quantitative variables, with two different types of scales, position and size. This makes it difficult to visually ascertain the strengths of associations between the various variables. Moreover, differences between data values encoded as bubble size are harder to perceive than differences between data values encoded as position. Because even the largest bubbles need to be somewhat small compared to the total figure size, the size differences between even the largest and the smallest bubbles are necessarily small. Consequently, smaller differences in data values will correspond to very small size differences that can be virtually impossible to see. In 
Figure \@ref(fig:blue-jays-scatter-bubbles), I used a size mapping that visually amplified the difference between the smallest skulls (around 28mm) and the largest skulls (around 34mm), and yet it is difficult to determine what the relationship is between skull size and either body mass or head length.

As an alternative to a bubble chart, it may be preferable to show an all-against-all matrix of scatter plots, where each individual plot shows two data dimensions (Figure \@ref(fig:blue-jays-scatter-all)). This figure shows clearly that the relationship between skull size and body mass is comparable for female and male birds except that the female birds tend to be somewhat smaller. However, the same is not true for the relationship between head length and body mass. There is a clear separation by sex. Male birds tend to have longer bills than female birds, all else equal.


(ref:blue-jays-scatter-all) All-against-all scatter plot matrix of head length, body mass, and skull size, for 123 blue jays. This figure shows the exact same data as Figure \@ref(fig:blue-jays-scatter-sex). However, because we are better at judging position than symbol size, correlations between skull size and the other two variables are easier to perceive in the pairwise scatter plots than in Figure \@ref(fig:blue-jays-scatter-sex). Data source: Keith Tarvin, Oberlin College


```{r blue-jays-scatter-all, fig.width = 5*6/4.2, fig.asp = 3/4, fig.cap='(ref:blue-jays-scatter-all)'}
blue_jays %>% select(BirdID, KnownSex, Head, Mass, Skull) %>%
  gather(var_x, val_x, Head:Skull) %>%
  left_join(select(blue_jays, BirdID, Head, Mass, Skull)) %>%
  gather(var_y, val_y, Head:Skull) -> bj_matrix

labels <- c(
  Head = "head length (mm)",
  Mass = "body mass (g)",
  Skull = "skull size (mm)"
)

ggplot(bj_matrix, aes(val_x, val_y, fill = KnownSex)) + 
  geom_point(pch = 21, color = "white", size = 2, stroke = 0.2) +
  scale_x_continuous(
    expand = expand_scale(mult = 0.1),
    breaks = scales::pretty_breaks(4, min.n = 3)
  ) +
  scale_y_continuous(
    expand = expand_scale(mult = 0.1),
    breaks = scales::pretty_breaks(4, min.n = 3)
  ) +
  scale_fill_manual(
    values = c(F = "#D55E00D0", M = "#0072B2D0"),
    breaks = c("F", "M"),
    labels = c("female birds   ", "male birds"),
    name = NULL,
    guide = guide_legend(
      direction = "horizontal",
      override.aes = list(size = 2.5)
      )
    ) +
  labs(x = NULL, y = NULL) +
  facet_grid(
    var_y ~ var_x,
    scales = "free",
    switch = "both",
    labeller = labeller(
      var_x = labels,
      var_y = labels
    )
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_grid(line_size = 0.3) +
  panel_border(colour = "grey85", size = 0.4) +
  theme(
    strip.placement = "outside",
    strip.text.x = element_text(vjust = 1, margin = margin(0, 0, 0, 0)),
    strip.text.y = element_text(vjust = 0, angle = -90, margin = margin(0, 3.5, 0, 0)),
    legend.box.spacing = grid::unit(0, "pt"),
    legend.text = element_text(vjust = 0.6),
    legend.position = "top",
    legend.justification = "right",
    legend.spacing.x = unit(2, "pt"),
    legend.key.width = unit(10, "pt"),
    plot.margin = margin(3.5, 1.5, 3.5, 1.5)
  )
```


## Correlograms {#associations-correlograms}

When we have more than three to four quantitative variables, all-against-all scatter plot matrices quickly become unwieldy. In this case, it is more useful to quantify the amount of association between pairs of variables and visualize this quantity rather than the raw data. One common way to do this is to calculate *correlation coefficients*. The correlation coefficient *r* is a number between -1 and 1 that measures to what extent two variables covary. A value of *r* = 0 means there is no association whatsoever, and a value of either 1 or -1 indicates a perfect association. The sign of the correlation coefficient indicates whether the variables are *correlated* (larger values in one variable coincide with larger values in the other) or *anticorrelated* (larger values in one variable coincide with smaller values in the other). To provide visual examples of what different correlation strengths look like, in Figure \@ref(fig:correlations) I show randomly generated sets of points that differ widely in the degree to which the *x* and *y* values are correlated.

(ref:correlations) Examples of correlations of different magnitude and direction, with associated correlation coefficient *r*. In both rows, from left to right correlations go from weak to strong. In the top row the correlations are positive (larger values for one quantity are associated with larger values for the other) and in the bottom row they are negative (larger values for one quantity are associated with smaller values for the other). In all six panels, the sets of *x* and *y* values are identical, but the pairings between individual *x* and *y* values have been reshuffled to generate the specified correlation coefficients.

```{r correlations, fig.width = 5*6/4.2, fig.asp = 283/434, fig.cap = '(ref:correlations)'}
r <- c(.2, .6, .9, -.2, -.6, -.9)
labels_df <- tibble(
  r = factor(r, levels = r),
  #label = paste0("r = ", r),
  label = c("r = 0.2", "r = 0.6", "r = 0.9", "r = –0.2", "r = –0.6", "r = –0.9"),
  x = c(-2.3, -2.3, -2.3, 2.3, 2.3, 2.3),
  hjust = c(0, 0, 0, 1, 1, 1),
  y = 2.3
)

ggplot(correlation_examples, aes(x, y)) + 
  geom_point(pch = 21, color = "white", size = 2, fill = "#0072B2") + 
  geom_text(
    data = labels_df,
    aes(label = label, hjust = hjust),
    vjust = 1,
    family = "Myriad Pro",
    size = 14/.pt
  ) +
  facet_wrap(~r) +
  coord_fixed(xlim = c(-2.5, 2.5), ylim = c(-2.5, 2.5), expand = FALSE, clip = "off") +
  theme_dviz_map() +
  theme(
    strip.text = element_blank(),
    panel.spacing = grid::unit(12, "pt"),
    plot.margin = margin(3.5, 1.5, 3.5, 1.5)
  ) +
  panel_border(colour = "grey80", size = 0.5)
```

The correlation coefficient is defined as

$$
r = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sqrt{\sum_i (x_i-\bar x)^2}\sqrt{\sum_i (y_i-\bar y)^2}},
$$
where $x_i$ and $y_i$ are two sets of observations and $\bar x$ and $\bar y$ are the corresponding sample means. We can make a number of observations from this formula. First, the formula is symmetric in $x_i$ and $y_i$, so the correlation of *x* with *y* is the same as the correlation of *y* with *x*. Second, the individual values $x_i$ and $y_i$ only enter the formula in the context of differences to the respective sample mean, so if we shift an entire dataset by a constant amount, e.g. we replace $x_i$ with $x_i' = x_i + C$ for some constant $C$, the correlation coefficient remains unchanged. Third, the correlation coefficient also remains unchanged if we rescale the data, $x_i' = C x_i$, since the constant $C$ will appear both in the numerator and the denominator of the formula and hence can be cancelled. 

Visualizations of correlation coefficients are called *correlograms*. To illustrate the use of a correlogram, we will consider a data set of over 200 glass fragments obtained during forensic work. For each glass fragment, we have measurements about its composition, expressed as the percent in weight of various mineral oxides. There are seven different oxides for which we have measurements, yielding a total of 6 + 5 + 4 + 3 + 2 + 1 = 21 pairwise correlations. We can display these 21 correlations at once as a matrix of colored tiles, where each tile represents one correlation coefficient (Figure \@ref(fig:forensic-correlations1)). This correlogram allows us to quickly grasp trends in the data, such as that magnesium is negatively correlated with nearly all other oxides, and that aluminum and barium have a strong positive correlation.

(ref:forensic-correlations1) Correlations in mineral content for 214 samples of glass fragments obtained during forensic work. The dataset contains seven variables measuring the amounts of magnesium (Mg), calcium (Ca), iron (Fe), potassium (K), sodium (Na), aluminum (Al), and barium (Ba) found in each glass fragment. The colored tiles represents the correlations between pairs of these variables. Data source: B. German

```{r forensic-correlations1, fig.width = 4., fig.asp = 1, fig.cap = '(ref:forensic-correlations1)'}
cm <- cor(select(forensic_glass, -type, -RI, -Si))
df_wide <- as.data.frame(cm)
df_long <- stack(df_wide)
names(df_long) <- c("cor", "var1")
df_long <- cbind(df_long, var2 = rep(rownames(cm), length(rownames(cm))))
clust <- hclust(as.dist(1-cm), method="average") 
levels <- clust$labels[clust$order]
df_long$var1 <- factor(df_long$var1, levels = levels)
df_long$var2 <- factor(df_long$var2, levels = levels)
ggplot(filter(df_long, as.integer(var1) < as.integer(var2)),
       aes(var1, var2, fill=cor)) + 
  geom_tile(color = "white", size = 1) + 
  scale_x_discrete(position = "top", name = NULL, expand = c(0, 0)) +
  scale_y_discrete(name = NULL, expand = c(0, 0)) +
  scale_fill_continuous_divergingx(
    palette = "Earth", rev = FALSE,
    limits = c(-.5, .5),
    breaks = c(-.5, 0, .5),
    labels = c("–0.5", "0.0", "0.5"),
    name = "correlation",
    guide = guide_colorbar(
      direction = "horizontal",
      label.position = "bottom",
      title.position = "top",
      barwidth = grid::unit(140, "pt"),
      barheight = grid::unit(17.5, "pt"),
      ticks.linewidth = 1
    )
  ) +
  coord_fixed() +
  theme_dviz_open(rel_small = 1) +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = grid::unit(3, "pt"),
    legend.position = c(.97, .0),
    legend.justification = c(1, 0),
    legend.title.align = 0.5
  )

```

One weakness of the correlogram of Figure \@ref(fig:forensic-correlations1) is that low correlations, i.e. correlations with absolute value near zero, are not as visually suppressed as they should be. For example, magnesium (Mg) and potassium (K) are not at all correlated but Figure \@ref(fig:forensic-correlations1) doesn't immediately show this. To overcome this limitation, we can display the correlations as colored circles and scale the circle size with the absolute value of the correlation coefficient (Figure \@ref(fig:forensic-correlations1)). In this way, low correlations are suppressed and high correlations stand out better.

(ref:forensic-correlations2) Correlations in mineral content for forensic glass samples. The color scale is identical to Figure \@ref(fig:forensic-correlations1). However, now the magnitude of each correlation is also encoded in the size of the colored circles. This choice visually deemphasizes cases with correlations near zero. Data source: B. German

```{r forensic-correlations2, fig.width = 4., fig.asp = 1, fig.cap = '(ref:forensic-correlations2)'}
ggplot(filter(df_long, as.integer(var1) < as.integer(var2)),
       aes(var1, var2, fill=cor, size = abs(cor))) + 
  geom_point(shape = 21, stroke = 0) + 
  scale_x_discrete(position = "top", name = NULL, expand = c(0, 0.5)) +
  scale_y_discrete(name = NULL, expand = c(0, 0.5)) +
  scale_size_area(max_size = 19, limits = c(0, 0.5), guide = "none") +
  scale_fill_continuous_divergingx(
    palette = "Earth", rev = FALSE,
    limits = c(-.5, .5),
    breaks = c(-.5, 0, .5),
    labels = c("–0.5", "0.0", "0.5"),
    name = "correlation",
    guide = guide_colorbar(
      direction = "horizontal",
      label.position = "bottom",
      title.position = "top",
      barwidth = grid::unit(140, "pt"),
      barheight = grid::unit(17.5, "pt"),
      ticks.linewidth = 1
    )
  ) +
  coord_fixed() +
  theme_dviz_open(rel_small = 1) +
  theme(
    axis.line = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = grid::unit(3, "pt"),
    legend.position = c(.97, .0),
    legend.justification = c(1, 0),
    legend.title.align = 0.5
  )

```

All correlograms have one important drawback: They are fairly abstract. While they show us important patterns in the data, they also hide the underlying data points and may cause us to draw incorrect conclusions. It is always better to visualize the raw data rather than abstract, derived quantities that have been calculated from it. Fortunately, we can frequently find a middle ground between showing important patterns and showing the raw data by applying techniques of dimension reduction.

## Dimension reduction

Dimension reduction relies on the key insight that most high-dimensional datasets consist of multiple correlated variables that convey overlapping information. Such datasets can be reduced to a smaller number of key dimensions without loss of much critical information. As a simple, intuitive example, consider a dataset of multiple physical traits of people, including quantities such as each person's height and weight, the lengths of the arms and legs, the circumferences of waist, hip, and chest, etc. We can understand immediately that all these quantities will relate first and foremost to the overall size of each person. All else being equal, a larger person will be taller, weigh more, have longer arms and legs, and larger waist, hip, and chest circumferences. The next important dimension is going to be the person's sex. Male and female measurements are substantially different for persons of comparable size. For example, a woman will tend to have higher hip circumference than a man, all else being equal.

There are many techniques for dimension reduction. I will discuss only one technique here, the most widely used one, called *principal components analysis* (PCA). PCA introduces a new set of variables (called principal components, PCs) by linear combination of the original variables in the data, standardized to zero mean and unit variance (see Figure \@ref(fig:blue-jays-PCA) for a toy example in two dimensions). The PCs are chosen such that they are uncorrelated, and they are ordered such that the first component captures the largest possible amount of variation in the data, and subsequent components capture increasingly less. Usually, key features in the data can be seen from only the first two or three PCs.

(ref:blue-jays-PCA) Example principal components (PC) analysis in two dimensions. (a) The original data. As example data, I am using the head-length and skull-size measurements from the blue jays dataset. Female and male birds are distinguished by color, but this distinction has no effect on the PC analysis. (b) As the first step in PCA, we scale the original data values to zero mean and unit variance. We then we define new variables (the principal components, PCs) along the directions of maximum variation in the data. (c) Finally, we project the data into the new coordinates. Mathematically, this projection is equivalent to a rotation of the data points around the origin. In the 2D example shown here, the data points are rotated clockwise by 45 degrees. 

```{r blue-jays-PCA, fig.width = 5.5*6/4.2, fig.asp = 0.35, fig.cap = '(ref:blue-jays-PCA)'}
p1 <- ggplot(blue_jays, aes(Skull, Head, fill = KnownSex)) + 
  geom_point(pch = 21, color = "white", size = 2, stroke = 0.2) +
  scale_x_continuous(
    name = "skull size (mm)", limits = c(26, 36),
    breaks = c(26, 28, 30, 32, 34, 36)
  ) +
  scale_y_continuous(
    name = "head length (mm)", limits = c(51, 61),
    breaks = c(52, 54, 56, 58, 60)
  ) +
  scale_fill_manual(
    values = c(F = "#D55E00", M = "#0072B2"),
    breaks = c("F", "M"),
    labels = c("female birds   ", "male birds"),
    name = NULL,
    guide = guide_legend(
      direction = "horizontal",
      override.aes = list(size = 3)
      )
    ) +
  coord_fixed() +
  theme_dviz_grid(12) +
  theme(
    legend.position = "top",
    legend.justification = "right",
    legend.box.spacing = unit(3.5, "pt"), # distance between legend and plot
    legend.text = element_text(vjust = 0.6),
    legend.spacing.x = unit(2, "pt"),
    legend.background = element_rect(fill = "white", color = NA),
    legend.key.width = unit(10, "pt")
  )

bj_standard <- data.frame(
  Skull = scale(blue_jays$Skull),
  Head = scale(blue_jays$Head),
  KnownSex = blue_jays$KnownSex
)

df_arrows <- data.frame(
  x = c(-3, 2),
  y = c(-3, -2),
  xend = c(3, -2),
  yend = c(3, 2)
)

df_labels <- data.frame(
  x = c(3.1, -2.1),
  y = c(3.1, 2.1),
  hjust = c(0, 1),
  vjust = c(0, 0),
  label = c("PC 1", "PC 2")
)

p2 <- ggplot(bj_standard, aes(Skull, Head, fill = KnownSex)) + 
  geom_point(pch = 21, color = "white", size = 2, stroke = 0.2) +
  geom_segment(
    data = df_arrows,
    aes(x = x, y = y, xend = xend, yend = yend),
    color = "black",
    arrow = arrow(angle = 15, length = grid::unit(9, "pt"), type = "closed"),
    inherit.aes = FALSE
  ) +
  geom_text(
    data = df_labels,
    aes(x = x, y = y, hjust = hjust, vjust = vjust, label = label),
    size = 12/.pt,
    family = dviz_font_family,
    color = "black", inherit.aes = FALSE
  ) +
  scale_x_continuous(
    name = "skull size (scaled)", limits = c(-4, 4),
    breaks = c(-4, -2, 0, 2, 4), labels = c("-4.0", "-2.0", "0.0", "2.0", "4.0")
  ) +
  scale_y_continuous(
    name = "head length (scaled)", limits = c(-4, 4),
    breaks = c(-4, -2, 0, 2, 4), labels = c("-4.0", "-2.0", "0.0", "2.0", "4.0")
  ) +
  scale_fill_manual(
    values = c(F = "#D55E00", M = "#0072B2"),
    breaks = c("F", "M"),
    labels = c("", ""),
    name = NULL,
    guide = guide_legend(
      direction = "horizontal",
      override.aes = list(shape = NA)
      )
    ) +
  coord_fixed(ratio = 1) +
  theme_dviz_grid(12) +
  theme(
    legend.position = "top",
    legend.justification = "right",
    legend.box.spacing = unit(3.5, "pt"), # distance between legend and plot
    legend.text = element_text(vjust = 0.6),
    legend.spacing.x = unit(2, "pt"),
    legend.background = element_rect(fill = "white", color = NA),
    legend.key.width = unit(10, "pt")
  )

df_arrows <- data.frame(
  x = c(-5.2, 0),
  y = c(0, -3.5),
  xend = c(5.2, 0),
  yend = c(0, 3.5)
)

df_labels <- data.frame(
  x = c(5.2, 0),
  y = c(0.4, 3.8),
  hjust = c(0.9, 0.5),
  vjust = c(0, 0),
  label = c("PC 1", "PC 2")
)


p3 <- ggplot(bj_standard, aes(Skull + Head, -Skull + Head, fill = KnownSex)) + 
  geom_point(pch = 21, color = "white", size = 2, stroke = 0.2) +
  geom_segment(
    data = df_arrows,
    aes(x = x, y = y, xend = xend, yend = yend),
    color = "black",
    arrow = arrow(angle = 15, length = grid::unit(9, "pt"), type = "closed"),
    inherit.aes = FALSE
  ) +
  geom_text(
    data = df_labels,
    aes(x = x, y = y, hjust = hjust, vjust = vjust, label = label),
    size = 12/.pt,
    family = dviz_font_family,
    color = "black", inherit.aes = FALSE
  ) +
  scale_x_continuous(
    name = "PC 1", limits = c(-5.2, 5.2),
    breaks = c(-5, -2.5, 0, 2.5, 5), labels = c("-5.0", "-2.5", "0.0", "2.5", "5.0")
) +
  scale_y_continuous(
    name = "PC 2", limits = c(-5.2, 5.2),
    breaks = c(-5, -2.5, 0, 2.5, 5), labels = c("-5.0", "-2.5", "0.0", "2.5", "5.0")
  ) +
  scale_fill_manual(
    values = c(F = "#D55E00", M = "#0072B2"),
    breaks = c("F", "M"),
    labels = c("", ""),
    name = NULL,
    guide = guide_legend(
      direction = "horizontal",
      override.aes = list(shape = NA)
      )
    ) +
  coord_fixed(ratio = 1) +
  theme_dviz_grid(12) +
  theme(
    legend.position = "top",
    legend.justification = "right",
    legend.box.spacing = unit(3.5, "pt"), # distance between legend and plot
    legend.text = element_text(vjust = 0.6),
    legend.spacing.x = unit(2, "pt"),
    legend.background = element_rect(fill = "white", color = NA),
    legend.key.width = unit(10, "pt")
  )

plot_grid(
  p1, NULL, p2, NULL, p3,
  nrow = 1, align = 'hv', rel_widths = c(1, .04, 1, .04, 1),
  labels = c("a", "", "b", "", "c"), label_y = 0.985
)
```

When we perform PCA, we are generally interested in two pieces of information: (i) the composition of the PCs and (ii) the location of the individual data points in the principal components space. Let's look at these two pieces in a PC analysis of the forensic glass dataset.

First, we look at the component composition (Figure \@ref(fig:forensic-PCA-rotation)). Here, we only consider the first two components, PC 1 and PC 2. Because the PCs are linear combinations of the original variables (after standardization), we can represent the original variables as arrows indicating to what extent they contribute to the PCs. Here, we see that barium and sodium contribute primarily to PC 1 and not to PC 2, calcium and potassium contribute primarily to PC 2 and not to PC 1, and the other variables contribute in varying amounts to both components (Figure \@ref(fig:forensic-PCA-rotation)). The arrows are of varying lengths because there are more than two PCs. For example the arrow for iron is particularly short because it contributes primarily to higher-order PCs (not shown).

(ref:forensic-PCA-rotation) Composition of the first two components in a principal components analysis (PCA) of the forensic glass dataset. Component one (PC 1) measures primarily the amount of aluminum, barium, sodium, and magnesium contents in a glass fragment, whereas component two (PC 2) measures primarily the amount of calcium and potassium content, and to some extent the amount of aluminum and magnesium.

```{r forensic-PCA-rotation, fig.width = 4.5, fig.asp = 1, fig.cap = '(ref:forensic-PCA-rotation)'}
select(forensic_glass, -type, -RI, -Si) %>%
  scale() %>%
  prcomp() -> pca

pca_data <- data.frame(pca$x, type = forensic_glass$type) %>%
  mutate(type = case_when(
    type == "WinF" ~ "window",
    type == "WinNF" ~ "window",
    type == "Veh" ~ "window",
    type == "Con" ~ "container",
    type == "Tabl" ~ "tableware",
    type == "Head" ~ "headlamp"
  )) %>%
  mutate(type = factor(type, levels = c("headlamp", "tableware", "container", "window")))

colors = darken(c("#D55E00", "#0072B2", "#009E73", "#E69F00"), .3)
fills = c("#D55E0040", "#0072B240", "#009E7340", "#E69F0040")

rotation_data <- data.frame(pca$rotation, type = row.names(pca$rotation))

rotation_labels <- data.frame(type = c("Ba", "Al", "Na", "K", "Fe", "Ca", "Mg"),
                              hjust = c(1, 1, 1, 0.5, 0, 0.5, 0),
                              vjust = c(0.5, 0.5, 1, 1, 0.5, 0, 0.5),
                              nudge_x = c(-0.01, -0.01, 0.01, 0, .01, 0, .01),
                              nudge_y = c(0, 0, -.01, -.01, 0, .01, 0))

rotation_data <- left_join(rotation_data, rotation_labels, by = "type")

arrow_style <- arrow(
  angle = 15, length = grid::unit(9, "pt"),
  type = "closed"
)
ggplot(rotation_data) + 
  geom_segment(aes(xend = PC1, yend = PC2),
               x = 0, y = 0,
               arrow = arrow_style,
               color = "#0072B2",
               fill = "#0072B2") + 
  geom_text(aes(x = PC1 + nudge_x, y = PC2 + nudge_y, label = type,
                hjust = hjust, vjust = vjust),
            size = 12/.pt, color = darken('#D55E00', .3),
            family = dviz_font_family) + 
  scale_x_continuous(limits = c(-.8, .8), name = "PC 1") + 
  scale_y_continuous(limits = c(-.7, .8), breaks = c(-0.5, 0, .5),
                     name = "PC 2") +
  coord_fixed() +
  theme_dviz_grid()
```

Next, we project the original data into the principal components space (Figure \@ref(fig:forensic-PCA)). We see a clear clustering of distinct types of glass fragments in this plot. Fragments from both headlamps and windows fall into clearly delineated regions in the PC plot, with few outliers. Fragments from tableware and from containers are a little more spread out, but nevertheless clearly distinct from both headlamp and window fragments. By comparing Figure \@ref(fig:forensic-PCA) with Figure \@ref(fig:forensic-PCA-rotation), we can conclude that window samples tend to have higher than average magnesium content and lower than average barium, aluminum, and sodium content, whereas the opposite is true for headlamp samples.

(ref:forensic-PCA) Composition of individual glass fragments visualized in the principal components space defined in Figure \@ref(fig:forensic-PCA-rotation). We see that the different types of glass samples cluster at characteristic values of PC 1 and 2. In particular, headlamps are characterized by a negative PC 1 value whereas windows tend to have a positive PC 1 value. Tableware and containers have PC 1 values close to zero and tend to have positive PC 2 values. However, there are a few exceptions where container fragments have both a negative PC 1 value and a negative PC 2 value. These are fragments whose composition drastically differs from all other fragments analyzed.


```{r forensic-PCA, fig.width = 5*6/4.2, fig.cap = '(ref:forensic-PCA)'}
ggplot(pca_data, aes(x = PC1, y = PC2, color = type, fill = type, shape = type)) + 
  geom_point(size = 2) +
  scale_color_manual(values = colors, name = "fragment type") +
  scale_fill_manual(values = fills, name = "fragment type") +
  scale_shape_manual(values = c(22:24, 21), name = "fragment type") +
  scale_x_continuous(name = "PC 1") +
  scale_y_continuous(name = "PC 2") +
  theme_dviz_grid()
```

## Paired data {#associations-paired-data}

A special case of multivariate quantitative data is paired data: Data where there are two or more measurements of the same quantity under slightly different conditions. Examples include two comparable measurements on each subject (e.g., the length of the right and the left arm of a person), repeat measurements on the same subject at different time points (e.g., a person's weight at two different times during the year), or measurements on two closely related subjects (e.g., the heights of two identical twins). For paired data, it is reasonable to assume that the two measurements belonging to a pair are more similar to each other than to the measurements belonging to other pairs. Two twins will be approximately of the same height but will differ in height from other twins. Therefore, for paired data, we need to choose visualizations that highlight any differences between the paired measurements.

An excellent choice in this case is a simple scatter plot on top of a diagonal line marking *x* = *y*. In such a plot, if the only difference between the two measurements of each pair is random noise, then all points in the sample will be scattered symmetrically around this line. Any systematic differences between the paired measurements, by contrast, will be visible in a systematic shift of the data points up or down relative to the diagonal. As an example, consider the carbon dioxide (CO<sub>2</sub>) emissions per person, measured for 166 countries both in 1970 and in 2010 (Figure \@ref(fig:CO2-paired-scatter)). This example highlights two common features of paired data. First, most points are relatively close to the diagonal line. Even though CO<sub>2</sub> emissions vary over nearly four orders of magnitude among countries, they are fairly consistent within each country over a 40-year time span. Second, the points are systematically shifted upwards relative to the diagonal line. The majority of countries has seen an increase in CO<sub>2</sub> emissions over the 40 years considered.

(ref:CO2-paired-scatter) Carbon dioxide (CO<sub>2</sub>) emissions per person in 1970 and 2010, for 166 countries. Each dot represents one country. The diagonal line represents identical CO<sub>2</sub> emissions in 1970 and 2010. The points are systematically shifted upwards relative to the diagonal line: In the majority of countries, emissions were higher in 2010 than in 1970. Data source: Carbon Dioxide Information Analysis Center

```{r CO2-paired-scatter, fig.width = 5, fig.asp = 0.9, fig.cap = '(ref:CO2-paired-scatter)'}
CO2_emissions %>% filter(year %in% c(1970, 2010)) %>%
  spread(year, emissions) %>% na.omit() -> emissions_1970_2010

df_segment <- data.frame(
  x = c(0.01, 0.008), 
  xend = c(100, 0.008),
  y = c(0.008, 0.01),
  yend = c(0.008, 100)
)


ggplot(emissions_1970_2010, aes(`1970`, `2010`)) + 
  geom_abline(slope = 1, color = "grey") +
  geom_point(pch = 21, fill = "#0072B2D0", color = "white", size = 2, stroke = 0.3) + 
  geom_segment(
    data = df_segment,
    aes(x = x, xend = xend, y = y, yend = yend),
    size = 0.5,
    inherit.aes = FALSE
  ) +
  scale_x_log10(
    limits = c(0.008, 125),
    breaks = c(0.01, 0.1, 1, 10, 100),
    labels = c("0.01", "0.1", "1.0", "10", "100"),
    #name = "1970 CO2 emissions (tons / person)"
    name = parse(text = "`1970 CO`[2]*` emissions (tons / person)`")
  ) + 
  scale_y_log10(
    limits = c(0.008, 125),
    breaks = c(0.01, 0.1, 1, 10, 100),
    labels = c("0.01", "0.1", "1.0", "10", "100"),
    #name = "2010 CO2 emissions (tons / person)"
    name = parse(text = "`2010 CO`[2]*` emissions (tons / person)`")
  ) +
  coord_fixed(expand = FALSE, clip = "off") +
  theme_dviz_open() +
  theme(axis.line = element_blank())
```

Scatter plots such as Figure \@ref(fig:CO2-paired-scatter) work well when we have a large number of data points and/or are interested in a systematic deviation of the entire data set from the null expectation. By contrast, if we have only a small number of observations and are primarily interested in the identity of each individual case, a *slopegraph* may be a better choice. In a slopegraph, we draw individual measurements as dots arranged into two columns and indicate pairings by connecting the paired dots with a line. The slope of each line highlights the magnitude and direction of change. Figure \@ref(fig:CO2-slopegraph) uses this approach to show the ten countries with the largest difference in CO<sub>2</sub> emissions per person from 2000 to 2010.

(ref:CO2-slopegraph) Carbon dioxide (CO<sub>2</sub>) emissions per person in 2000 and 2010, for the ten countries with the largest difference between these two years. Data source: Carbon Dioxide Information Analysis Center

```{r CO2-slopegraph, fig.width = 4.5, fig.asp = 1, fig.cap = '(ref:CO2-slopegraph)'}
CO2_emissions %>% filter(year %in% c(2000, 2005, 2010)) %>%
  spread(year, emissions) %>% 
  na.omit() %>%
  mutate(diff = `2010` - `2000`, absdiff = abs(diff)) %>%
  arrange(desc(absdiff)) -> emissions_diff

emissions_diff[1:10,] %>% select(-diff) %>% 
  gather(year, emissions, `2000`:`2010`) %>%
  mutate(year = as.numeric(year)) -> emissions_top_ten

textcol <- "gray30"

labels_df <- filter(emissions_top_ten, year == 2010) %>%
  left_join(
    tibble(
      country = c("Trinidad and Tobago", "Qatar", "United Arab Emirates", "Oman", "Bahrain", "Singapore", "Netherlands Antilles", "Kazakhstan", "Equatorial Guinea", "Kuwait"),
      nudge_y = c(.1, .1, .1, .1, -.3, .1, .7, .1, .1, .1)
    )
  )

ggplot(filter(emissions_top_ten, year != 2005), aes(x = year, y = emissions)) +
  geom_line(aes(group = country), color = "gray60") +
  geom_point(color = "white", size = 4) +
  geom_point(color = "#0072B2", size = 2) +
  geom_text(
    data = labels_df,
    aes(x = year + 0.45, y = emissions + nudge_y, label = country),
    family = "Myriad Pro",
    size = 10/.pt,
    hjust = 0) +
  scale_x_continuous(
    limits = c(2000, 2020), breaks = c(2000, 2010),
    labels = c("2000", "2010"),
    expand = expand_scale(add = c(1, 0)),
    name = NULL,
    position = "top"
  ) +
  scale_y_continuous(
    limits = c(0, 63.2),
    expand = c(0, 0),
    #name = "CO2 emissions (tons / person)"
    name = parse(text = "`CO`[2]*` emissions (tons / person)`")
  ) +
  theme_dviz_open() +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank(),
    axis.title.y = element_text(color = textcol),
    axis.text.y = element_text(color = textcol),
    axis.line.y.left = element_line(color = textcol),
    axis.text.y.right = element_text(
      hjust = 0, vjust = .5,
      margin = margin(0, 0, 0, 0),
      color = "black",
      lineheight = 0.8
    ),
    axis.line.y.right = element_blank(),
    axis.ticks.y.right = element_blank(),
    plot.margin = margin(3.5, 1.5, 7, 1.5)
  )

```

Slopegraphs have one important advantage over scatter plots: They can be used to compare more than two measurements at a time. For example, we can modify Figure \@ref(fig:CO2-slopegraph) to show CO<sub>2</sub> emissions at three time points, here the years 2000, 2005, and 2010 (Figure \@ref(fig:CO2-slopegraph-three-year)). This choice highlights both countries with a large change in emissions over the entire decade as well as countries such as Qatar or Trinidad and Tobago for which there is a large difference in the trend seen for the first five-year interval and the second one. 


(ref:CO2-slopegraph-three-year) CO<sub>2</sub> emissions per person in 2000, 2005, and 2010, for the ten countries with the largest difference between the years 2000 and 2010. Data source: Carbon Dioxide Information Analysis Center

```{r CO2-slopegraph-three-year, fig.width = 4.5, fig.asp = 1, fig.cap = '(ref:CO2-slopegraph-three-year)'}
ggplot(emissions_top_ten, aes(x = year, y = emissions)) +
  geom_line(aes(group = country), color = "gray60") +
  geom_point(color = "white", size = 4) +
  geom_point(color = "#0072B2", size = 2) +
  geom_text(
    data = labels_df,
    aes(x = year + 0.45, y = emissions + nudge_y, label = country),
    family = "Myriad Pro",
    size = 10/.pt,
    hjust = 0) +
  scale_x_continuous(
    limits = c(2000, 2020), breaks = c(2000, 2005, 2010),
    labels = c("2000", "2005", "2010"),
    expand = expand_scale(add = c(1, 0)),
    name = NULL,
    position = "top"
  ) +
  scale_y_continuous(
    limits = c(0, 63.2),
    expand = c(0, 0),
    #name = "CO2 emissions (tons / person)"
    name = parse(text = "`CO`[2]*` emissions (tons / person)`")
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_open() +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank(),
    axis.title.y = element_text(color = textcol),
    axis.text.y = element_text(color = textcol),
    axis.line.y.left = element_line(color = textcol),
    axis.text.y.right = element_text(
      hjust = 0, vjust = .5,
      margin = margin(0, 0, 0, 0),
      color = "black",
      lineheight = 0.8
    ),
    axis.line.y.right = element_blank(),
    axis.ticks.y.right = element_blank(),
    plot.margin = margin(3.5, 1.5, 7, 1.5)
  )

```



```{r eval = FALSE}
## Notes:
# The slopegraph was introduced by Tufte, first in VDQI, but not named there. It's unclear when and where the name was first mentioned.
# See also:
#   https://github.com/leeper/slopegraph,
#   https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0003nk)
```

<!--chapter:end:visualizing_associations.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(tidyr)
library(ggridges) # for geom_density_line()
```


# Visualizing distributions: Histograms and density plots {#histograms-density-plots}

We frequently encounter the situation where we would like to understand how a particular variable is distributed in a dataset. To give a concrete example, we will consider the passengers of the Titanic, a data set we encountered already in Chapter \@ref(visualizing-amounts). There were approximately 1300 passengers on the Titanic (not counting crew), and we have reported ages for 756 of them. We might want to know how many passengers of what ages there were on the Titanic, i.e., how many children, young adults, middle-aged people, seniors, and so on. We call the relative proportions of different ages among the passengers the *age distribution* of the passengers.

## Visualizing a single distribution

We can obtain a sense of the age distribution among the passengers by grouping all passengers into bins with comparable ages and then counting the number of passengers in each bin. This procedure results in a table such as Table \@ref(tab:titanic-ages). 
```{r titanic-ages}
titanic <- titanic_all
age_counts <- hist(titanic$age, breaks = (0:15) * 5 + .01, plot = FALSE)$counts
age_hist <- data.frame(
  `age range` = c("0--5", "6--10", "11--15", "16--20", "21--25", "26--30", "31--35", "36--40", "41--45", "46--50", "51--55", "56--60", "61--65", "66--70", "71--75"),
  count = age_counts,
  check.names = FALSE
)

age_hist_display <- rename(
  age_hist,
  `Age range` = `age range`,
  Count = count
)

knitr::kable(
  list(
    age_hist_display[1:6,], age_hist_display[7:12,], age_hist_display[13:15,]
  ),
  caption = 'Numbers of passenger with known age on the Titanic.', booktabs = TRUE,
  row.names = FALSE
)

```

We can visualize this table by drawing filled rectangles whose heights correspond to the counts and whose widths correspond to the width of the age bins (Figure \@ref(fig:titanic-ages-hist1)). Such a visualization is called a histogram. (Note that all bins must have the same width for the visualization to be a valid histogram.)

(ref:titanic-ages-hist1) Histogram of the ages of Titanic passengers.

```{r titanic-ages-hist1, fig.cap='(ref:titanic-ages-hist1)'}
age_hist <- cbind(age_hist, age = (1:15) * 5 - 2.5)
h1 <- ggplot(age_hist, aes(x = age, y = count)) + geom_col(width = 4.7, fill = "#56B4E9")  + 
  scale_y_continuous(expand = c(0, 0), breaks = 25 * (0:5)) +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 7, 3, 1.5)
  )
h1
```

Because histograms are generated by binning the data, their exact visual appearance depends on the choice of the bin width. Most visualization programs that generate histograms will choose a bin width by default, but chances are that bin width is not the most appropriate one for any histogram you may want to make. It is therefore critical to always try different bin widths to verify that the resulting histogram reflects the underlying data accurately. In general, if the bin width is too small, then the histogram becomes overly peaky and visually busy and the main trends in the data may be obscured. On the other hand, if the bin width is too large, then smaller features in the distribution of the data, such as the dip around age 10, may disappear.

For the age distribution of Titanic passengers, we can see that a bin width of one year is too small and a bin width of fifteen years is too large, whereas bin widths between three to five years work fine (Figure \@ref(fig:titanic-ages-hist-grid)).

(ref:titanic-ages-hist-grid) Histograms depend on the chosen bin width. Here, the same age distribution of Titanic passengers is shown with four different bin widths: (a) one year; (b) three years; (c) five years; (d) fifteen years.

```{r titanic-ages-hist-grid, fig.width=5.5*6/4.2, fig.cap='(ref:titanic-ages-hist-grid)'}
age_hist_1 <- data.frame(
  age = (1:75) - 0.5, 
  count = hist(titanic$age, breaks = (0:75) + .01, plot = FALSE)$counts
)

age_hist_3 <- data.frame(
  age = (1:25) * 3 - 1.5, 
  count = hist(titanic$age, breaks = (0:25) * 3 + .01, plot = FALSE)$counts
)

age_hist_15 <- data.frame(
  age = (1:5) * 15 - 7.5, 
  count = hist(titanic$age, breaks = (0:5) * 15 + .01, plot = FALSE)$counts
)

h2 <- ggplot(age_hist_1, aes(x = age, y = count)) + 
  geom_col(width = .85, fill = "#56B4E9")  + 
  scale_y_continuous(expand = c(0, 0), breaks = 10 * (0:5)) +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 1.5, 3, 1.5)
  )

h3 <- ggplot(age_hist_3, aes(x = age, y = count)) + geom_col(width = 2.75, fill = "#56B4E9")  + 
  scale_y_continuous(expand = c(0, 0), breaks = 25 * (0:5)) +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 1.5, 3, 1.5)
  )

h4 <- ggplot(age_hist_15, aes(x = age, y = count)) + geom_col(width = 14.5, fill = "#56B4E9")  + 
  scale_y_continuous(expand = c(0, 0), breaks = 100 * (0:4)) +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 1.5, 3, 1.5)
  )


plot_grid(
  h2, NULL, h3,
  NULL, NULL, NULL,
  h1 + theme_dviz_hgrid(12) + theme(axis.line.x = element_blank(), plot.margin = margin(3, 1.5, 3, 1.5)),
  NULL, h4,
  align = 'hv',
  labels = c("a", "", "b", "", "", "", "c", "", "d"),
  rel_widths = c(1, .04, 1),
  rel_heights = c(1, .04, 1)
)
```

```{block type='rmdtip', echo=TRUE}
When making a histogram, always explore multiple bin widths.
```

Histograms have been a popular visualization option since at least the 18th century, in part because they are easily generated by hand. More recently, as extensive computing power has become available in everyday devices such as laptops and cell phones, we see them increasingly being replaced by density plots. In a density plot, we attempt to visualize the underlying probability distribution of the data by drawing an appropriate continuous curve (Figure \@ref(fig:titanic-ages-dens1)). This curve needs to be estimated from the data, and the most commonly used method for this estimation procedure is called *kernel density estimation.* In kernel density estimation, we draw a continuous curve (the kernel) with a small width (controlled by a parameter called *bandwidth*) at the location of each data point, and then we add up all these curves to obtain the final density estimate. The most widely used kernel is a Gaussian kernel (i.e., a Gaussian bell curve), but there are many other choices.

(ref:titanic-ages-dens1) Kernel density estimate of the age distribution of passengers on the Titanic. The height of the curve is scaled such that the area under the curve equals one. The density estimate was performed with a Gaussian kernel and a bandwidth of 2.

```{r titanic-ages-dens1, fig.cap='(ref:titanic-ages-dens1)'}
ggplot(titanic, aes(x = age)) + 
  geom_density_line(fill = "#56B4E9", color = darken("#56B4E9", 0.5), bw = 2, kernel = "gaussian") + 
  scale_y_continuous(limits = c(0, 0.046), expand = c(0, 0), name = "density") +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 7, 3, 1.5)
  )
```

Just as is the case with histograms, the exact visual appearance of a density plot depends on the kernel and bandwidth choices (Figure \@ref(fig:titanic-ages-dens-grid)). The bandwidth parameter behaves similarly to the bin width in histograms. If the bandwidth is too small, then the density estimate can become overly peaky and visually busy and the main trends in the data may be obscured. On the other hand, if the bandwidth is too large, then smaller features in the distribution of the data may disappear. In addition, the choice of the kernel affects the shape of the density curve. For example, a Gaussian kernel will have a tendency to produce density estimates that look Gaussian-like, with smooth features and tails. By contrast, a rectangular kernel can generate the appearance of steps in the density curve (Figure \@ref(fig:titanic-ages-dens-grid)d). In general, the more data points there are in the data set, the less the choice of the kernel matters. Therefore, density plots tend to be quite reliable and informative for large data sets but can be misleading for data sets of only a few points.

(ref:titanic-ages-dens-grid) Kernel density estimates depend on the chosen kernel and bandwidth. Here, the same age distribution of Titanic passengers is shown for four different combinations of these parameters: (a) Gaussian kernel, bandwidth = 0.5; (b) Gaussian kernel, bandwidth = 2; (c) Gaussian kernel, bandwidth = 5; (d) Rectangular kernel, bandwidth = 2.

```{r titanic-ages-dens-grid, fig.width=5.5*6/4.2, fig.cap='(ref:titanic-ages-dens-grid)'}
pdens1 <- ggplot(titanic, aes(x = age)) + 
  geom_density_line(fill = "#56B4E9", color = darken("#56B4E9", 0.5), bw = .5, kernel = "gaussian") + 
  scale_y_continuous(limits = c(0, 0.046), expand = c(0, 0), name = "density") +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 1.5, 3, 1.5)
  )

pdens2 <- ggplot(titanic, aes(x = age)) + 
  geom_density_line(fill = "#56B4E9", color = darken("#56B4E9", 0.5), bw = 2, kernel = "gaussian") + 
  scale_y_continuous(limits = c(0, 0.046), expand = c(0, 0), name = "density") +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 1.5, 3, 1.5)
  )

pdens3 <- ggplot(titanic, aes(x = age)) + 
  geom_density_line(fill = "#56B4E9", color = darken("#56B4E9", 0.5), bw = 5, kernel = "gaussian") + 
  scale_y_continuous(limits = c(0, 0.046), expand = c(0, 0), name = "density") +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 1.5, 3, 1.5)
  )

pdens4 <- ggplot(titanic, aes(x = age)) + 
  geom_density_line(fill = "#56B4E9", color = darken("#56B4E9", 0.5), bw = 2, kernel = "rectangular") + 
  scale_y_continuous(limits = c(0, 0.046), expand = c(0, 0), name = "density") +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12) +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 1.5, 3, 1.5)
  )

plot_grid(
  pdens1, NULL, pdens2, 
  NULL, NULL, NULL,
  pdens3, NULL, pdens4,
  align = 'hv',
  labels = c("a", "", "b", "", "", "", "c", "", "d"),
  rel_widths = c(1, .04, 1),
  rel_heights = c(1, .04, 1)
)
```

Density curves are usually scaled such that the area under the curve equals one. This convention can make the *y* axis scale confusing, because it depends on the units of the *x* axis. For example, in the case of the age distribution, the data range on the *x* axis goes from 0 to approximately 75. Therefore, we expect the mean height of the density curve to be 1/75 = 0.013. Indeed, when looking at the age density curves (e.g., Figure \@ref(fig:titanic-ages-dens-grid)), we see that the *y* values range from 0 to approximately 0.04, with an average of somewhere close to 0.01.

Kernel density estimates have one pitfall that we need to be aware of: They have a tendency to produce the appearance of data where none exists, in particular in the tails. As a consequence, careless use of density estimates can easily lead to figures that make nonsensical statements. For example, if we don't pay attention, we might generate a visualization of an age distribution that includes negative ages (Figure \@ref(fig:titanic-ages-dens-negative)).


(ref:titanic-ages-dens-negative) Kernel density estimates can extend the tails of the distribution into areas where no data exist and no data are even possible. Here, the density estimate has been allowed to extend into the negative age range. This is clearly nonsensical and should be avoided.

```{r titanic-ages-dens-negative, fig.cap='(ref:titanic-ages-dens-negative)'}
pdens_neg <- ggplot(titanic, aes(x = age)) + 
  geom_density_line(fill = "#56B4E9", color = darken("#56B4E9", 0.5)) + 
  scale_y_continuous(limits = c(0, 0.046), expand = c(0, 0), name = "density") +
  scale_x_continuous(name = "age (years)", limits = c(-10, 79), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    plot.margin = margin(3, 7, 3, 1.5)
  )

stamp_wrong(pdens_neg)
```

```{block type='rmdtip', echo=TRUE}
Always verify that your density estimate does not predict the existence of nonsensical data values.
```

So should you use a histogram or a density plot to visualize a distribution? Heated discussions can be had on this topic. Some people are vehemently against density plots and believe that they are arbitrary and misleading. Others realize that histograms can be just as arbitrary and misleading. I think the choice is largely a matter of taste, but sometimes one or the other option may more accurately reflect the specific features of interest in the data at hand. There is also the possibility of using neither and instead choosing empirical cumulative density functions or q-q plots (Chapter \@ref(ecdf-qq)). Finally, I believe that density estimates have an inherent advantage over histograms as soon as we want to visualize more than one distribution at a time (see next section).

## Visualizing multiple distributions at the same time {#multiple-histograms-densities}

In many scenarios we have multiple distributions we would like to visualize simultaneously. For example, let's say we'd like to see how the ages of Titanic passengers are distributed between men and women. Were men and women passengers generally of the same age, or was there an age difference between the genders? One commonly employed visualization strategy in this case is a stacked histogram, where we draw the histogram bars for women on top of the bars for men, in a different color (Figure \@ref(fig:titanic-age-stacked-hist)).

(ref:titanic-age-stacked-hist) Histogram of the ages of Titanic passengers stratified by gender. This figure has been labeled as "bad" because stacked histograms are easily confused with overlapping histograms (see also Figure \@ref(fig:titanic-age-overlapping-hist)). In addition, the heights of the bars representing female passengers cannot easily be compared to each other.

```{r titanic-age-stacked-hist, fig.cap='(ref:titanic-age-stacked-hist)'}
data.frame(
  age = (1:25)*3 - 1.5, 
  male = hist(filter(titanic, sex == "male")$age, breaks = (0:25)*3 + .01, plot = FALSE)$counts,
  female = hist(filter(titanic, sex == "female")$age, breaks = (0:25)*3 + .01, plot = FALSE)$counts
) %>%
  gather(gender, count, -age) -> gender_counts

gender_counts$gender <- factor(gender_counts$gender, levels = c("female", "male"))

p_hist_stacked <- ggplot(gender_counts, aes(x = age, y = count, fill = gender)) + 
  geom_col(position = "stack") +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  scale_y_continuous(limits = c(0, 89), expand = c(0, 0), name = "count") +
  scale_fill_manual(values = c("#D55E00", "#0072B2")) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    legend.position = c(.9, .87),
    legend.justification = c("right", "top"),
    legend.box.background = element_rect(fill = "white", color = "white"),
    plot.margin = margin(3, 7, 3, 1.5)
  )

stamp_bad(p_hist_stacked)
```

In my opinion, this type of visualization should be avoided. There are two key problems here: First, from just looking at the figure, it is never entirely clear where exactly the bars begin. Do they start where the color changes or are they meant to start at zero? In other words, are there about 25 females of age 18--20 or are there almost 80? (The former is the case.) Second, the bar heights for the female counts cannot be directly compared to each other, because the bars all start at a different height. For example, the men were on average older than the women, and this fact is not at all visible in Figure \@ref(fig:titanic-age-stacked-hist).

We could try to address these problems by having all bars start at zero and making the bars partially transparent (Figure \@ref(fig:titanic-age-overlapping-hist)).

(ref:titanic-age-overlapping-hist) Age distributions of male and female Titanic passengers, shown as two overlapping histograms. This figure has been labeled as "bad" because there is no clear visual indication that all blue bars start at a count of 0.

```{r titanic-age-overlapping-hist, fig.cap='(ref:titanic-age-overlapping-hist)'}
p_hist_overlapped <- ggplot(gender_counts, aes(x = age, y = count, fill = gender)) + 
  geom_col(position = "identity", alpha = 0.7) +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  scale_y_continuous(limits = c(0, 56), expand = c(0, 0), name = "count") +
  scale_fill_manual(
    values = c("#D55E00", "#0072B2"),
    guide = guide_legend(reverse = TRUE)
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    legend.position = c(.9, .87),
    legend.justification = c("right", "top"),
    legend.box.background = element_rect(fill = "white", color = "white"),
    plot.margin = margin(3, 7, 3, 1.5)
  )

stamp_bad(p_hist_overlapped)
```

However, this approach generates new problems. Now it appears that there are actually three different groups, not just two, and we're still not entirely sure where each bar starts and ends. Overlapping histograms don't work well because a semi-transparent bar drawn on top of another tends to not look like a semi-transparent bar but instead like a bar drawn in a different color.

Overlapping density plots don't typically have the problem that overlapping histograms have, because the continuous density lines help the eye keep the distributions separate. However, for this particular dataset, the age distributions for male and female passengers are nearly identical up to around age 17 and then diverge, so that the resulting visualization is still not ideal (Figure \@ref(fig:titanic-age-overlapping-dens)). 

(ref:titanic-age-overlapping-dens) Density estimates of the ages of male and female Titanic passengers. To highlight that there were more male than female passengers, the density curves were scaled such that the area under each curve corresponds to the total number of male and female passengers with known age (468 and 288, respectively).

```{r titanic-age-overlapping-dens, fig.cap='(ref:titanic-age-overlapping-dens)'}
titanic2 <- titanic
titanic2$sex <- factor(titanic2$sex, levels = c("male", "female"))

ggplot(titanic2, aes(x = age, y = ..count.., fill = sex, color = sex)) + 
  geom_density_line(bw = 2, alpha = 0.7) +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  scale_y_continuous(limits = c(0, 19), expand = c(0, 0), name = "scaled density") +
  scale_fill_manual(values = c("#0072B2", "#D55E00"), name = "gender") +
  scale_color_manual(values = darken(c("#0072B2", "#D55E00"), 0.5), name = "gender") +
  guides(fill = guide_legend(override.aes = list(linetype = 0))) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    legend.position = c(.9, .87),
    legend.justification = c("right", "top"),
    legend.box.background = element_rect(fill = "white", color = "white"),
    plot.margin = margin(3, 7, 3, 1.5)
  )
```

A solution that works well for this dataset is to show the age distributions of male and female passengers separately, each as a proportion of the overall age distribution (Figure \@ref(fig:titanic-age-fractional-dens)). This visualization shows intuitively and clearly that there were many fewer women than men in the 20--50-year age range on the Titanic.

(ref:titanic-age-fractional-dens) Age distributions of male and female Titanic passengers, shown as proportion of the passenger total. The colored areas show the density estimates of the ages of male and female passengers, respectively, and the gray areas show the overall passenger age distribution.

```{r titanic-age-fractional-dens, fig.width = 5.5*6/4.2, fig.asp = .45, fig.cap='(ref:titanic-age-fractional-dens)'}
ggplot(titanic2, aes(x = age, y = ..count..)) +
  geom_density_line(
    data = select(titanic, -sex), aes(fill = "all passengers"),
    color = "transparent"
  ) + 
  geom_density_line(aes(fill = sex), bw = 2, color = "transparent") +
  scale_x_continuous(limits = c(0, 75), name = "passenger age (years)", expand = c(0, 0)) +
  scale_y_continuous(limits = c(0, 26), name = "scaled density", expand = c(0, 0)) +
  scale_fill_manual(
    values = c("#b3b3b3a0", "#D55E00", "#0072B2"), 
    breaks = c("all passengers", "male", "female"),
    labels = c("all passengers  ", "males  ", "females"),
    name = NULL,
    guide = guide_legend(direction = "horizontal")
  ) +
  coord_cartesian(clip = "off") +
  facet_wrap(~sex, labeller = labeller(sex = function(sex) paste(sex, "passengers"))) +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    strip.text = element_text(size = 14, margin = margin(0, 0, 0.2, 0, "cm")),
    legend.position = "bottom",
    legend.justification = "right",
    legend.margin = margin(4.5, 0, 1.5, 0, "pt"),
    legend.spacing.x = grid::unit(4.5, "pt"),
    legend.spacing.y = grid::unit(0, "pt"),
    legend.box.spacing = grid::unit(0, "cm")
  )
```


Finally, when we want to visualize exactly two distributions, we can also make two separate histograms, rotate them by 90 degrees, and have the bars in one histogram point into the opposite direction of the other. This trick is commonly employed when visualizing age distributions, and the resulting plot is usually called an *age pyramid* (Figure \@ref(fig:titanic-age-pyramid)).

(ref:titanic-age-pyramid) The age distributions of male and female Titanic passengers visualized as an age pyramid.

```{r titanic-age-pyramid, fig.cap='(ref:titanic-age-pyramid)'}
ggplot(gender_counts, aes(x = age, y = ifelse(gender == "male",-1, 1)*count, fill = gender)) + 
  geom_col() +
  scale_x_continuous(name = "age (years)", limits = c(0, 75), expand = c(0, 0)) +
  scale_y_continuous(name = "count", breaks = 20*(-2:1), labels = c("40", "20", "0", "20")) +
  scale_fill_manual(values = c("#D55E00", "#0072B2"), guide = "none") +
  draw_text(x = 70, y = -39, "male", hjust = 0) +
  draw_text(x = 70, y = 21, "female", hjust = 0) +
  coord_flip() +
  theme_dviz_grid() +
  theme(axis.title.x = element_text(hjust = 0.61))
```

Importantly, this trick does not work when there are more than two distributions we want to visualize at the same time. For multiple distributions, histograms tend to become highly confusing, whereas density plots work well as long as the distributions are somewhat distinct and contiguous. For example, to visualize the distribution of butterfat percentage among cows from four different cattle breeds, density plots are fine (Figure \@ref(fig:butterfat-densitites)).

(ref:butterfat-densitites) Density estimates of the butterfat percentage in the milk of four cattle breeds. Data Source: Canadian Record of Performance for Purebred Dairy Cattle

```{r butterfat-densitites, fig.cap='(ref:butterfat-densitites)'}
cows %>% 
  mutate(breed = as.character(breed)) %>%
  filter(breed != "Canadian") -> cows_filtered

# compute densities for sepal lengths
cows_dens <- group_by(cows_filtered, breed) %>%
  do(ggplot2:::compute_density(.$butterfat, NULL)) %>%
  rename(butterfat = x)

# get the maximum values
cows_max <- filter(cows_dens, density == max(density)) %>%
  ungroup() %>%
  mutate(
    hjust = c(0, 0, 0, 0),
    vjust = c(0, 0, 0, 0),
    nudge_x = c(-0.2, -0.2, 0.1, 0.23),
    nudge_y = c(0.03, 0.03, -0.2, -0.06) 
  )

cows_p <- ggplot(cows_dens, aes(x = butterfat, y = density, color = breed, fill = breed)) + 
  geom_density_line(stat = "identity") +
  geom_text(
    data = cows_max,
    aes(
      label = breed, hjust = hjust, vjust = vjust,
      color = breed,
      x = butterfat + nudge_x, 
      y = density + nudge_y
    ),
    inherit.aes = FALSE,
    size = 12/.pt
  ) +
  scale_color_manual(
    values = darken(c("#56B4E9", "#E69F00", "#D55E00", "#009E73"), 0.3),
    breaks = c("Ayrshire", "Guernsey", "Holstein-Friesian", "Jersey"),
    guide = "none"
  ) +
  scale_fill_manual(
    values = c("#56B4E950", "#E69F0050", "#D55E0050", "#009E7350"),
    breaks = c("Ayrshire", "Guernsey", "Holstein-Friesian", "Jersey"),
    guide = "none"
  ) +
  scale_x_continuous(
    expand = c(0, 0),
    labels = scales::percent_format(accuracy = 1, scale = 1),
    name = "butterfat contents"
  ) +
  scale_y_continuous(limits = c(0, 1.99), expand = c(0, 0)) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(axis.line.x = element_blank())
  
cows_p
```

```{block type='rmdtip', echo=TRUE}
To visualize several distributions at once, kernel density plots will generally work better than histograms.
```


<!--chapter:end:visualizing_distributions_I.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")
```


# Visualizing distributions: Empirical cumulative distribution functions and q-q plots {#ecdf-qq}

In Chapter \@ref(histograms-density-plots), I described how we can visualize distributions with histograms or density plots. Both of these approaches are highly intuitive and visually appealing. However, as discussed in that chapter, they both share the limitation that the resulting figure depends to a substantial degree on parameters the user has to choose, such as the bin width for histograms and the bandwidth for density plots. As a result, both have to be considered as an interpretation of the data rather than a direct visualization of the data itself.

As an alternative to using histograms or density plots, we could simply show all the data points individually, as a point cloud. However, this approach becomes unwieldy for very large datasets, and in any case there is value in aggregate methods that highlight properties of the distribution rather than the individual data points. To solve this problem, statisticians have invented empirical cumulative distribution functions (ecdfs) and quantile--quantile (q-q) plots. These types of visualizations require no arbitrary parameter choices, and they show all of the data at once. Unfortunately, they are a little less intuitive than a histogram or a density plot is, and I don't see them used frequently outside of highly technical publications. They are quite popular among statisticians, though, and I think anybody interested in data visualization should be familiar with these techniques.


## Empirical cumulative distribution functions

To illustrate cumulative empirical distribution functions, I will begin with a hypothetical example that is closely modeled after something I deal with a lot as a professor in the classroom: a dataset of student grades. Assume our hypothetical class has 50 students, and the students just completed an exam on which they could score between 0 and 100 points. How can we best visualize the class performance, for example to determine appropriate grade boundaries?

We can plot the total number of students that have received at most a certain number of points versus all possible point scores. This plot will be an ascending function, starting at 0 for 0 points and ending at 50 for 100 points. A different way of thinking about this visualization is the following: We can rank all students by the number of points they obtained, in ascending order (so the student with the fewest points receives the lowest rank and the student with the most points the highest), and then plot the rank versus the actual points obtained. The result is an *empirical cumulative distribution function* (ecdf) or simply *cumulative distribution.* Each dot represents one student, and the lines visualize the highest student rank observed for any possible point value (Figure \@ref(fig:student-grades)).

(ref:student-grades) Empirical cumulative distribution function of student grades for a hypothetical class of 50 students.

```{r student-grades, fig.cap='(ref:student-grades)'}
set.seed(4211)
points = round(c(rnorm(47, mean = 82, sd = 10), 45, 51, 67))
points[points > 100] <- 100
student_data <- data.frame(points, rank = rank(points, ties.method = "random"))
ggplot(student_data, aes(x = points, y = 50*..y..)) + 
  stat_ecdf(geom = "step", color = "#0072B2") +
  geom_point(aes(y = rank), color = "#0072B2") +
  scale_x_continuous(limits = c(40, 102), expand = c(0, 0), breaks = 10*(4:10)) +
  scale_y_continuous(limits = c(-.5, 55), expand = c(0, 0), name = "student rank (ascending)") +
  coord_cartesian(clip = "off") +
  theme_dviz_grid() +
  theme(axis.line.x = element_blank())
```


You may wonder what happens if we rank the students the other way round, in descending order. This ranking simply flips the function on its head. The result is still an empirical cumulative distribution function, but the lines now represent the lowest student rank observed for any possible point value (Figure \@ref(fig:student-grades-desc)). 

(ref:student-grades-desc) Distribution of student grades plotted as a descending ecdf.

```{r student-grades-desc, fig.cap='(ref:student-grades-desc)'}
ggplot(student_data, aes(x = points, y = 51-50*..y..)) + 
  stat_ecdf(geom = "step", color = "#0072B2") +
  geom_point(aes(y = 51-rank), color = "#0072B2") +
  scale_x_continuous(limits = c(40, 102), expand = c(0, 0), breaks = 10*(4:10)) +
  scale_y_continuous(limits = c(-.5, 55), expand = c(0, 0), name = "student rank (descending)") +
  coord_cartesian(clip = "off") +
  theme_dviz_grid() +
  theme(axis.line.x = element_blank())

```

Ascending cumulative distribution functions are more widely known and more commonly used than descending ones, but both have important applications. Descending cumulative distribution functions are critical when we want to visualize highly skewed distributions (see Section \@ref(skewed-distributions)).

In practical applications, it is quite common to draw the ecdf without highlighting the individual points and to normalize the ranks by the maximum rank, so that the *y* axis represents the cumulative frequency (Figure \@ref(fig:student-grades-normalized)). 

(ref:student-grades-normalized) Ecdf of student grades. The student ranks have been normalized to the total number of students, such that the *y* values plotted correspond to the fraction of students in the class with at most that many points.

```{r student-grades-normalized, fig.cap='(ref:student-grades-normalized)'}
ggplot(student_data, aes(x = points, y = ..y..)) + 
  stat_ecdf(geom = "step", color = "#0072B2", size = 0.75) +
  scale_x_continuous(limits = c(40, 102), expand = c(0, 0), breaks = 10*(4:10)) +
  scale_y_continuous(limits = c(-.01, 1.01), expand = c(0, 0), name = "cumulative frequency") +
  coord_cartesian(clip = "off") +
  theme_dviz_grid() +
  theme(axis.line.x = element_blank())
```

We can directly read off key properties of the student grade distribution from this plot. For example, approximately a quarter of the students (25%) received less than 75 points. The median point value (corresponding to a cumulative frequency of 0.5) is 81. Approximately 20% of the students received 90 points or more.

I find ecdfs handy for assigning grade boundaries because they help me locate the exact cutoffs that minimize student unhappiness. For example, in this example, there's a fairly long horizontal line right below 80 points, followed by a steep rise right at 80. This feature is caused by three students receiving 80 points on their exam while the next poorer performing student received only 76. In this scenario, I might decide that everybody with a point score of 80 or more receives a B and everybody with 79 or less receives a C. The three students with 80 points are happy that they just made a B, and the student with 76 realizes that they would have had to perform much better to not receive a C. If I had set the cutoff at 77, the distribution of letter grades would have been exactly the same, but I might find the student with 76 points visiting my office hoping to negotiate their grade up. Likewise, if I had set the cutoff at 81, I would likely have had three students in my office trying to negotiate their grade.

## Highly skewed distributions {#skewed-distributions}

Many empirical datasets display highly skewed distributions, in particular with heavy tails to the right, and these distributions can be challenging to visualize. Examples of such distributions include the number of people living in different cities or counties, the number of contacts in a social network, the frequency with which individual words appear in a book, the number of academic papers written by different authors, the net worth of individuals, and the number of interaction partners of individual proteins in protein--protein interaction networks (@Clauset-et-al-2009). All these distributions have in common that their right tail decays slower than an exponential function. In practice, this means that very large values are not that rare, even if the mean of the distribution is small. An important class of such distributions are *power-law* distributions, where the likelihood to observe a value that is *x* times larger than some reference point declines as a power of *x*. To give a concrete example, consider net worth in the US, which is distributed according to a power law with exponent 2. At any given level of net worth (say, $1 million), people with half that net worth are four times as frequent, and people with twice that net worth are one-fourth as frequent. Importantly, the same relationship holds if we use $10,000 as reference point or if we use $100 million. For this reason, power-law distributions are also called *scale-free* distributions.

Here, I will first discuss the number of people living in different US counties according to the 2010 US Census. This distribution has a very long tail to the right. Even though most counties have relatively small numbers of inhabitants (the median is 25,857), a few counties have extremely large numbers of inhabitants (e.g., Los Angeles County, with 9,818,605 inhabitants). If we try to visualize the distribution of population counts as either a density plot or an ecdf, we obtain figures that are essentially useless (Figure \@ref(fig:county-populations)).

(ref:county-populations) Distribution of the number of inhabitants in US counties, according to the 2010 US Census. (a) Density plot. (b) Empirical cumulative distribution function.

```{r county-populations, fig.width = 5, fig.asp = 2*0.618, fig.cap='(ref:county-populations)'}
p1 <- ggplot(US_census, aes(x=pop2010)) + 
  geom_density(fill = "#56B4E9", color = "transparent") +
  scale_x_continuous(expand = c(0.01, 0), name = "number of inhabitants",
                     limits = c(0, 2.3e6),
                     breaks = 0.5e6*(0:4),
                     labels = label_log10) +
  scale_y_continuous(expand = c(0, 0), name = expression(paste("density [x", 10^-5, "]")),
                     breaks = c(0, 4e-6, 8e-6, 1.2e-5, 1.6e-5),
                     labels = c(0, 0.4, 0.8, 1.2, 1.6)) +
  coord_cartesian(clip = "off") +
  theme_dviz_grid(12) +
  theme(plot.margin = margin(3, 1.5, 12, 1.5))

p2 <- ggplot(US_census, aes(x=pop2010)) + 
  stat_ecdf(geom = "step", color = "#0072B2", pad = FALSE) +
  scale_x_continuous(expand = c(0.01, 0), name = "number of inhabitants",
                     limits = c(0, 2.3e6),
                     breaks = 0.5e6*(0:4),
                     labels = label_log10) +
  scale_y_continuous(expand = c(0.01, 0), name = "cumulative frequency") +
  coord_cartesian(clip = "off") +
  theme_dviz_grid(12) +
  theme(plot.margin = margin(3, 1.5, 12, 1.5))

plot_grid(p1, p2, ncol = 1, align = 'v', labels = 'auto')
```

The density plot (Figure \@ref(fig:county-populations)a) shows a sharp peak right at 0 and virtually no details of the distribution are visible. Similarly, the ecdf (Figure \@ref(fig:county-populations)b) shows a rapid rise near 0 and again no details of the distribution are visible. For this particular dataset, we can log-transform the data and visualize the distribution of the log-transformed values. This transformation works here because the population numbers in counties is not actually a power law, but instead follow a nearly perfect log-normal distribution (see Section \@ref(qq-plots)). Indeed, the density plot of the log-transformed values shows a nice bell curve and the corresponding ecdf shows a nice sigmoidal shape (Figure \@ref(fig:county-populations-log)).

(ref:county-populations-log) Distribution of the logarithm of the number of inhabitants in US counties. (a) Density plot. (b) Empirical cumulative distribution function.

```{r county-populations-log, fig.width = 5, fig.asp = 2*0.618, fig.cap='(ref:county-populations-log)'}
p1_log <- ggplot(US_census, aes(x=log10(pop2010))) + 
  geom_density(fill = "#56B4E9", color = "transparent") +
  scale_x_continuous(
    expand = c(0.01, 0),
    name = expression(paste("log"["10"], "(number of inhabitants)"))
  ) +
  scale_y_continuous(expand = c(0, 0), name = "density") +
  coord_cartesian(clip = "off") +
  theme_dviz_grid(12) +
  theme(plot.margin = margin(3, 1.5, 12, 1.5))

p2_log <- ggplot(US_census, aes(x=log10(pop2010))) + 
  stat_ecdf(geom = "step", color = "#0072B2", pad = FALSE) +
  scale_x_continuous(
    expand = c(0.01, 0),
    name = expression(paste("log"["10"], "(number of inhabitants)"))
  ) +
  scale_y_continuous(expand = c(0.01, 0), name = "cumulative frequency") +
  coord_cartesian(clip = "off") +
  theme_dviz_grid(12) +
  theme(plot.margin = margin(3, 1.5, 12, 1.5))

plot_grid(p1_log, p2_log, ncol = 1, align = 'v', labels = 'auto')
```

To see that this distribution is not a power law, we plot it as a descending ecdf with logarithmic *x* and *y* axes. In this visualization, a power law appears as a perfect straight line. For the population counts in counties, the right tail forms almost but not quite a straight line on the descending log-log ecdf plot (Figure \@ref(fig:county-populations-tail-log-log)).

(ref:county-populations-tail-log-log) Relative frequency of counties with at least that many inhabitants versus the number of county inhabitants.

```{r county-populations-tail-log-log, fig.cap='(ref:county-populations-tail-log-log)'}
ggplot(US_census, aes(x=pop2010, y = 1-..y..)) + 
  stat_ecdf(geom = "step", color = "#0072B2", size = 0.75, pad = FALSE) +
  scale_x_log10(expand = c(0.01, 0),
                breaks = c(1e2, 1e3, 1e4, 1e5, 1e6, 1e7),
                labels = c(expression(10^2), expression(10^3), expression(10^4),
                           expression(10^5), expression(10^6), expression(10^7)),
                name = "number of county inhabitants") +
  scale_y_log10(expand = c(0.01, 0), breaks = c(1e-3, 1e-2, 1e-1, 1), name = "relative frequency") +
  theme_dviz_grid() +
  theme(plot.margin = margin(3.5, 7, 3.5, 1.5))
```

As a second example, I will use the distribution of word frequencies for all words that appear in the novel Moby Dick. This distribution follows a perfect power law. When plotted as descending ecdf with logarithmic axes, we see a nearly perfect straight line (Figure \@ref(fig:word-counts-tail-log-log)).

(ref:word-counts-tail-log-log) Distribution of word counts in the novel Moby Dick. Shown is the relative frequency of words that occur at least that many times in the novel versus the number of times words are used.

```{r word-counts-tail-log-log, fig.cap='(ref:word-counts-tail-log-log)' }
ggplot(Moby_Dick, aes(x=count, y = 1-..y..)) + 
  stat_ecdf(geom = "step", color = "#0072B2", size = 0.75, pad = FALSE) +
  scale_x_log10(expand = c(0.01, 0), breaks = c(1, 10, 100, 1000, 10000),
                name = "number of times word is used") +
  scale_y_log10(expand = c(0.01, 0), breaks = c(1e-5, 1e-4, 1e-3, 1e-2, 1e-1, 1),
                labels = c(expression(10^-5), expression(10^-4), expression(10^-3),
                           expression(10^-2), expression(10^-1), expression(10^0)),
                name = "fraction of words") +
  theme_dviz_grid()
```

## Quantile--quantile plots {#qq-plots}

Quantile--quantile (q-q) plots are a useful visualization when we want to determine to what extent the observed data points do or do not follow a given distribution. Just like ecdfs, q-q plots are also based on ranking the data and visualizing the relationship between ranks and actual values. However, in q-q plots we don't plot the ranks directly, we use them to predict where a given data point should fall if the data were distributed according to a specified reference distribution. Most commonly, q-q plots are constructed using a normal distribution as the reference. To give a concrete example, assume the actual data values have a mean of 10 and a standard deviation of 3. Then, assuming a normal distribution, we would expect a data point ranked at the 50th percentile to lie at position 10 (the mean), a data point at the 84th percentile to lie at position 13 (one standard deviation above from the mean), and a data point at the 2.3rd percentile to lie at position 4 (two standard deviations below the mean). We can carry out this calculation for all points in the dataset and then plot the observed values (i.e., values in the dataset) against the theoretical values (i.e., values expected given each data point's rank and the assumed reference distribution).

When we perform this procedure for the student grades distribution from the beginning of this chapter, we obtain Figure \@ref(fig:student-grades-qq).

(ref:student-grades-qq) q-q plot of student grades.

```{r student-grades-qq, fig.width = 5, fig.asp = .98, fig.cap='(ref:student-grades-qq)'}
# estimate distribution parameters (mean and sd)
params <- as.list(MASS::fitdistr(student_data$points, "normal")$estimate)

# axis line segments
df_segment <- data.frame(
  x = c(50, -Inf), 
  xend = c(100, -Inf),
  y = c(-Inf, 50),
  yend = c(-Inf, 100)
)

ggplot(student_data, aes(sample = points)) + 
  geom_abline(slope = 1, intercept = 0, color = "grey70") +
  stat_qq(dparams = params, color = "#0072B2") +
  geom_segment(
    data = df_segment,
    aes(x = x, xend = xend, y = y, yend = yend),
    size = 0.5, inherit.aes = FALSE) +
  scale_x_continuous(limits = c(43, 107), expand = c(0, 0), breaks = 10*(5:10)) +
  scale_y_continuous(limits = c(43, 102), expand = c(0, 0), breaks = 10*(5:10), name = "observed") +
  coord_fixed(clip = "off") +
  theme_dviz_open() +
  theme(axis.line = element_blank())
```

The solid line here is not a regression line but indicates the points where *x* equals *y*, i.e., where the observed values equal the theoretical ones. To the extent that points fall onto that line, the data follow the assumed distribution (here, normal). We see that the student grades follow mostly a normal distribution, with a few deviations at the bottom and at the top (a few students performed worse than expected on either end). The deviations from the distribution at the top end are caused by the maximum point value of 100 in the hypothetical exam; regardless of how good the best student is, he or she could at most obtain 100 points.


We can also use a q-q plot to test my assertion from earlier in this chapter that the population counts in US counties follow a log-normal distribution. If these counts are log-normally distributed, then their log-transformed values are normally distributed and hence should fall right onto the *x* = *y* line. When making this plot, we see that the agreement between the observed and the theoretical values is exceptional (Figure \@ref(fig:county-populations-qq)). This demonstrates that the distribution of population counts among counties is indeed log-normal.

(ref:county-populations-qq) q-q plot of the logarithm of the number of inhabitants in US counties.

```{r county-populations-qq, fig.width = 5, fig.asp = 1, fig.cap='(ref:county-populations-qq)'}
# estimate distribution parameters (mean and sd)
params <- as.list(MASS::fitdistr(log(US_census$pop2010), "normal")$estimate)

# axis line segments
df_segment <- data.frame(
  x = c(5, -Inf), 
  xend = c(15, -Inf),
  y = c(-Inf, 5),
  yend = c(-Inf, 15)
)

ggplot(US_census, aes(sample = log(pop2010))) + 
  geom_abline(slope = 1, intercept = 0, color = "grey70") +
  stat_qq(dparams = params, color = "#0072B2") +
  geom_segment(
    data = df_segment,
    aes(x = x, xend = xend, y = y, yend = yend),
    size = 0.5, inherit.aes = FALSE) +
  scale_x_continuous(limits = c(4, 16), expand = c(0, 0), breaks = 5+2.5*(0:4)) +
  scale_y_continuous(limits = c(4, 16), expand = c(0, 0), name = "observed", breaks = 5+2.5*(0:4)) +
  coord_fixed(clip = "off") + 
  theme_dviz_open() +
  theme(axis.line = element_blank())
```



<!--chapter:end:visualizing_distributions_II.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(tidyr)
library(forcats)
library(ggforce)
library(ggridges)

main_size = 14 / .pt
small_rel <- 12/14
small_size <- small_rel * main_size
```


# Visualizing proportions {#visualizing-proportions}

We often want to show how some group, entity, or amount breaks down into individual pieces that each represent a *proportion* of the whole. Common examples include the proportions of men and women in a group of people, the percentages of people voting for different political parties in an election, or the market shares of companies. The archetypal such visualization is the pie chart, omnipresent in any business presentation and much maligned among data scientists. As we will see, visualizing proportions can be challenging, in particular when the whole is broken into many different pieces or when we want to see changes in proportions over time or across conditions. There is no single ideal visualization that always works. To illustrate this issue, I discuss a few different scenarios that each call for a different type of visualization.

```{block type='rmdtip', echo=TRUE}
Remember: You always need to pick the visualization that best fits your specific dataset and that highlights the key data features you want to show.
```


## A case for pie charts

From 1961 to 1983, the German parliament (called the *Bundestag*) was composed of members of three different parties, CDU/CSU, SPD, and FDP. During most of this time, CDU/CSU and SPD had approximately comparable numbers of seats, while the FDP typically held only a small fraction of seats. For example, in the 8th Bundestag, from 1976--1980, the CDU/CSU held 243 seats, SPD 214, and FDP 39, for a total of 496. Such parliamentary data is most commonly visualized as a pie chart (Figure \@ref(fig:bundestag-pie)).

(ref:bundestag-pie) Party composition of the 8th German Bundestag, 1976--1980, visualized as a pie chart. This visualization shows clearly that the ruling coalition of SPD and FDP had a small majority over the opposition CDU/CSU.

```{r, bundestag-pie, fig.cap='(ref:bundestag-pie)'}
# calculate the start and end angles for each pie
bund_pie <- bundestag %>%
  arrange(seats) %>%
  mutate(
    seat_total = sum(seats),
    end_angle = 2*pi*cumsum(seats)/seat_total,   # ending angle for each pie slice
    start_angle = lag(end_angle, default = 0),   # starting angle for each pie slice
    mid_angle = 0.5*(start_angle + end_angle),   # middle of each pie slice, for the text label
    hjust = ifelse(mid_angle>pi, 1, 0),
    vjust = ifelse(mid_angle<pi/2 | mid_angle>3*pi/2, 0, 1)
  )

rpie = 1
rlabel = 1.05 * rpie

ggplot(bund_pie) + 
  geom_arc_bar(
    aes(
      x0 = 0, y0 = 0, r0 = 0, r = rpie,
      start = start_angle, end = end_angle, fill = party
    ),
    color = "white", size = 0.5
  ) +
  geom_text(
    aes(
      x = rlabel*sin(mid_angle),
      y = rlabel*cos(mid_angle),
      label = party,
      hjust = hjust, vjust = vjust
    ),
    family = dviz_font_family, size = main_size
  ) +
  geom_text(
    aes(
      x = 0.6*sin(mid_angle),
      y = 0.6*cos(mid_angle),
      label = seats
    ),
    family = dviz_font_family, size = main_size,
    color = c("black", "white", "white")
  ) +
  coord_fixed(clip = "off") +
  scale_x_continuous(
    limits = c(-1.5, 1.5),
    expand = c(0, 0),
    name = "",
    breaks = NULL,
    labels = NULL
  ) +
  scale_y_continuous(
    limits = c(-1.01, 1.15),
    expand = c(0, 0),
    name = "",
    breaks = NULL,
    labels = NULL
  ) +
  scale_fill_manual(
    values = bund_pie$colors[order(bund_pie$party)]
  ) +
  theme_dviz_map() +
  theme(
    legend.position = "none",
    plot.margin = margin(3.5, 1.5, 3.5, 1.5)
  )
```

A pie chart breaks a circle into slices such that the area of each slice is proportional to the fraction of the total it represents. The same procedure can be performed on a rectangle, and the result is a stacked bar chart (Figure \@ref(fig:bundestag-stacked-bars)). Depending on whether we slice the bar vertically or horizontally, we obtain vertically stacked bars (Figure \@ref(fig:bundestag-stacked-bars)a) or horizontally stacked bars (Figure \@ref(fig:bundestag-stacked-bars)b).

(ref:bundestag-stacked-bars) Party composition of the 8th German Bundestag, 1976--1980, visualized as stacked bars. (a) Bars stacked vertically. (b) Bars stacked horizontally. It is not immediately obvious that SPD and FDP jointly had more seats than CDU/CSU.

```{r, bundestag-stacked-bars, fig.width = 5.5*6/4.2, fig.cap='(ref:bundestag-stacked-bars)'}
bundestag <- mutate(bundestag,
                    label_y = cumsum(seats) - seats/2)

bt_bars_stacked_base <- ggplot(bundestag, aes(x = 1, y = seats, fill = factor(party, levels = rev(party)))) + 
  geom_col(position = "stack", color = "white") + 
  geom_text(
    aes(x = 1., y = label_y, label = seats), 
    size = main_size, family = dviz_font_family,
    color = c("white", "white", "black")
  ) +
  scale_y_continuous(expand = c(0, 0)) +
  scale_x_discrete(expand = c(0, 0), breaks = NULL, name = NULL) +
  scale_fill_manual(values = rev(bundestag$colors), guide = "none")

bt_bars_yax <- axis_canvas(bt_bars_stacked_base, axis = "y") +
  geom_text(
    data = bundestag,
    aes(x = 0.06, y = label_y, label = party),
    hjust = 0, vjust = 0.5, size = small_size,
    family = dviz_font_family
  ) +
  scale_x_continuous(expand = c(0, 0), limits = c(0, 1))

bt_bars_stacked <- insert_yaxis_grob(
  bt_bars_stacked_base + 
    theme_dviz_hgrid() +
    theme(
      axis.ticks = element_line(color = "gray70"),
      plot.margin = margin(7, 1.5, 7, 1.5)
    ),
  bt_bars_yax, grid::unit(.5, "null"))

bt_bars_xax <- axis_canvas(bt_bars_stacked_base, axis = "y") +
  geom_text(
    data = bundestag,
    aes(x = 0., y = label_y, label = party, hjust = 0.5, vjust = 0, size = small_size),
    family = dviz_font_family
  ) +
  scale_x_continuous(expand = c(0, 0), limits = c(0, 1)) +
  coord_flip()

bt_bars_hstacked <- insert_xaxis_grob(
  bt_bars_stacked_base + coord_flip() +
    scale_y_continuous(expand = c(0, 0), position = "right") +
    theme_dviz_vgrid() +
    theme(
      axis.ticks = element_line(color = "gray70"),
      plot.margin = margin(3, 1.5, 3, 3)
    ),
  bt_bars_xax, grid::unit(14, "pt"), position = "bottom")

plot_grid(
  bt_bars_stacked,
  plot_grid(
    NULL, bt_bars_hstacked, NULL,
    ncol = 1, rel_heights = c(1, 6, 7.5)
  ),
  rel_widths = c(4, 7), labels = "auto"
)
```

We can also take the bars from Figure \@ref(fig:bundestag-stacked-bars)a and place them side-by-side rather than stacking them on top of each other. This visualization makes it easier to perform a direct comparison of the three groups, though it obscures other aspects of the data (Figure \@ref(fig:bundestag-bars-side-by-side)). Most importantly, in a side-by-side bar plot the relationship of each bar to the total is not visually obvious.

(ref:bundestag-bars-side-by-side) Party composition of the 8th German Bundestag, 1976--1980, visualized as side-by-side bars. As in Figure \@ref(fig:bundestag-stacked-bars), it is not immediately obvious that SPD and FDP jointly had more seats than CDU/CSU.

```{r bundestag-bars-side-by-side, fig.width = 5, fig.asp = 3/4, fig.cap='(ref:bundestag-bars-side-by-side)'}
bt_bars <- ggplot(bundestag, aes(x = factor(party, levels = bundestag$party), y = seats, fill = party)) + 
  geom_col() + 
  geom_text(aes(label = seats), size = main_size, vjust = 2, color = c("white", "white", "black")) +
  scale_x_discrete(name = NULL) +
  scale_y_continuous(expand = c(0, 0)) +
  scale_fill_manual(values = bundestag$colors[order(bundestag$party)], guide = "none") + 
  #geom_hline(yintercept = c(50, 100, 150, 200), color = "#ffffff70", size = .5) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    axis.ticks.x = element_blank()
  )

bt_bars
```

Many authors categorically reject pie charts and argue in favor of side-by-side or stacked bars. Others defend the use of pie charts in some applications. My own opinion is that none of these visualizations is consistently superior over any other. Depending on the features of the dataset and the specific story you want to tell, you may want to favor one or the other approach. In the case of the 8th German Bundestag, I think that a pie chart is the best option. It shows clearly that the ruling coalition of SPD and FDP jointly had a small majority over the CDU/CSU (Figure \@ref(fig:bundestag-pie)). This fact is not visually obvious in any of the other plots (Figures \@ref(fig:bundestag-stacked-bars) and \@ref(fig:bundestag-bars-side-by-side)). 

In general, pie charts work well when the goal is to emphasize simple fractions, such as one-half, one-third, or one-quarter. They also work well when we have very small datasets. A single pie chart, as in Figure \@ref(fig:bundestag-pie), looks just fine, but a single column of stacked bars, as in Figure \@ref(fig:bundestag-stacked-bars)a, looks awkward. Stacked bars, on the other hand, can work for side-by-side comparisons of multiple conditions or in a time series, and side-by-side bars are preferred when we want to directly compare the individual fractions to each other. A summary of the various pros and cons of pie charts, stacked bars, and side-by-side bars is provided in Table \@ref(tab:pros-cons-pie-bar). 

Table: (\#tab:pros-cons-pie-bar) Pros and cons of common approaches to visualizing proportions: pie charts, stacked bars, and side-by-side bars. 

----------------------------------------------------------------------------------------
                                    Pie chart         Stacked bars      Side-by-side bars
-----------------------------  ------------------- ------------------- -------------------
Clearly visualizes the data             ✔                 ✔                   ✖
as proportions of a whole

Allows easy visual comparison           ✖                 ✖                   ✔ 
of the relative proportions 

Visually emphasizes simple              ✔                 ✖                   ✖
fractions, such as 1/2, 1/3,
1/4

Looks visually appealing                ✔                 ✖                   ✔
even for very small datasets

Works well when the whole is            ✖                 ✖                   ✔ 
broken into many pieces

Works well for the                      ✖                 ✔                   ✖
visualization of many sets of
proportions or time series
of proportions
----------------------------------------------------------------------------------------


## A case for side-by-side bars {#side-by-side-bars}

I will now demonstrate a case where pie charts fail. This example is modeled after a critique of pie charts originally posted on Wikipedia [@Schutz-piecharts]. Consider the hypothetical scenario of five companies, A, B, C, D, and E, who all have roughly comparable market share of approximately 20%. Our hypothetical dataset lists the market share of each company for three consecutive years. When we visualize this dataset with pie charts, it is difficult to see what exactly is going on (Figure \@ref(fig:marketshare-pies)). It appears that the market share of company A is growing and the one of company E is shrinking, but beyond this one observation we can't tell what's going on. In particular, it is unclear how exactly the market shares of the different companies compare within each year.


(ref:marketshare-pies) Market share of five hypothetical companies, A--E, for the years 2015--2017, visualized as pie charts. This visualization has two major problems: 1. A comparison of relative market share within years is nearly impossible. 2. Changes in market share across years are difficult to see.

```{r marketshare-pies, fig.width = 5*6/4.2, fig.asp = .35, fig.cap='(ref:marketshare-pies)'}
# calculate the start and end angles for each pie
market_pies_df <- marketshare %>%
  group_by(year) %>%
  mutate(total = sum(percent),
         end_angle = 2*pi*cumsum(percent)/total,      # ending angle for each pie slice
         start_angle = lag(end_angle, default = 0),   # starting angle for each pie slice
         mid_angle = 0.5*(start_angle + end_angle),   # middle of each pie slice, for the text label
         hjust = ifelse(mid_angle>pi, 1, 0),
         vjust = ifelse(mid_angle<pi/2 | mid_angle>3*pi/2, 0, 1))

rpie = 1
rlabel = 1.05 * rpie

market_pies <- ggplot(market_pies_df) + 
  geom_arc_bar(
    aes(
      x0 = 0, y0 = 0, r0 = 0, r = rpie,
      start = start_angle, end = end_angle, fill = company
    ),
    color = NA
  ) +
  geom_text(
    aes(x = rlabel*sin(mid_angle), y = rlabel*cos(mid_angle), label = company, hjust = hjust, vjust = vjust),
    family = dviz_font_family,
    size = small_size
  ) +
  coord_fixed() +
  facet_wrap(~year) +
  scale_x_continuous(limits = c(-1.2, 1.2), expand = c(0, 0), name = NULL, breaks = NULL, labels = NULL) +
  scale_y_continuous(limits = c(-1.2, 1.2), expand = c(0, 0), name = NULL, breaks = NULL, labels = NULL) +
  scale_fill_OkabeIto(order = c(1:3, 5, 4)) + 
  guides(fill = "none") +
  theme_dviz_open() +
  theme(
    axis.line = element_blank(),
    axis.title = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = grid::unit(0, "pt"),
    plot.margin = margin(7, 7, 0, 7),
    legend.position = "none",
    strip.background = element_blank(),
    strip.text.x = element_text(size= 14, margin = margin(0, 0, 0.1, 0))
  )

stamp_bad(market_pies)
```

The picture becomes a little clearer when we switch to stacked bars (Figure \@ref(fig:marketshare-stacked)). Now the trends of a growing market share for company A and a shrinking market share for company E are clearly visible. However, the relative market shares of the five companies within each year are still hard to compare. And it is difficult to compare the market shares of companies B, C, and D across years, because the bars are shifted relative to each other across years. This is a general problem of stacked-bar plots, and the main reason why I normally do not recommend this type of visualization.

(ref:marketshare-stacked) Market share of five hypothetical companies for the years 2015--2017, visualized as stacked bars. This visualization has two major problems: 1. A comparison of relative market shares within years is difficult. 2. Changes in market share across years are difficult to see for the middle companies B, C, and D, because the location of the bars changes across years.

```{r marketshare-stacked, fig.cap='(ref:marketshare-stacked)'}
stacked_bars <- ggplot(marketshare, aes(x = year, y = percent, fill = company)) + 
  geom_col(position = "stack") + 
  scale_y_continuous(
    name = "market share",
    labels = scales::percent_format(accuracy = 1, scale = 1),
    expand = c(0, 0)
  ) +
  scale_fill_OkabeIto(order = c(1:3, 5, 4)) + 
  theme_dviz_open() +
  theme(plot.margin = margin(14, 7, 3, 1.5))

stamp_bad(stacked_bars)
```

For this hypothetical data set, side-by-side bars are the best choice (Figure \@ref(fig:marketshare-side-by-side)). This visualization highlights that both companies A and B have increased their market share from 2015 to 2017 while both companies D and E have reduced theirs. It also shows that market shares increase sequentially from company A to E in 2015 and similarly decrease in 2017.

(ref:marketshare-side-by-side) Market share of five hypothetical companies for the years 2015--2017, visualized as side-by-side bars.

```{r marketshare-side-by-side, fig.cap='(ref:marketshare-side-by-side)'}
ggplot(marketshare, aes(x = company, y = percent, fill = company)) + 
  geom_col() + 
  facet_wrap(~year) +
  scale_y_continuous(
    name = "market share",
    labels = scales::percent_format(accuracy = 1, scale = 1),
    expand = c(0, 0)
  ) +
  scale_fill_OkabeIto(order = c(1:3, 5, 4), guide = "none") + 
  theme_dviz_open() +
  theme(strip.background = element_blank())
```


## A case for stacked bars and stacked densities {#stacked-densities}

In Section \@ref(side-by-side-bars), I wrote that I don't normally recommend sequences of stacked bars, because the location of the internal bars shifts along the sequence. However, the problem of shifting internal bars disappears if there are only two bars in each stack, and in those cases the resulting visualization can be quite clear. As an example, consider the proportion of women in a country's national parliament. We will specifically look at the African country Rwanda, which as of 2016 tops the list of countries with the highest proportion of female parliament members. Rwanda has had a majority female parliament since 2008, and since 2013 nearly two-thirds of its members of parliament are female. To visualize how the proportion of women in the Rwandan parliament has changed over time, we can draw a sequence of stacked bar graphs (Figure \@ref(fig:women-parliament)). This figure provides an immediate visual representation of the changing proportions over time. To help the reader see exactly when the majority turned female, I have added a dashed horizontal line at 50%. Without this line, it would be near impossible to determine whether from 2003 to 2007 the majority was male or female. I have not added similar lines at 25% and 75%, to avoid making the figure too cluttered.

(ref:women-parliament) Change in the gender composition of the Rwandan parliament over time, 1997 to 2016. Data source: Inter-Parliamentary Union (IPU), ipu.org.

```{r women-parliament, fig.width = 6, fig.asp = .55, fig.cap = '(ref:women-parliament)'}
ccode = "RWA" # Rwanda
#ccode = "BEL" # Belgium
#ccode = "ARB" # Arab world
#ccode = "BOL" # Bolivia
#ccode = "EUU" # European Union

women_parliaments %>% filter(country_code == ccode & year > 1990) %>%
  mutate(women = perc_women, men = 100 - perc_women) %>%
  select(-perc_women) %>%
  gather(gender, percent, women, men) %>%
  mutate(gender = factor(gender, levels = c("women", "men"))) -> women_rwanda

plot_base <- ggplot(women_rwanda, aes(x = year, y = percent, fill = gender)) +
    #geom_col(position = "stack", width = .9, color = "white") +
    geom_col(position = "stack", width = 1, color = "#FFFFFF", size = .75, alpha = 0.8) +
    geom_hline(
      yintercept = c(50),
      color = "#000000FF", size = 0.4, linetype = 2
      #color = "#FFFFFFA0"
    ) +
    geom_hline(yintercept = 100, color = "black") +
    scale_x_continuous(expand = c(0, 0)) +
    scale_y_continuous(
      name = "relative proportion",
      labels = scales::percent_format(accuracy = 1, scale = 1),
      expand = c(0, 0)
    ) +
    scale_fill_manual(values = c("#D55E00E0", "#0072B2E0"), guide = "none") +
    coord_cartesian(clip = "off") +
    theme_dviz_open() + 
    theme(
      #axis.ticks.y = element_blank(),
      #axis.ticks.x = element_blank(),
      #axis.line.x = element_blank(),
      axis.line.y = element_blank(),
      plot.margin = margin(14, 1.5, 3, 1.5)
    )

# calculate label position
labels <- filter(women_rwanda, year == max(year)) %>%
  mutate(pos = 100 - cumsum(percent) + 0.5*percent)

yax <- axis_canvas(plot_base, axis = "y") +
  geom_text(data = labels, aes(y = pos, label = paste0(" ", gender)),
            family = dviz_font_family,
            x = 0, hjust = 0, size = 14/.pt)

ggdraw(insert_yaxis_grob(plot_base, yax, grid::unit(.15, "null")))

```

If we want to visualize how proportions change in response to a continuous variable, we can switch from stacked bars to stacked densities. Stacked densities can be thought of as the limiting case of infinitely many infinitely small stacked bars arranged side-by-side. The densities in stacked-density plots are typically obtained from kernel density estimation, as described in Chapter \@ref(histograms-density-plots), and I refer you to that chapter for a general discussion of the strengths and weaknesses of this method.

To give an example where stacked densities may be appropriate, consider the health status of people as a function of age. Age can be considered a continuous variable, and visualizing the data in this way works reasonably well (Figure \@ref(fig:health-vs-age)). Even though we have four health categories here, and I'm generally not a fan of stacking multiple conditions, as discussed above, I think in this case the figure is acceptable. We can see clearly that overall health declines as people age, and we can also see that despite this trend, over half of the population remain in good or excellent health until very old age.

(ref:health-vs-age) Health status by age, as reported by the general social survey (GSS).

```{r health-vs-age, fig.asp = .5, fig.cap='(ref:health-vs-age)'}
df_health <- select(happy, age, health) %>%
  na.omit()

# color brewer 5-class PuBu
colors = c('#f1eef6', '#bdc9e1', '#74a9cf', '#2b8cbe', '#045a8d')[5:1]

p_health <- ggplot(df_health, aes(x = age, y = ..count.., fill = health, color = health)) + 
  geom_density(position = "fill") +
  #geom_hline(yintercept = c(.25, .50, .75), color = "#FFFFFF60") +
  scale_x_continuous(name = "age (years)", expand = c(0, 0)) +
  scale_y_continuous(
    expand = c(0, 0), name = "relative proportion",
    labels = scales::percent
  ) + 
  scale_color_manual(values = colors) + 
  scale_fill_manual(values = colors) +
  theme_dviz_open() + 
  theme(
    #axis.ticks.y = element_blank(),
    #axis.ticks.x = element_blank(),
    axis.line.x = element_blank(),
    axis.line.y = element_blank(),
    plot.margin = margin(7, 7, 3, 1.5)
  )

df_marital <- select(happy, age, marital) %>%
  na.omit() %>%
  filter(marital != "separated") %>% # remove separated to make plot simpler
  mutate(marital = factor(marital, levels = c("widowed", "divorced", "married", "never married")))

p_marital <- ggplot(df_marital, aes(x = age, y = ..count.., fill = marital, color = marital)) + 
  geom_density(position = "fill") +
  scale_x_continuous(name = "age (years)", expand = c(0, 0)) +
  scale_y_continuous(
    expand = c(0, 0), name = "relative proportion",
    labels = scales::percent
  ) + 
  scale_color_manual(values = colors, name = "marital status") + 
  scale_fill_manual(values = colors, name = "marital status") +
  theme_dviz_open() + 
  theme(
    #axis.ticks.y = element_blank(),
    axis.line.x = element_blank(),
    axis.line.y = element_blank(),
    plot.margin = margin(7, 7, 3, 1.5)
  )

p_aligned <- align_plots(p_health, p_marital, align = 'v')
ggdraw(p_aligned[[1]])
```

Nevertheless, this figure has a major limitation: By visualizing the proportions of the four health conditions as percent of the total, the figure obscures that there are many more young people than old people in the dataset. Thus, even though the *percentage* of people reporting to be in good health remains approximately unchanged across ages spanning seven decades, the *absolute number* of people in good health declines as the total number of people at a given age declines. I will present a potential solution to this problem in the next section.


## Visualizing proportions separately as parts of the total

Side-by-side bars have the problem that they don't clearly visualize the size of the individual parts relative to the whole and stacked bars have the problem that the different bars cannot be compared easily because they have different baselines. We can resolve these two issues by making a separate plot for each part and in each plot showing the respective part relative to the whole. For the health dataset of Figure \@ref(fig:health-vs-age), this procedure results in Figure \@ref(fig:health-vs-age-facets). The overall age distribution in the dataset is shown as the shaded gray areas, and the age distributions for each health status are shown in blue. This figure highlights that in absolute terms, the number people with excellent or good health declines past ages 30--40, while the number of people with fair health remains approximately constant across all ages.

(ref:health-vs-age-facets) Health status by age, shown as proportion of the total number of people in the survey. The colored areas show the density estimates of the ages of people with the respective health status and the gray areas show the overall age distribution.

```{r health-vs-age-facets, fig.width = 5.5*6/4.2, fig.asp = 0.35, fig.cap='(ref:health-vs-age-facets)'}
ggplot(mutate(df_health, health = fct_rev(health)), aes(x = age, y = ..count..)) +
  geom_density_line(data = select(df_health, -health), aes(fill = "all people surveyed   "), color = "transparent") +
  geom_density_line(aes(fill = "highlighted group"), color = "transparent") +
  facet_wrap(~health, nrow = 1) +
  scale_x_continuous(name = "age (years)", limits = c(15, 98), expand = c(0, 0)) +
  scale_y_continuous(name = "count", expand = c(0, 0)) +
  scale_fill_manual(
    values = c("#b3b3b3a0", "#2b8cbed0"),
    name = NULL,
    guide = guide_legend(direction = "horizontal")
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    strip.text = element_text(size = 14, margin = margin(0, 0, 0.2, 0, "cm")),
    legend.position = "bottom",
    legend.justification = "right",
    legend.margin = margin(4.5, 0, 1.5, 0, "pt"),
    legend.spacing.x = grid::unit(4.5, "pt"),
    legend.spacing.y = grid::unit(0, "pt"),
    legend.box.spacing = grid::unit(0, "cm")
  )
```



To provide a second example, let's consider a different variable from the same survey: marital status. Marital status changes much more drastically with age than does health status, and a stacked densities plot of marital status vs age is not very illuminating (Figure \@ref(fig:marital-vs-age)).

(ref:marital-vs-age) Marital status by age, as reported by the general social survey (GSS). To simplify the figure, I have removed a small number of cases that report as separated. I have labeled this figure as "bad" because the frequency of people who have never been married or are widowed changes so drastically with age that the age distributions of married and divorced people are highly distorted and difficult to interpret. 

```{r marital-vs-age, fig.asp = 0.5, fig.cap='(ref:marital-vs-age)'}
stamp_bad(p_aligned[[2]])
```

The same dataset visualized as partial densities is much clearer (Figure \@ref(fig:marital-vs-age-facets)). In particular, we see that the proportion of married people peaks around the late 30s, the proportion of divorced people peaks around the early 40s, and the proportion of widowed people peaks around the mid 70s.

(ref:marital-vs-age-facets) Marital status by age, shown as proportion of the total number of people in the survey. The colored areas show the density estimates of the ages of people with the respective marital status, and the gray areas show the overall age distribution.

```{r marital-vs-age-facets, fig.width = 5.5*6/4.2, fig.asp = 0.35, fig.cap='(ref:marital-vs-age-facets)'}
ggplot(mutate(df_marital, marital = fct_rev(marital)), aes(x = age, y = ..count..)) +
  geom_density_line(data = select(df_marital, -marital), aes(fill = "all people surveyed  "), color = "transparent") +
  geom_density_line(aes(fill = "highlighted group"), color = "transparent") +
  facet_wrap(~marital, nrow = 1) +
  scale_x_continuous(name = "age (years)", limits = c(15, 98), expand = c(0, 0)) +
  scale_y_continuous(name = "count", expand = c(0, 0)) +
  scale_fill_manual(
    values = c("#b3b3b3a0", "#2b8cbed0"),
    name = NULL,
    guide = guide_legend(direction = "horizontal")
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    strip.text = element_text(size = 14, margin = margin(0, 0, 0.2, 0, "cm")),
    legend.position = "bottom",
    legend.justification = "right",
    legend.margin = margin(4.5, 0, 1.5, 0, "pt"),
    legend.spacing.x = grid::unit(4.5, "pt"),
    legend.spacing.y = grid::unit(0, "pt"),
    legend.box.spacing = grid::unit(0, "cm")
  )
```

However, one downside of Figure \@ref(fig:marital-vs-age-facets) is that this representation doesn't make it easy to determine relative proportions at any given point in time. For example, if we wanted to know at what age more than 50% of all people surveyed are married, we could not easily tell from Figure \@ref(fig:marital-vs-age-facets). To answer this question, we can instead use the same type of display but show relative proportions instead of absolute counts along the *y* axis (Figure \@ref(fig:marital-vs-age-proportions)). Now we see that married people are in the majority starting in their late 20s, and widowed people are in the majority starting in their mid 70s.

(ref:marital-vs-age-proportions) Marital status by age, shown as proportion of the total number of people in the survey. The areas colored in blue show the percent of people at the given age with the respective status, and the areas colored in gray show the percent of people with all other marital statuses.

```{r marital-vs-age-proportions, fig.width = 5.5*6/4.2, fig.asp = 0.35, fig.cap='(ref:marital-vs-age-proportions)'}
df_marital2 <- rbind(
  mutate(df_marital,
    marital = as.character(fct_collapse(marital, `never married` = "never married", aother = c("married", "divorced", "widowed"))),
    highlight = "never married"
  ),
  mutate(df_marital,
    marital = as.character(fct_collapse(marital, married = "married", aother = c("never married", "divorced", "widowed"))),
    highlight = "married"
  ),
  mutate(df_marital,
    marital = as.character(fct_collapse(marital, divorced = "divorced", aother = c("never married", "married", "widowed"))),
    highlight = "divorced"
  ),
  mutate(df_marital,
    marital = as.character(fct_collapse(marital, widowed = "widowed", aother = c("never married", "married", "divorced"))),
    highlight = "widowed"
  )
) %>%
  mutate(
    highlight = factor(highlight, levels = c("never married", "married", "divorced", "widowed"))
  )

ggplot(df_marital2, aes(age)) +
  annotate(geom = "rect", xmin = -Inf, xmax = Inf, ymin = -Inf, ymax = Inf, fill = "#b3b3b3a0", color = NA) +
  geom_density_line(
    aes(y = stat(count), fill = marital), color = "transparent", position = "fill"
  ) +
  facet_wrap(~highlight, nrow = 1) +
  scale_x_continuous(
    name = "age (years)",
    #limits = c(15, 98), 
    expand = c(0, 0)
  ) +
  scale_y_continuous(name = "relative proportion", expand = c(0, 0), labels = scales::percent) +
  scale_fill_manual(
    values = c("transparent", "#2b8cbed0", "#2b8cbed0", "#2b8cbed0", "#2b8cbed0"),
    name = NULL,
    breaks = c("aother", "divorced"),
    labels = c("all people surveyed  ", "highlighted group"),
    guide = guide_legend(
      direction = "horizontal",
      override.aes = list(fill = c("#bebebe", "#3590c0"))
    )
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_blank(),
    strip.text = element_text(size = 14, margin = margin(0, 0, 0.2, 0, "cm")),
    legend.position = "bottom",
    legend.justification = "right",
    legend.margin = margin(4.5, 0, 1.5, 0, "pt"),
    legend.spacing.x = grid::unit(4.5, "pt"),
    legend.spacing.y = grid::unit(0, "pt"),
    legend.box.spacing = grid::unit(0, "cm")
  )
```

<!--chapter:end:visualizing_proportions.Rmd-->

```{r echo = FALSE, message = FALSE}
# run setup script
source("_common.R")

library(lubridate)
library(tidyr)
```

# Visualizing trends {#visualizing-trends}

When making scatter plots (Chapter \@ref(visualizing-associations)) or time series (Chapter \@ref(time-series)), we are often more interested in the overarching trend of the data than in the specific detail of where each individual data point lies. By drawing the trend on top of or instead of the actual data points, usually in the form of a straight or curved line, we can create a visualization that helps the reader immediately see key features of the data. There are two fundamental approaches to determining a trend: We can either smooth the data by some method, such as a moving average, or we can fit a curve with a defined functional form and then draw the fitted curve. Once we have identified a trend in a dataset, it may also be useful to look specifically at deviations from the trend or to separate the data into multiple components, including the underlying trend, any existing cyclical components, and episodic components or random noise.

## Smoothing

Let us consider a time series of the Dow Jones Industrial Average (Dow Jones for short), a stock-market index representing the price of 30 large, publicly owned U.S. companies. Specifically, we will look at the year 2009, right after the 2008 crash (Figure \@ref(fig:dow-jones)). During the tail end of the crash, in the first three months of the year 2009,  the market lost over 2400 points (~27%). Then it slowly recovered for the remainder of the year. How can we visualize these longer-term trends while de-emphasizing the less important short-term fluctuations?

(ref:dow-jones) Daily closing values of the Dow Jones Industrial Average for the year 2009. Data source: Yahoo! Finance

```{r dow-jones, fig.width = 5*6/4.2, fig.asp = 0.5, fig.cap = '(ref:dow-jones)'}
dow_jones_industrial %>% filter(date >= ymd("2008/12/31") & date <= ymd("2010/01/10")) %>%
  ggplot(aes(date, close)) + 
  geom_line(color = "grey20", size = 0.75) + 
  scale_x_date(limits = c(ymd("2008-12-31"), ymd("2010-01-10")), expand = c(0, 0)) +
  xlab(NULL) + ylab("Dow Jones Industrial Average") +
  theme_dviz_grid() +
  theme(
    plot.margin = margin(3, 12, 3, 1.5)
  )
```

In statistical terms, we are looking for a way to *smooth* the stock-market time series. The act of smoothing produces a function that captures key patterns in the data while removing irrelevant minor detail or noise. Financial analysts usually smooth stock-market data by calculating moving averages. To generate a moving average, we take a time window, say the first 20 days in the time series, calculate the average price over these 20 days, then move the time window by one day, so it now spans the 2nd to 21st day, calculate the average over these 20 days, move the time window again, and so on. The result is a new time series consisting of a sequence of averaged prices.

To plot this sequence of moving averages, we need to decide which specific time point to associate with the average for each time window. Financial analysts often plot each average at the end of its respective time window. This choice results in curves that lag the original data (Figure \@ref(fig:dow-jones-moving-ave)a), with more severe lags corresponding to larger averaging time windows. Statisticians, on the other hand, plot the average at the center of the time window, which results in a curve that overlays perfectly on the original data (Figure \@ref(fig:dow-jones-moving-ave)b).

(ref:dow-jones-moving-ave) Daily closing values of the Dow Jones Industrial Average for the year 2009, shown together with their 20-day, 50-day, and 100-day moving averages. (a) The moving averages are plotted at the end of the moving time windows. (b) The moving averages are plotted in the center of the moving time windows. Data source: Yahoo! Finance

```{r dow-jones-moving-ave, fig.width = 5*6/4.2, fig.asp = 1, fig.cap = '(ref:dow-jones-moving-ave)'}
p1 <- dow_jones_industrial %>% filter(date >= ymd("2008/12/31") & date <= ymd("2010/01/10")) %>%
  mutate(
    close_20d_ave = moving_ave(date, close, 20, center = FALSE),
    close_50d_ave = moving_ave(date, close, 50, center = FALSE),
    close_100d_ave = moving_ave(date, close, 100, center = FALSE)
  ) %>%
  ggplot(aes(date, close)) + 
  geom_line(color = "grey20", size = .35) +
  geom_line(aes(date, close_20d_ave, color = "20d"), size = 1, na.rm = TRUE) +
  geom_line(aes(date, close_50d_ave, color = "50d"), size = 1, na.rm = TRUE) +
  geom_line(aes(date, close_100d_ave, color = "100d"), size = 1, na.rm = TRUE) +
  scale_color_manual(
    values = c(
      `20d` = "#009e73",
      `50d` = "#d55e00",
      `100d` = "#0072b2"
    ),
    breaks = c("20d", "50d", "100d"),
    labels = c("20-day average", "50-day average", "100-day average"),
    name = NULL
  ) + 
  scale_x_date(
    limits = c(ymd("2008-12-31"), ymd("2010-01-10")), expand = c(0, 0),
    labels = NULL
  ) +
  xlab(NULL) + ylab("Dow Jones Industrial Average") +
  theme_dviz_grid() +
  theme(
    plot.margin = margin(3, 12, 3, 1.5),
    legend.position = c(1, 0),
    legend.justification = c(1, 0),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.margin = margin(6, 12, 0, 12),
    axis.ticks.x = element_blank()
  )

p2 <- dow_jones_industrial %>% filter(date >= ymd("2008/12/31") & date <= ymd("2010/01/10")) %>%
  mutate(
    close_20d_ave = moving_ave(date, close, 20, center = TRUE),
    close_50d_ave = moving_ave(date, close, 50, center = TRUE),
    close_100d_ave = moving_ave(date, close, 100, center = TRUE)
  ) %>%
  ggplot(aes(date, close)) + 
  geom_line(color = "grey20", size = .35) +
  geom_line(aes(date, close_20d_ave, color = "20d"), size = 1, na.rm = TRUE) +
  geom_line(aes(date, close_50d_ave, color = "50d"), size = 1, na.rm = TRUE) +
  geom_line(aes(date, close_100d_ave, color = "100d"), size = 1, na.rm = TRUE) +
  scale_color_manual(
    values = c(
      `20d` = "#009e73",
      `50d` = "#d55e00",
      `100d` = "#0072b2"
    ),
    breaks = c("20d", "50d", "100d"),
    labels = c("20-day average", "50-day average", "100-day average"),
    name = NULL,
    guide = "none"
  ) + 
  scale_x_date(limits = c(ymd("2008-12-31"), ymd("2010-01-10")), expand = c(0, 0)) +
  xlab(NULL) + ylab("Dow Jones Industrial Average") +
  theme_dviz_grid() +
  theme(
    plot.margin = margin(3, 12, 3, 1.5)
  )

plot_grid(p1, p2, ncol = 1, align = 'h', labels = "auto")
```

Regardless of whether we plot the smoothed time series with or without lag, we can see that the length of the time window over which we average sets the scale of the fluctuations that remain visible in the smoothed curve. The 20-day moving average only removes small, short-term spikes but otherwise follows the daily data closely. The 100-day moving average, on the other hand, removes even fairly substantial drops or spikes that play out over a time span of multiple weeks. For example, the massive drop to below 7000 points in the first quarter of 2009 is not visible in the 100-day moving average, which replaces it with a gentle curve that doesn't dip much below 8000 points (Figure \@ref(fig:dow-jones-moving-ave)). Similarly, the drop around July 2009 is completely invisible in the 100-day moving average.

The moving average is the most simplistic approach to smoothing, and it has some obvious limitations. First, it results in a smoothed curve that is shorter than the original curve (Figure \@ref(fig:dow-jones-moving-ave)). Parts are missing at either the beginning or the end or both. And the more the time series is smoothed (i.e., the larger the averaging window), the shorter the smoothed curve. Second, even with a large averaging window, a moving average is not necessarily that smooth. It may exhibit small bumps and wiggles even though larger-scale smoothing has been achieved (Figure \@ref(fig:dow-jones-moving-ave)). These wiggles are caused by individual data points that enter or exit the averaging window. Since all data points in the window are weighted equally, individual data points at the window boundaries can have visible impact on the average.

Statisticians have developed numerous approaches to smoothing that alleviate the downsides of moving averages. These approaches are much more complex and computationally costly, but they are readily available in modern statistical computing environments. One widely used method is LOESS (locally estimated scatterplot smoothing, @Cleveland1979), which fits low-degree polynomials to subsets of the data. Importantly, the points in the center of each subset are weighted more heavily than points at the boundaries, and this weighting scheme yields a much smoother result than we get from a weighted average (Figure \@ref(fig:dow-jones-loess)). The LOESS curve shown here looks similar to the 100-day average, but this similarity should not be overinterpreted. The smoothness of a LOESS curve can be tuned by adjusting a parameter, and different parameter choices would have produced LOESS curves looking more like the 20-day or 50-day average. 

(ref:dow-jones-loess) Comparison of LOESS fit to 100-day moving average for the Dow Jones data of Figure \@ref(fig:dow-jones-moving-ave). The overall trend shown by the LOESS smooth is nearly identical to the 100-day moving average, but the LOESS curve is much smoother and it extends to the entire range of the data. Data source: Yahoo! Finance

```{r dow-jones-loess, fig.width = 5*6/4.2, fig.asp = 0.5, fig.cap = '(ref:dow-jones-loess)'}
# LOESS (locally estimated scatterplot smoothing) 

dow_jones_industrial %>% filter(date >= ymd("2008/12/31") & date <= ymd("2010/01/10")) %>%
  mutate(
    close_100d_ave = moving_ave(date, close, 100)
  ) %>%
  ggplot(aes(date, close)) + 
  geom_line(color = "grey20", size = .35) +
  geom_line(aes(date, close_100d_ave, color = "100d"), size = 1, na.rm = TRUE) +
  geom_smooth(aes(color = "smooth"), size = 1, na.rm = TRUE, se = FALSE) +
  scale_color_manual(
    values = c(
      `100d` = "#d55e00",
      smooth = "#0072b2"
    ),
    breaks = c("smooth", "100d"),
    labels = c("LOESS smoother", "100-day average"),
    name = NULL
  ) + 
  scale_x_date(limits = c(ymd("2008-12-31"), ymd("2010-01-10")), expand = c(0, 0)) +
  xlab(NULL) + ylab("Dow Jones Industrial Average") +
  theme_dviz_grid() +
  theme(
    legend.position = c(1, 0.48),
    legend.justification = c(1, 0.5),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.margin = margin(0, 12, 6, 12),
    plot.margin = margin(3, 12, 3, 1.5)
  )
```

Importantly, LOESS is not limited to time series. It can be applied to arbitrary scatter plots, as is apparent from its name, *locally estimated scatterplot smoothing*. For example, we can use LOESS to look for trends in the relationship between a car's fuel-tank capacity and its price (Figure \@ref(fig:tank-capacity-loess)). The LOESS line shows that tank capacity grows approximately linearly with price for cheap cars (below \$20,000) but it levels off for more expensive cars. Above a price of approximately \$20,000, buying a more expensive car will not get you one with a larger fuel tank.

(ref:tank-capacity-loess) Fuel-tank capacity versus price of 93 cars released for the 1993 model year. Each dot corresponds to one car. The solid line represents a LOESS smooth of the data. We see that fuel-tank capacity increases approximately linearly with price, up to a price of approximately $20,000, and then it levels off. Data source: Robin H. Lock, St. Lawrence University

```{r tank-capacity-loess, fig.width = 5, fig.asp = 0.75, fig.cap='(ref:tank-capacity-loess)'}
cars93 <- MASS::Cars93

ggplot(cars93, aes(x = Price, y = Fuel.tank.capacity)) + 
  geom_point(color = "grey60") + 
  geom_smooth(se = FALSE, method = "loess", formula = y ~ x, color = "#0072B2") +
  scale_x_continuous(
    name = "price (USD)",
    breaks = c(20, 40, 60),
    labels = c("$20,000", "$40,000", "$60,000")
  ) +
  scale_y_continuous(name = "fuel-tank capacity\n(US gallons)") +
  theme_dviz_grid()
```

LOESS is a very popular smoothing approach because it tends to produce results that look right to the human eye. However, it requires the fitting of many separate regression models. This makes it slow for large datasets, even on modern computing equipment.

As a faster alternative to LOESS, we can use spline models. A spline is a piecewise polynomial function that is highly flexible yet always looks smooth. When working with splines, we will encounter the term *knot.* The knots in a spline are the endpoints of the individual spline segments. If we fit a spline with *k* segments, we need to specify *k* + 1 knots. While spline fitting is computationally efficient, in particular if the number of knots is not too large, splines have their own downsides. Most importantly, there is a bewildering array of different types of splines, including cubic splines, B-splines, thin-plate splines, Gaussian process splines, and many others, and which one to pick may not be obvious. The specific choice of the type of spline and number of knots used can result in widely different smoothing functions for the same data (Figure \@ref(fig:tank-capacity-smoothers)).

(ref:tank-capacity-smoothers) Different smoothing models display widely different behaviors, in particular near the boundaries of the data. (a) LOESS smoother, as in Figure \@ref(fig:tank-capacity-loess). (b) Cubic regression splines with 5 knots. (c) Thin-plate regression spline with 3 knots. (d)  Gaussian process spline with 6 knots. Data source: Robin H. Lock, St. Lawrence University

```{r tank-capacity-smoothers, fig.width = 5.5*6/4.2, fig.asp = 0.75, fig.cap='(ref:tank-capacity-smoothers)'}
cars_base <- ggplot(cars93, aes(x = Price, y = Fuel.tank.capacity)) + geom_point(color = "grey60") + 
  scale_x_continuous(
    name = "price (USD)",
    breaks = c(20, 40, 60),
    labels = c("$20,000", "$40,000", "$60,000")
  ) +
  scale_y_continuous(name = "fuel-tank capacity\n(US gallons)") +
  theme_dviz_grid(12)  
  
p1 <- cars_base + geom_smooth(se = FALSE, method = "loess", formula = y ~ x, color = "#0072B2")
p2 <- cars_base + geom_smooth(se = FALSE, method = "gam", formula = y ~ s(x, k = 5, bs = 'cr'), color = "#0072B2")
p3 <- cars_base + geom_smooth(se = FALSE, method = "gam", formula = y ~ s(x, k = 3), color = "#0072B2")
p4 <- cars_base + geom_smooth(se = FALSE, method = "gam", formula = y ~ s(x, k = 6, bs = 'gp'), color = "#0072B2")

plot_grid(
  p1, NULL, p2,
  NULL, NULL, NULL,
  p3, NULL, p4,
  align = 'hv',
  labels = c("a", "", "b", "", "", "", "c", "", "d"),
  rel_widths = c(1, .02, 1),
  rel_heights = c(1, .02, 1)
)
```

Most data visualization software will provide smoothing features, likely implemented as either a type of local regression (such as LOESS) or a type of spline. The smoothing method may be referred to as a GAM, a generalized additive model, which is a superset of all these types of smoothers. It is important to be aware that the output of the smoothing feature is highly dependent on the specific GAM model that is fit. Unless you try out a number of different choices you may never realize to what extent the results you see depend on the specific default choices made by your statistical software.

```{block type='rmdtip', echo=TRUE}
Be careful when interpreting the results from a smoothing function. The same dataset can be smoothed in many different ways.
```


## Showing trends with a defined functional form

As we can see in Figure \@ref(fig:tank-capacity-smoothers), the behavior of general-purpose smoothers can be somewhat unpredictable for any given dataset. These smoothers also do not provide parameter estimates that have a meaningful interpretation. Therefore, whenever possible, it is preferable to fit a curve with a specific functional form that is appropriate for the data and that uses parameters with clear meaning. 

For the fuel-tank data, we need a curve that initially rises linearly but then levels off at a constant value. The function $y = A - B \exp(-mx)$ may fit that bill. Here, $A$, $B$, and $m$ are the constants we adjust to fit the curve to the data. The function is approximately linear for small $x$, with $y \approx A - B + B m x$, it approaches a constant value for large $x$, $y \approx A$, and it is strictly increasing for all values of $x$. Figure \@ref(fig:tank-capacity-model) shows that this equation fits the data at least as well as any of the smoothers we considered previously (Figure \@ref(fig:tank-capacity-smoothers)).

(ref:tank-capacity-model) Fuel-tank data represented with an explicit analytical model. The solid line corresponds to a least-squares fit of the formula $y = A - B \exp(-mx)$ to the data. Fitted parameters are $A = 19.6$, $B = 29.2$, $m = 0.00015$.  Data source: Robin H. Lock, St. Lawrence University

```{r tank-capacity-model, fig.width = 5, fig.asp = 0.75, fig.cap = '(ref:tank-capacity-model)'}
# first model
fit.out <- nls(
  Fuel.tank.capacity ~ a*Price/(Price + b) + c,
  data = cars93,
  start = c(a = -45, b = -1, c = 70)
)

# second model
fit.out <- nls(
  Fuel.tank.capacity ~ A1 - A0*exp(-m*Price),
  data = cars93,
  start = c(A0 = 29.249, A1 = 19.621, m = 0.149),
  control = nls.control(maxiter = 1000, warnOnly = TRUE)
)

fit.df <- data.frame(
  Price = 7:62,
  Fuel.tank.capacity = predict(fit.out, data.frame(Price = 7:62))
)

ggplot(cars93, aes(x = Price, y = Fuel.tank.capacity)) + 
  geom_point(color = "grey60") + 
  geom_line(data = fit.df, size = 1, color = "#0072B2") +
  #stat_function(fun = function(x) -24.22+1.38*x  + 22) +
  scale_x_continuous(
    name = "price (USD)",
    breaks = c(20, 40, 60),
    labels = c("$20,000", "$40,000", "$60,000")
  ) +
  scale_y_continuous(name = "fuel-tank capacity\n(US gallons)") +
  theme_dviz_grid()
```

A functional form that is applicable in many different contexts is the simple straight line, $y = A + mx$. Approximately linear relationships between two variables are surprisingly common in real-world datasets. For example, in Chapter \@ref(visualizing-associations), I discussed the relationship between head length and body mass in blue jays. This relationship is approximately linear, for both female and male birds, and drawing linear trend lines on top of the points in a scatter plot helps the reader perceive the trends (Figure \@ref(fig:blue-jays-scatter-line)).

(ref:blue-jays-scatter-line) Head length versus body mass for 123 blue jays. The birds' sex is indicated by color. This figure is equivalent to Figure \@ref(fig:blue-jays-scatter-sex), except that now we have drawn linear trend lines on top of the individual data points. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-scatter-line, fig.width = 5, fig.asp = 3.2/4, fig.cap='(ref:blue-jays-scatter-line)'}
ggplot(blue_jays, aes(Mass, Head, color = KnownSex, fill = KnownSex)) + 
  geom_point(pch = 21, color = "white", size = 2.5) +
  geom_smooth(method = "lm", size = 0.75, se = FALSE, fullrange = TRUE) +
  scale_x_continuous(name = "body mass (g)") +
  scale_y_continuous(name = "head length (mm)") +
  scale_fill_manual(
    values = c(F = "#D55E00C0", M = "#0072B2C0"),
    breaks = c("F", "M"),
    labels = c("female birds   ", "male birds"),
    name = NULL,
    guide = guide_legend(
      direction = "horizontal",
      override.aes = list(size = 3, linetype = 0)
    )
  ) +
  scale_color_manual(
    values = c(F = "#D55E00", M = "#0072B2"),
    breaks = c("F", "M"),
    labels = c("female birds   ", "male birds"),
    name = NULL
  ) +
  theme_dviz_grid() +
  theme(
    #legend.position = c(1, 0.01),
    #legend.justification = c(1, 0),
    legend.position = "top",
    legend.justification = "right",
    legend.box.spacing = unit(3.5, "pt"), # distance between legend and plot
    legend.text = element_text(vjust = 0.6),
    legend.spacing.x = unit(2, "pt"),
    legend.background = element_rect(fill = "white", color = NA),
    legend.key.width = unit(10, "pt")
  )
```

When the data display a non-linear relationship, we need to guess what an appropriate functional form might be. In this case, we can assess the accuracy of our guess by transforming the axes in such a way that a linear relationship emerges. To demonstrate this principle, let's return to the monthly submissions to the preprint server bioRxiv, discussed in Chapter \@ref(visualizing-associations). If the increase in submissions in each month is proportional to the number of submissions in the previous month, i.e., if submissions grow by a fixed percentage each month, then the resulting curve is exponential. This assumption seems to be met for the bioRxiv data, because a curve with exponential form, $y = A\exp(mx)$, fits the bioRxiv submission data well (Figure \@ref(fig:biorxiv-expfit)).

(ref:biorxiv-expfit) Monthly submissions to the preprint server bioRxiv. The solid blue line represents the actual monthly preprint counts and the dashed black line represents an exponential fit to the data, $y = 60\exp[0.77(x - 2014)]$. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r biorxiv-expfit, fig.cap = '(ref:biorxiv-expfit)'}
preprint_growth %>% filter(archive == "bioRxiv") %>%
  filter(count > 0) %>%
  mutate(date_dec = decimal_date(date)) -> biorxiv_growth

expfit.out <- nls(
  count ~ a*exp(b*(date_dec-2014)),
  data = biorxiv_growth,
  start = c(a = 60.004, b = .773)
)

linfit.out <- nls(
  log(count) ~ log(a) + b*(date_dec-2014),
  data = biorxiv_growth,
  start = c(a = 42.576, b = .878)
)

date_seq = seq(min(biorxiv_growth$date_dec), max(biorxiv_growth$date_dec), by = 0.1)
expfit.df <- data.frame(
  date_dec = date_seq,
  count = predict(expfit.out, data.frame(date_dec = date_seq))
)

linfit.df <- data.frame(
  date_dec = date_seq,
  count = exp(predict(linfit.out, data.frame(date_dec = date_seq)))
)

ggplot(biorxiv_growth, aes(date_dec, count)) + 
  geom_line(data = expfit.df, aes(color = "expfit"), size = .5, linetype = 2) +
  geom_point(aes(fill = "expfit"), shape = NA, na.rm = TRUE) + # dummy for legend
  geom_line(aes(color = "data"), size = .5) +
  geom_point(aes(fill = "data"), color = "white", shape = 21, size = 2) +
  scale_y_continuous(
     limits = c(0, 1550),
     breaks = c(0, 500, 1000, 1500),
     expand = c(0, 0),
     name = "preprints / month"
  ) + 
  scale_x_continuous(name = NULL) +
  scale_color_manual(
    name = NULL,
    values = c(data = "#0072B2", expfit = "black"),
    breaks = c("data", "expfit"),
    labels = c("actual counts", "exponential fit"),
    guide = guide_legend(
      override.aes = list(
        color = c("white", "black"),
        shape = c(21, NA),
        size = c(2, 0.5),
        linetype = c(0, 2)
      )
    )
  ) +
  scale_fill_manual(
    name = NULL,
    values = c(data = "#0072B2", expfit = "black"),
    breaks = c("data", "expfit"),
    labels = c("actual counts", "exponential fit")
  ) +
  theme_dviz_open() +
  theme(
    legend.position = c(.05, 1),
    legend.justification = c(0, 1),
    legend.spacing.x = unit(3, "pt"),
    legend.title = element_blank(), 
    plot.margin = margin(7, 7, 3, 1.5)
  )
```

If the original curve is exponential, $y = A\exp(mx)$, then a log-transformation of the *y* values will turn it into a linear relationship, $\log(y) = \log(A) + mx$. Therefore, plotting the data with log-transformed *y* values (or equivalently, with a logarithmic *y* axis) and looking for a linear relationship is a good way of determining whether a dataset exhibits exponential growth. For the bioRxiv submission numbers, we indeed obtain a linear relationship when using a logarithmic *y* axis (Figure \@ref(fig:biorxiv-logscale)).

(ref:biorxiv-logscale) Monthly submissions to the preprint server bioRxiv, shown on a log scale. The solid blue line represents the actual monthly preprint counts, the dashed black line represents the exponential fit from Figure \@ref(fig:biorxiv-expfit), and the solid black line represents a linear fit to log-transformed data, corresponding to $y = 43\exp[0.88(x - 2014)]$. Data source: Jordan Anaya, http://www.prepubmed.org/

```{r biorxiv-logscale, fig.cap = '(ref:biorxiv-logscale)'}
ggplot(biorxiv_growth, aes(date_dec, count)) +
  geom_line(data = expfit.df, aes(color = "expfit"), size = .5, linetype = 2) +
  geom_point(aes(fill = "expfit"), shape = NA, na.rm = TRUE) + # dummy for legend
  geom_line(data = linfit.df, aes(color = "linfit"), size = .5) +
  geom_point(aes(fill = "linfit"), shape = NA, na.rm = TRUE) + # dummy for legend
  geom_line(aes(color = "data"), size = .5) +
  geom_point(aes(fill = "data"), color = "white", shape = 21, size = 2) +
  scale_y_log10(
    limits = c(30, 1670),
    breaks = c(10*(3:9), 100*(1:9), 1000*(1:2)),
    labels = c(
      "", "", "50", "", "", "", "", "100", "", "", "", "500",
      "", "", "", "", "1000", ""
    ),
    expand = c(0, 0),
    name = "preprints / month"
  ) + 
  scale_x_continuous(name = NULL) +
  scale_color_manual(
    name = NULL,
    values = c(data = "#0072B2", expfit = "black", linfit = "black"),
    breaks = c("data", "expfit", "linfit"),
    labels = c("actual counts", "exponential fit", "linear fit, log-transformed data"),
    guide = guide_legend(
      override.aes = list(
        color = c("white", "black", "black"),
        shape = c(21, NA, NA),
        size = c(2, 0.5, 0.5),
        linetype = c(0, 2, 1)
      )
    )
  ) +
  scale_fill_manual(
    name = NULL,
    values = c(data = "#0072B2", expfit = "black", linfit = "black"),
    breaks = c("data", "expfit", "linfit"),
    labels = c("actual counts", "exponential fit", "linear fit, log-transformed data")
  ) +
  theme_dviz_open() +
  theme(
    legend.position = c(.05, 1),
    legend.justification = c(0, 1),
    legend.spacing.x = unit(3, "pt"),
    legend.title = element_blank(), 
    plot.margin = margin(7, 7, 3, 1.5)
  )

```

In Figure \@ref(fig:biorxiv-logscale), in addition to the actual submission counts, I am also showing the exponential fit from Figure \@ref(fig:biorxiv-expfit) and a linear fit to the log-transformed data. These two fits are similar but not identical. In particular, the slope of the dashed line seems somewhat off. The line systematically falls above the individual data points for half the time series. This is a common problem with exponential fits: The square deviations from the data points to the fitted curve are so much larger for the largest data values than for the smallest data values that the deviations of the smallest data values contribute little to the overall sum squares that the fit minimizes. As a result, the fitted line systematically overshoots or undershoots the smallest data values. For this reason, I generally advise to avoid exponential fits and instead use linear fits on log-transformed data.

```{block type='rmdtip', echo=TRUE}
It is usually better to fit a straight line to transformed data than to fit a nonlinear curve to untransformed data.
```

A plot such as Figure \@ref(fig:biorxiv-logscale) is commonly referred to as *log--linear*, since the *y* axis is logarithmic and the *x* axis is linear. Other plots we may encounter include *log--log*, where both the *y* and the *x* axis are logarithmic, or *linear--log*, where *y* is linear and *x* is logarithmic. In a log--log plot, power laws of the form $y \sim x^\alpha$ appear as straight lines (see Figure \@ref(fig:word-counts-tail-log-log) for an example), and in a linear--log plot, logarithmic relationships of the form $y \sim \log(x)$ appear as a straight lines. Other functional forms can be turned into linear relationships with more specialized coordinate transformations, but these three (log--linear, log--log, linear--log) cover a wide range of real-world applications.


## Detrending and time-series decomposition

For any time series with a prominent long-term trend, it may be useful to remove this trend to specifically highlight any notable deviations. This technique is called *detrending*, and I will demonstrate it here with house prices. In the U.S., the mortgage lender Freddie Mac publishes a monthly index, called the *Freddie Mac House Price Index,* that tracks the change in house prices over time. The index attempts to capture the state of the entire house market in a given region, such that an increase in the index by, for example, 10% can be interpreted as an average house price increase of 10% in the respective market. The index is arbitrarily set to a value of 100 in December 2000.

Over long periods of time, house prices tend to display consistent annual growth, approximately in line with inflation. However, overlaid on top of this trend are housing bubbles that lead to severe boom and bust cycles. Figure \@ref(fig:hpi-trends) shows the actual house price index and its long-term trend for four select U.S. states. We see that between 1980 and 2017, California underwent two bubbles, one in 1990 and one in the mid-2000s. During the same period, Nevada experienced only one bubble, in the mid-2000s, and house prices in Texas and West Virginia closely followed their long-term trends the entire time. Because house prices tend to grow in percent increments, i.e., exponentially, I have chosen a logarithmic *y* axis in Figure \@ref(fig:hpi-trends). The straight lines correspond to a 4.7% annual price increase in California and a 2.8% annual price increase each in Nevada, Texas, and West Virginia. 

(ref:hpi-trends) Freddie Mac House Price Index from 1980 through 2017, for four selected states (California, Nevada, Texas, and West Virginia). The House Price Index is a unitless number that tracks relative house prices in the chosen geographic region over time. The index is scaled arbitrarily such that it equals 100 in December of the year 2000. The blue lines show the monthly fluctuations in the index and the straight gray lines show the long-term price trends in the respective states. Note that the *y* axes are logarithmic, so that the straight gray lines represent consistent exponential growth. Data source: Freddie Mac House Prices Index

```{r hpi-trends, fig.width = 5*6/4.2, fig.cap = '(ref:hpi-trends)'}
hpi_trends <- house_prices %>%
  filter(year(date) >= 1980) %>%
  filter(state %in% c("California", "Nevada", "West Virginia", "Texas")) %>%
  mutate(
    date_numeric = as.numeric(date),
    hpi = house_price_index,
    log_hpi = log(hpi)
  ) %>%
  group_by(state) %>%
  mutate(
    hpi_trend = {
      coefs <- coef(lm(log_hpi ~ date_numeric))
      exp(coefs[1] + coefs[2]*date_numeric)
    },
    hpi_detrended = hpi/hpi_trend
  )

ggplot(hpi_trends, aes(date, hpi)) +
  geom_line(aes(y = hpi_trend), color = "grey50", size = 0.4) +
  geom_line(color = "#0072B2", size = 0.75) +
  scale_x_date(name = NULL) +
  scale_y_log10(name = "House Price Index (Dec. 2000 = 100)") +
  facet_wrap(~state, scales = "free_x") +
  theme_dviz_hgrid() +
  theme(
    strip.text = element_text(size = 12),
    strip.background = element_rect(fill = "grey85"),
    axis.line.x = element_line(color = "grey50"),
    axis.ticks.x = element_line(color = "grey50"),
    axis.ticks.y = element_blank(),
    axis.text.y = element_text(margin = margin(0, 0, 0, 0))
  )
    
```

We detrend housing prices by dividing the actual price index at each time point by the respective value in the long-term trend. Visually, this division will look like we are subtracting the gray lines from the blue lines in Figure \@ref(fig:hpi-trends), because a division of the untransformed values is equivalent to a subtraction of the log-transformed values. The resulting detrended house prices show the housing bubbles more clearly (Figure \@ref(fig:hpi-detrended)), as the detrending emphasizes the unexpected movements in a time series. For example, in the original time series, the decline in home prices in California from 1990 to about 1998 looks modest (Figure \@ref(fig:hpi-trends)). However, during that same time period, on the basis of the long-term trend we would have expected prices to rise. Relative to the expected rise the drop in prices was substantial, amounting to 25% at the lowest point (Figure \@ref(fig:hpi-detrended)).

(ref:hpi-detrended) Detrended version of the Freddie Mac House Price Index shown in Figure \@ref(fig:hpi-trends). The detrended index was calculated by dividing the actual index (blue lines in Figure \@ref(fig:hpi-trends)) by the expected value based on the long-term trend (straight gray lines in Figure \@ref(fig:hpi-trends)). This visualization shows that California experienced two housing bubbles, around 1990 and in the mid-2000s, identifiable from a rapid rise and subsequent decline in the actual housing prices relative to what would have been expected from the long-term trend. Similarly, Nevada experienced one housing bubble, in the mid-2000s, and neither Texas nor West Virginia experienced much of a bubble at all. Data source: Freddie Mac House Prices Index

```{r hpi-detrended, fig.width = 5*6/4.2, fig.cap = '(ref:hpi-detrended)'}
ggplot(hpi_trends, aes(date, hpi_detrended)) +
  geom_hline(yintercept = 1, color = "grey50", size = 0.4) +
  geom_line(color = "#0072B2", size = 0.75) +
  scale_x_date(name = NULL) +
  scale_y_log10(
    name = "House Price Index (detrended)",
    breaks = c(0.752, 1, 1.33, 1.77),
    labels = c("0.75", "1.00", "1.33", "1.77")
  ) +
  facet_wrap(~state, scales = "free_x") +
  theme_dviz_hgrid() +
  theme(
    strip.text = element_text(size = 12),
    strip.background = element_rect(fill = "grey85"),
    axis.line.x = element_line(color = "grey50"),
    axis.ticks.x = element_line(color = "grey50"),
    axis.ticks.y = element_blank(),
    axis.text.y = element_text(margin = margin(0, 0, 0, 0))
  )
 
 
```

Beyond simple detrending, we can also separate a time series into multiple distinct components, such that their sum recovers the original time series. In general, in addition to a long-term trend, there are three distinct components that may shape a time series. First, there is random noise, which causes small, erratic movements up and down. This noise is visible in all the time series shown in this chapter, but maybe the most clearly in Figure \@ref(fig:biorxiv-logscale). Second, there can be unique external events that leave their mark in the time series, such as the distinct housing bubbles seen in Figure \@ref(fig:hpi-trends). Third, there can be cyclical variations. For example, outside temperatures show daily cyclical variations. The highest temperatures are reached in the early afternoon and the lowest temperatures in the early morning. Outside temperatures also show yearly cyclical variations. They tend to rise in the spring, reach their maximum in the summer, and then decline in fall and reach their minimum in the winter (Figure \@ref(fig:temperature-normals-Houston)).

To demonstrate the concept of distinct time-series components, I will here decompose the Keeling curve, which shows changes in CO<sub>2</sub> abundance over time (Figure \@ref(fig:keeling-curve)). CO<sub>2</sub> is measured in parts per million (ppm). We see a long-term increase in CO<sub>2</sub> abundance that is slightly faster than linear, from below 325 ppm in the 1960s to above 400 in the second decade of the 21st century (Figure \@ref(fig:keeling-curve)). CO<sub>2</sub> abundance also fluctuates annually, following a consistent up-and-down pattern overlaid on top of the overall increase. The annual fluctuation are driven by plant growth in the northern hemisphere. Plants consume CO<sub>2</sub> during photosynthesis. Because most of the globe's land masses are located in the northern hemisphere, and plant growth is most active in the spring and summer, we see an annual global decline in atmospheric CO<sub>2</sub> that coincides with the summer months in the northern hemisphere.

(ref:keeling-curve) The Keeling curve. The Keeling curve shows the change of CO<sub>2</sub> abundance in the atmosphere over time. Since 1958, CO<sub>2</sub> abundance has been continuously monitored at the Mauna Loa Observatory in Hawaii, initially under the direction of Charles Keeling. Shown here are monthly average CO<sub>2</sub> readings, expressed in parts per million (ppm). The CO<sub>2</sub> readings fluctuate annually with the seasons but show a consistent long-term trend of increase. Data source: Dr. Pieter Tans, NOAA/ESRL, and Dr. Ralph Keeling, Scripps Institution of Oceanography

```{r keeling-curve, fig.cap = '(ref:keeling-curve)'}
# use complete years only
ggplot(CO2, aes(date_dec, co2_interp)) +
  geom_line(color = "#0072B2", size = 0.6) +
  scale_y_continuous(
    limits = c(295, 418),
    breaks = c(300, 325, 350, 375, 400),
    name = parse(text = "`CO`[2]*` concentration (ppm)`"),
    expand = c(0, 0)
  ) +
  scale_x_continuous(
    limits = c(1958, 2019),
    name = NULL,
    breaks = c(1960, 1970, 1980, 1990, 2000, 2010),
    labels = c("", "1970", "", "1990", "", "2010"),
    expand = c(0, 0)
  ) +
  theme_dviz_grid()
```

We can decompose the Keeling curve into its long-term trend, seasonal fluctuations, and remainder (Figure \@ref(fig:keeling-curve-decomposition)). The specific method I am using here is called STL (Seasonal decomposition of Time series by LOESS, @Cleveland_et_al_1990), but there are many other methods that achieve similar goals. The decomposition shows that over the last three decades, CO<sub>2</sub> abundance has increased by over 50 ppm. By comparison, seasonal fluctuations amount to less than 8 ppm (they never cause an increase or a decrease in more than 4 ppm relative to the long-term trend), and the remainder amounts to less than 1.6 ppm (Figure \@ref(fig:keeling-curve-decomposition)). The remainder is the difference between the actual readings and the sum of the long-term trend and the seasonal fluctuations, and here it corresponds to random noise in the monthly CO<sub>2</sub> readings. More generally, however, the remainder could also capture unique external events. For example, if a massive volcano eruption released substantial amounts of CO<sub>2</sub>, such an event might be visible as a sudden spike in the remainder. Figure \@ref(fig:keeling-curve-decomposition) shows that no such unique external events have had a major effect on the Keeling curve in recent decades. 

(ref:keeling-curve-decomposition) Time-series decomposition of the Keeling curve, showing the monthly average (as in Figure \@ref(fig:keeling-curve)), the long-term trend, seasonal fluctuations, and the remainder. The remainder is the difference between the actual readings and the sum of the long-term trend and the seasonal fluctuations, and it represents random noise. I have zoomed into the most recent 30 years of data to more clearly show the shape of the annual fluctuations. Data source: Dr. Pieter Tans, NOAA/ESRL, and Dr. Ralph Keeling, Scripps Institution of Oceanography

```{r keeling-curve-decomposition, fig.width = 5*6/4.2, fig.asp = 0.9, fig.cap = '(ref:keeling-curve-decomposition)'}
# use complete years only
CO2_complete <- filter(CO2, year >= 1959, year < 2018)
# convert to time series object
CO2_ts <- ts(data = CO2_complete$co2_interp, start = 1959, end = c(2017, 12), frequency = 12)
# detrend via STL method
CO2_stl <- stl(CO2_ts, s.window = 7)
CO2_detrended <- mutate(
  CO2_complete,
  `monthly average` = co2_interp,
  `seasonal fluctuations` = t(CO2_stl$time.series)[1, ],
  `long-term trend` = t(CO2_stl$time.series)[2, ],
  remainder = t(CO2_stl$time.series)[3, ]
)

facet_labels <- c("monthly average", "long-term trend", "seasonal fluctuations", "remainder")

CO2_detrended %>%
  select(date_dec, `monthly average`, `seasonal fluctuations`, `long-term trend`, remainder) %>%
  gather(variable, value, -date_dec) %>%
  mutate(variable = factor(variable, levels = facet_labels)) %>%
  filter(date_dec >= 1989) %>%
  ggplot(aes(date_dec, value)) +
  geom_line(color = "#0072B2", size = 0.6) +
  geom_point(
    data = data.frame(
      variable = factor(
        rep(facet_labels, each = 2),
        levels = facet_labels
      ),
      x = 1990,
      #y = c(295, 424, 295, 424, -4.1, 4.3, -.81, .85)
      y = c(324, 419, 324, 419, -4.1, 4.3, -.81, .85)
    ),
    aes(x, y),
    color = NA,
    na.rm = TRUE
  ) +
  scale_y_continuous(
    name = parse(text = "`CO`[2]*` concentration (ppm)`"),
    expand = c(0, 0)
  ) +
  scale_x_continuous(
    limits = c(1989, 2018.2),
    name = NULL,
    breaks = c(1990, 1995, 2000, 2005, 2010, 2015),
    labels = c("1990", "", "2000", "", "2010", ""),
    expand = c(0, 0)
  ) +
  facet_wrap(facets = vars(variable), ncol = 1, scales = "free") +
  theme_dviz_grid() +
  theme(
    plot.margin = margin(3, 1.5, 3, 1.5),
    strip.text = element_text(size = 12)
  )
```

<!--chapter:end:visualizing_trends.Rmd-->

```{r echo = FALSE, message = FALSE, warning = FALSE}
# run setup script
source("_common.R")

library(forcats)
library(lubridate)
library(mgcv)
library(tidyr)
library(purrr)
library(broom)
library(emmeans)
library(ungeviz)
library(ggridges)
library(tidybayes)

animate <- TRUE # should animated figures be included or not
#animate <- FALSE
```

# Visualizing uncertainty {#visualizing-uncertainty}

One of the most challenging aspects of data visualization is the visualization of uncertainty. When we see a data point drawn in a specific location, we tend to interpret it as a precise representation of the true data value. It is difficult to conceive that a data point could actually lie somewhere it hasn't been drawn. Yet this scenario is ubiquitous in data visualization. Nearly every data set we work with has some uncertainty, and whether and how we choose to represent this uncertainty can make a major difference in how accurately our audience perceives the meaning of the data. 

Two commonly used approaches to indicate uncertainty are error bars and confidence bands. These approaches were developed in the context of scientific publications, and they require some amount of expert knowledge to be interpreted correctly. Yet they are precise and space efficient. By using error bars, for example, we can show the uncertainties of many different parameter estimates in a single graph. For a lay audience, however, visualization strategies that create a strong intuitive impression of the uncertainty will be preferable, even if they come at the cost of either reduced visualization accuracy or less data-dense displays. Options here include frequency framing, where we explicitly draw different possible scenarios in approximate proportions, or animations that cycle through different possible scenarios.

## Framing probabilities as frequencies {#frequency-framing}

Before we can discuss how to visualize uncertainty, we need to define what it actually is. We can intuitively grasp the concept of uncertainty most easily in the context of future events. If I am going to flip a coin I don't know ahead of time what the outcome will be. The eventual outcome is uncertain. I can also be uncertain about events in the past, however. If yesterday I looked out of my kitchen window exactly twice, once at 8am and once at 4pm, and I saw a red car parked across the street at 8am but not at 4pm, then I can conclude the car left at some point during the eight-hour window but I don't know exactly when. It could have been 8:01am, 9:30am, 2pm, or any other time during those eight hours.

Mathematically, we deal with uncertainty by employing the concept of probability. A precise definition of probability is complicated and far beyond the scope of this book. Yet we can successfully reason about probabilities without understanding all the mathematical intricacies. For many problems of practical relevance it is sufficient to think about relative frequencies. Assume you perform some sort of random trial, such as a coin flip or rolling a die, and look for a particular outcome (e.g., heads or rolling a six). You can call this outcome *success,* and any other outcome *failure.* Then, the probability of success is approximately given by the fraction of times you'd see that outcome if you repeated the random trial over and over again. For instance, if a particular outcome occurs with a probability of 10%, then we expect that among many repeated trials that outcome will be seen in approximately one out of ten cases.

Visualizing a single probability is difficult. How would you visualize the chance of winning in the lottery, or the chance of rolling a six with a fair die? In both cases, the probability is a single number. We could treat that number as an amount and display it using any of the techniques discussed in Chapter \@ref(visualizing-amounts), such as a bar graph or a dot plot, but the result would not be very useful. Most people lack an intuitive understanding of how a probability value translates into experienced reality. Showing the probability value as a bar or as a dot placed on a line does not help with this problem.

We can make the concept of probability tangible by creating a graph that emphasizes both the frequency aspect and the unpredictability of a random trial, for example by drawing squares of different colors in a random arrangement. In Figure \@ref(fig:probability-waffle), I use this technique to visualize three different probabilities, a 1% chance of success, a 10% chance of success, and a 40% chance of success. To read this figure, imagine you are given the task of picking a dark square by choosing a square before you can see which of the squares will be dark and which ones will be light. (If you will, you can think of picking a square with your eyes closed.) Intuitively, you will probably understand that it would be unlikely to select the one dark square in the 1%-chance case. Similarly, it would still be fairly unlikely to select a dark square in the 10%-chance case. However, in the 40%-chance case the odds don't look so bad. This style of visualization, where we show specific potential outcomes, is called a *discrete outcome visualization,* and the act of visualizing a probability as a frequency is called *frequency framing.* We are framing the probabilistic nature of a result in terms of easily understood frequencies of outcomes.

(ref:probability-waffle) Visualizing probability as frequency. There are 100 squares in each grid, and each square represents either success of failure in some random trial. A 1% chance of success corresponds to one dark and 99 light squares, a 10% chance of success corresponds to ten dark and 90 light squares, and a 40% chance of success corresponds to 40 dark and 60 light squares. By randomly placing the dark squares among the light squares, we can create a visual impression of randomness that emphasizes the uncertainty of the outcome of a single trial.

```{r probability-waffle, fig.width = 5*6/4.2, fig.asp = 1.2/3, fig.cap = '(ref:probability-waffle)'}
g <- expand.grid(x = 1:10, y = 1:10)

set.seed(84524)

data <- data.frame(ratio = c(0.01, 0.1, 0.4)) %>%
  mutate(
    out = purrr::map(
      ratio,
      ~g %>% mutate(
        value = {
          n <- n()
          i <- round(n*.x)
          sample(c(rep("S", i), rep("F", n - i)), n)
        }
      )
    )
  ) %>%
  unnest() %>%
  mutate(
    label = paste0(round(100*ratio), "% chance")
  )

ggplot(data, aes(x, y, fill = value)) +
  geom_tile(color = "white", size = 1) +
  coord_fixed(expand = FALSE, clip = "off") +
  scale_x_continuous(name = NULL, breaks = NULL) +
  scale_y_continuous(name = NULL, breaks = NULL) +
  scale_fill_manual(
    name = NULL,
    breaks = c("S", "F"),
    labels = c("success   ", "failure"),
    values = c(
      "S" = desaturate(darken("#0072B2", .4), .5),
      "F" = desaturate(lighten("#0072B2", .7), .5)
    ),
    guide = guide_legend(override.aes = list(size = 0))
  ) +
  facet_wrap(~label) +
  theme_dviz_grid() +
  theme(
    panel.spacing = unit(12, "pt"),
    legend.position = "bottom",
    legend.direction = "horizontal",
    legend.justification = "right",
    legend.box.spacing = unit(6, "pt"),
    legend.spacing.x = unit(3, "pt"),
    legend.key.size = unit(10, "pt"),
    plot.margin = margin(0, 0, 3.5, 0) # crop plot a little more tightly
  )
```

If we are only interested in two discrete outcomes, success or failure, then a visualization such as Figure \@ref(fig:probability-waffle) works fine. However, often we are dealing with more complex scenarios where the outcome of a random trial is a numeric variable. One common scenario is that of election predictions, where we are interested not only in who will win but also by how much. Let's consider a hypothetical example of an upcoming election with two parties, the yellow party and the blue party. Assume you hear on the radio that the blue party is predicted to have a one percentage point advantage over the yellow party, with a margin of error of 1.76 percentage points. What does this information tell you about the likely outcome of the election? It is human nature to hear "the blue party will win," but reality is more complicated. First, and most importantly, there is a range of different possible outcomes. The blue party could end up winning with a lead of two percentage points or the yellow party could end up winning with a lead of half a percentage point. The range of possible outcomes with their associated likelihoods is called a probability distribution, and we can draw it as a smooth curve that rises and then falls over the range of possible outcomes (Figure \@ref(fig:election-prediction)). The higher the curve for a specific outcome, the more likely that outcome is. Probability distributions are closely related to the histograms and kernel densities discussed in Chapter \@ref(histograms-density-plots), and you may want to re-read that chapter to refresh your memory.

(ref:election-prediction) Hypothetical prediction of an election outcome. The blue party is predicted to win over the yellow party by approximately one percentage point (labeled "best estimate"), but that prediction has a margin of error (here drawn so it covers 95% of the likely outcomes, 1.76 percentage points in either direction from the best estimate). The area shaded in blue, corresponding to 87.1% of the total, represents all outcomes under which blue would win. Likewise, the area shaded in yellow, corresponding to 12.9% of the total, represents all outcomes under which yellow would win. In this example, blue has an 87% chance of winning the election.

```{r election-prediction, fig.asp = 0.5, fig.cap = '(ref:election-prediction)'}
x <- c(seq(-2.5, 0, length.out = 50), seq(0.00001, 5, length.out = 100))
mu <- 1.02
sd <- .9

df_norm <- data.frame(
  x,
  y = dnorm(x, mu, sd),
  type = ifelse(x <= 0, "A", "B")
)

ci_x <- c(qnorm(.025, mu, sd), qnorm(0.975, mu, sd))
ci_y <- dnorm(ci_x, mu, sd)

df_annot <- data.frame(
  x = c(mu + 0.05, mu + 0.1, mu + 2.3*sd, mu - 2.5*sd),
  y = c(dnorm(mu, mu, sd) + 0.04, ci_y[1] + 0.01, 3*ci_y[1], 3*ci_y[1]),
  hjust = c(0, 0, 0.5, 0.5),
  vjust = c(1, 0, 0.5, 0.5),
  label = c("best estimate", "margin of error", "blue wins", "yellow wins")
)

ggplot(df_norm, aes(x, y)) +
  geom_area(aes(fill = type)) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  geom_line() +
  geom_segment(
    data = data.frame(x = 1),
    x = ci_x[1], xend = ci_x[2], y = ci_y[1], yend = ci_y[2],
    arrow = arrow(angle = 15, length = grid::unit(9, "pt"), ends = "both", type = "closed"),
    inherit.aes = FALSE
  ) +
  geom_segment(
    data = data.frame(x = 1),
    x = mu, xend = mu, y = 0, yend = dnorm(mu, mu, sd) + 0.04,
    inherit.aes = FALSE
  ) +
  geom_text(
    data = df_annot,
    aes(x, y, label = label, hjust = hjust, vjust = vjust),
    family = dviz_font_family,
    size = 12/.pt
  ) +
  scale_x_continuous(
    name = "percentage point advantage for blue",
    labels = scales::percent_format(accuracy = 0.1, scale = 1)
  ) +
  scale_y_continuous(
    name = NULL,
    breaks = NULL,
    expand = c(0, 0),
    limits = c(0, dnorm(mu, mu, sd) + 0.045)
  ) +
  scale_fill_manual(
    values = c(A = "#f8f1a9", B = "#b1daf4"),
    guide = "none"
  ) +
  theme_dviz_open()
```

By doing some math, we can calculate that for our made-up example, the chance of the yellow party winning is 12.9%. So the chance of yellow winning is a tad better than the 10% chance scenario shown in Figure \@ref(fig:probability-waffle). If you favor the blue party, you may not be overly worried, but the yellow party has enough of a chance of winning that they might just be successful. If you compare Figure \@ref(fig:election-prediction) to Figure \@ref(fig:probability-waffle), you may find that Figure \@ref(fig:probability-waffle) creates a much better sense of the uncertainty in outcome, even though the shaded areas in Figure \@ref(fig:election-prediction) accurately represent the probabilities of blue or yellow winning. This is the power of a discrete outcome visualization. Research in human perception shows that we are much better at perceiving, counting, and judging the relative frequencies of discrete objects---as long as their total number is not too large---than we are at judging the relative sizes of different areas.

We can combine the discrete outcome nature of Figure \@ref(fig:probability-waffle) with a continuous distribution as in Figure \@ref(fig:election-prediction) by drawing a quantile dotplot [@Kay_et_al_2016]. In the quantile dotplot, we subdivide the total area under the curve into evenly sized units and draw each unit as a circle. We then stack the circles such that their arrangement approximately represents the original distribution curve (Figure \@ref(fig:election-quantile-dot)).

(ref:election-quantile-dot) Quantile dotplot representations of the election outcome distribution of Figure \@ref(fig:election-prediction). (a) The smooth distribution is approximated with 50 dots representing a 2% chance each. The six yellow dots thus correspond to a 12% chance, reasonably close to the true value of 12.9%. (b) The smooth distribution is approximated with 10 dots representing a 10% chance each. The one yellow dot thus corresponds to a 10% chance, still close to the true value. Quantile dot plots with a smaller number of dots tend to be easier to read, so in this example, the 10-dot version might be preferable to the 50-dot version.

```{r election-quantile-dot, fig.asp = 0.72, fig.cap = '(ref:election-quantile-dot)'}
mu <- 1.02
sd <- 0.9
binwidth <- 0.31
binwidth <- 0.29

df_q <- data.frame(x = qnorm(ppoints(50), mu, sd)) %>%
  mutate(type = ifelse(x <= 0, "A", "B"))

p1 <- ggplot(df_q, aes(x, fill = type)) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  geom_line(data = df_norm, aes(x, 1.92*y)) + # factor 1.92 manually determined
  geom_dotplot(binwidth = binwidth) +
  scale_x_continuous(
    name = NULL, #"percent point advantage for blue",
    labels = scales::percent_format(accuracy = 0.1, scale = 1)
  ) +
  scale_y_continuous(
    name = NULL,
    breaks = NULL,
    expand = c(0, 0),
    limits = c(0, 0.9)
  ) +
  scale_fill_manual(
    values = c(A = "#f8f1a9", B = "#b1daf4"),
    guide = "none"
  ) +
  theme_dviz_open()

binwidth <- 0.31*2.1

df_q <- data.frame(x = qnorm(ppoints(10), mu, sd)) %>%
  mutate(type = ifelse(x <= 0, "A", "B"))

p2 <- ggplot(df_q, aes(x, fill = type)) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  geom_line(data = df_norm, aes(x, 1.92*y)) + # factor 1.92 manually determined
  geom_dotplot(binwidth = binwidth) +
  scale_x_continuous(
    name = "percentage point advantage for blue",
    labels = scales::percent_format(accuracy = 0.1, scale = 1)
  ) +
  scale_y_continuous(
    name = NULL,
    breaks = NULL,
    expand = c(0, 0),
    limits = c(0, 0.9)
  ) +
  scale_fill_manual(
    values = c(A = "#f8f1a9", B = "#b1daf4"),
    guide = "none"
  ) +
  theme_dviz_open()

plot_grid(p1, p2, align = 'h', labels = "auto", ncol = 1)
```

As a general principle, quantile dotplots should use a small to moderate number of dots. If there are too many dots, then we tend to perceive them as a continuum rather than as individual, discrete units. This negates the advantages of the discrete plots. Figure \@ref(fig:election-quantile-dot) shows variants with 50 dots (Figure \@ref(fig:election-quantile-dot)a) and with ten dots (Figure \@ref(fig:election-quantile-dot)b). While the version with 50 dots more accurately captures the true probability distribution, the number of dots is too large to easily discriminate individual ones. The version with ten dots more immediately conveys the relative chances of blue or yellow winning. One objection to the ten-dot version might be that it is not very precise. We are underrepresenting the chance of yellow winning by 2.9 percentage points. However, it is often worthwhile to trade some mathematical precision for more accurate human perception of the resulting visualization, in particular when communicating to a lay audience. A visualization that is mathematically correct but not properly perceived is not that useful in practice.


## Visualizing the uncertainty of point estimates

In Figure \@ref(fig:election-prediction), I showed a "best estimate" and a "margin of error," but I didn't explain what exactly these quantities are or how they might be obtained. To understand them better, we need to take a quick detour into basic concepts of statistical sampling. In statistics, our overarching goal is to learn something about the world by looking at a small portion of it. To continue with the election example, assume there are many different electoral districts and the citizens of each district are going to vote for either the blue or the yellow party. We might want to predict how each district is going to vote, as well as the overall vote average across districts (the *mean*). To make a prediction before the election, we cannot poll each individual citizen in each district about how they are going to vote. Instead, we have to poll a subset of citizens in a subset of districts and use that data to arrive at a best guess. In statistical language, the total set of possible votes of all citizens in all districts is called the *population,* and the subset of citizens and/or districts we poll is the *sample.* The population represents the underlying, true state of the world and the sample is our window into that world.

We are normally interested in specific quantities that summarize important properties of the population. In the election example, these could be the mean vote outcome across districts or the standard deviation among district outcomes. Quantities that describe the population are called *parameters,* and they are generally not knowable. However, we can use a sample to make a guess about the true parameter values, and statisticians refer to such guesses as *estimates.* The sample mean (or average) is an estimate for the population mean, which is a parameter. The estimates of individual parameter values are also called *point estimates,* since each can be represented by a point on a line.

Figure \@ref(fig:sampling-schematic) shows how these key concepts are related to each other. The variable of interest (e.g., vote outcome in each district) has some distribution in the population, with a population mean and a population standard deviation. A sample will consist of a set of specific observations. The number of the individual observations in the sample is called the *sample size.* From the sample we can calculate a sample mean and a sample standard deviation, and these will generally differ from the population mean and standard deviation. Finally, we can define a *sampling distribution,* which is the distribution of estimates we would obtain if we repeated the sampling process many times. The width of the sampling distribution is called the *standard error,* and it tells us how precise our estimates are. In other words, the standard error provides a measure of the uncertainty associated with our parameter estimate. As a general rule, the larger the sample size, the smaller the standard error and thus the less uncertain the estimate.

(ref:sampling-schematic) Key concepts of statistical sampling. The variable of interest that we are studying has some true distribution in the population, with a true population mean and standard deviation. Any finite sample of that variable will have a sample mean and standard deviation that differ from the population parameters. If we sampled repeatedly and calculated a mean each time, then the resulting means would be distributed according to the sampling distribution of the mean. The standard error provides information about the width of the sampling distribution, which informs us about how precisely we are estimating the parameter of interest (here, the population mean).

```{r sampling-schematic, fig.asp = 3/4, fig.cap = '(ref:sampling-schematic)'}
fill_color <- lighten("#56B4E9", 0.2)
fill_color <- "lightblue"

set.seed(452061)


empty_theme <- theme_dviz_open(12, rel_small = 1, rel_large = 1) +
  theme(
    axis.line = element_blank(),
    axis.text = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = grid::unit(0, "pt")
  )

x <- c(seq(-4, 4, length.out = 200))

df_norm <- data.frame(
  x,
  y = dnorm(x)
)

sd_x <- c(-1, 1)
sd_y <- dnorm(sd_x)

df_annot <- data.frame(
  x = c(0.05, sd_x[2] + 0.04, -Inf),
  y = c(dnorm(0) * 0.4, sd_y[2] * 1.01, Inf), #sd_y[1] * 1.1
  hjust = c(0, 0, 0),
  vjust = c(1, 0.5, 1),
  label = c("mean", "standard deviation", "population distribution")
)


p1 <- ggplot(df_norm, aes(x, y)) +
  geom_area(fill = fill_color) +
  geom_segment( # standard deviation
    data = data.frame(x = 1),
    x = 0, xend = sd_x[2], y = sd_y[1], yend = sd_y[2],
    arrow = arrow(angle = 90, length = grid::unit(3, "pt"), ends = "both", type = "closed"),
    inherit.aes = FALSE
  ) +
  geom_segment( # vertical line representing mean
    data = data.frame(x = 1),
    x = 0, xend = 0, y = 0, yend = dnorm(0),
    linetype = 2,
    inherit.aes = FALSE
  ) +
  geom_text(
    data = df_annot,
    aes(x, y, label = label, hjust = hjust, vjust = vjust),
    family = dviz_font_family,
    size = c(12, 12, 14)/.pt
  ) +
  scale_x_continuous(
    limits = c(-4, 4), expand = c(0, 0),
    breaks = 0, # workaround to fix missing axis line
    name = "variable of interest"
  ) +
  scale_y_continuous(breaks = NULL, name = NULL, expand = expand_scale(mult = c(0, 0.1))) +
  empty_theme +
  theme(axis.line.x = element_line(), axis.title.x = element_text(hjust = 1))

n <- 15
df_sample <- data.frame(
  x = rnorm(n),
  y = 0
)

df_annot2 <- data.frame(
  x = c(mean(df_sample$x) + 0.05, sort(df_sample$x)[2],
        mean(df_sample$x) + sd(df_sample$x) + 0.05, -Inf),
  y = c(-0.15, 0.12, .13 + 0.01, Inf),
  hjust = c(0, 0.3, 0, 0),
  vjust = c(0.5, 0.5, 0.5, 1),
  label = c("sample mean", "observations", "sample standard deviation", "sample")
)


p2 <- ggplot(df_sample, aes(x, y)) +
  geom_point(
    size = 3, fill = fill_color, shape = 21, stroke = 0.5,
    position = position_jitter(width = 0, height = 0.01, seed = 127)
  ) +
  geom_segment( # vertical bar representing mean
    data = data.frame(x = 1),
    aes(x = mean(df_sample$x), xend = mean(df_sample$x), y = -.2, yend = .2),
    size = 1.5,
    color = "#D55E00",
    inherit.aes = FALSE
  ) +
  geom_segment( # horizontal bar representing sd
    data = data.frame(x = 1),
    x = mean(df_sample$x), xend = mean(df_sample$x) + sd(df_sample$x), y = .13, yend = .13,
    arrow = arrow(angle = 90, length = grid::unit(3, "pt"), ends = "both", type = "closed"),
    inherit.aes = FALSE
  ) +
  geom_text(
    data = df_annot2,
    aes(x, y, label = label, hjust = hjust, vjust = vjust),
    family = dviz_font_family,
    size = c(12, 12, 12, 14)/.pt
  ) +
  scale_x_continuous(limits = c(-4, 4), expand = c(0, 0), breaks = NULL, name = NULL) +
  scale_y_continuous(expand = c(0.1, 0), breaks = NULL, name = NULL) +
  empty_theme

df_samplingdist <- data.frame(
  x,
  y = dnorm(x, 0, 1/sqrt(n))
)

se_x <- c(-1/sqrt(n), 1/sqrt(n))
se_y <- dnorm(se_x, 0, 1/sqrt(n))

df_annot3 <- data.frame(
  x = c(0.05, se_x[2] + 0.04, -Inf),
  y = c(dnorm(0, 0, 1/sqrt(n)) * 0.4, se_y[2] * 1.01, Inf),
  hjust = c(0, 0, 0),
  vjust = c(1, 0.5, 1),
  label = c("mean of the sample means", "standard error", "sampling distribution of the mean")
)

p3 <- ggplot(df_samplingdist, aes(x, y)) +
  geom_area(fill = fill_color) +
  geom_segment( # standard error
    data = data.frame(x = 1),
    x = 0, xend = se_x[2], y = se_y[1], yend = se_y[2],
    arrow = arrow(angle = 90, length = grid::unit(3, "pt"), ends = "both", type = "closed"),
    inherit.aes = FALSE
  ) +
  geom_segment(
    data = data.frame(x = 1),
    x = 0, xend = 0, y = 0, yend = dnorm(0, 0, 1/sqrt(n)),
    linetype = 2,
    inherit.aes = FALSE
  ) +
  geom_text(
    data = df_annot3,
    aes(x, y, label = label, hjust = hjust, vjust = vjust),
    family = dviz_font_family,
    size = c(12, 12, 14)/.pt
  ) +
  scale_x_continuous(
    limits = c(-4, 4), expand = c(0, 0),
    breaks = 0, # workaround to fix missing axis line
    name = "sample mean"
  ) +
  scale_y_continuous(breaks = NULL, name = NULL, expand = expand_scale(mult = c(0, 0.1))) +
  empty_theme +
  theme(axis.line.x = element_line(), axis.title.x = element_text(hjust = 1))

plot_grid(
  p1,
  p2,
  p3,
  ncol = 1, rel_heights = c(1, .4, 1), align = 'v'
)
```

It is critical that we don't confuse the standard deviation and the standard error. The standard deviation is a property of the population. It tells us how much spread there is among individual observations we could make. For example, if we consider the population of voting districts, the standard deviation tells us how different different districts are from one another. By contrast, the standard error tells us how precisely we have determined a parameter estimate. If we wanted to estimate the mean voting outcome over all districts, the standard error would tells us how accurate our estimate for the mean is.

All statisticians use samples to calculate parameter estimates and their uncertainties. However, they are divided in how they approach these calculations, into Bayesians and frequentists. Bayesians assume that they have some prior knowledge about the world, and they use the sample to update this knowledge. By contrast, frequentists attempt to make precise statements about the world without having any prior knowledge in hand. Fortunately, when it comes to visualizing uncertainty, Bayesians and frequentists can generally employ the same types of strategies. Here, I will first discuss the frequentist approach and then describe a few specific issues unique to the Bayesian context.

Frequentists most commonly visualize uncertainty with error bars. While error bars can be useful as a visualization of uncertainty, they are not without problems, as I already alluded to in Chapter \@ref(boxplots-violins) (see Figure \@ref(fig:lincoln-temp-points-errorbars)). It is easy for readers to be confused about what an error bar represents. To highlight this problem, in Figure \@ref(fig:cocoa-data-vs-CI) I show five different uses of error bars for the same dataset. The dataset contains expert ratings of chocolate bars, rated on a scale from 1 to 5, for chocolate bars manufactured in a number of different countries. For Figure \@ref(fig:cocoa-data-vs-CI) I have extracted all ratings for chocolate bars manufactured in Canada. Underneath the sample, which is shown as a strip chart of jittered dots, we see the sample mean plus/minus the standard deviation of the sample, the sample mean plus/minus the standard error, and 80%, 95%, and 99% confidence intervals. All five error bars are derived from the variation in the sample, and they are all mathematically related, but they have different meanings. And they are also visually quite distinct.

(ref:cocoa-data-vs-CI) Relationship between sample, sample mean, standard deviation, standard error, and confidence intervals, in an example of chocolate bar ratings. The observations (shown as jittered green dots) that make up the sample represent expert ratings of 125 chocolate bars from manufacturers in Canada, rated on a scale from 1 (unpleasant) to 5 (elite). The large orange dot represents the mean of the ratings. Error bars indicate, from top to bottom, twice the standard deviation, twice the standard error (standard deviation of the mean), and 80%, 95%, and 99% confidence intervals of the mean. Data source: Brady Brelinski, Manhattan Chocolate Society

```{r cocoa-data-vs-CI, fig.width = 5*6/4.2, fig.asp = 0.48, warning = FALSE, message = FALSE, fig.cap = '(ref:cocoa-data-vs-CI)'}

# color for individual small data points
point_color <- darken("#009E73", .3)

cacao %>% 
  filter(location == "Canada") -> cacao_single

fit <- lm(rating ~ 1, data = cacao_single)

CI_df <- data.frame(type = c(0.8, 0.95, 0.99)) %>%
  mutate(df = map(type, ~tidy(emmeans(fit, ~ 1, options = list(level = .x))))) %>%
  unnest() %>%
  select(type, estimate, std.error, conf.low, conf.high) %>%
  mutate(type = paste0(signif(100*type, 2), "% confidence interval"))

CI_df <- rbind(
  CI_df,
  data.frame(
    type = "standard error",
    estimate = CI_df$estimate[1],
    std.error = CI_df$std.error[1],
    conf.low = CI_df$estimate[1] - CI_df$std.error[1],
    conf.high = CI_df$estimate[1] + CI_df$std.error[1]
  ),
  data.frame(
    type = "standard deviation",
    estimate = mean(cacao_single$rating),
    std.error = CI_df$std.error[1],
    conf.low = mean(cacao_single$rating) - sd(cacao_single$rating),
    conf.high = mean(cacao_single$rating) + sd(cacao_single$rating)
  ),
  data.frame(
    type = "sample", estimate = mean(cacao_single$rating), std.error = NA,
    conf.low = NA, conf.high = max(cacao_single$rating)
  )
) %>%
  mutate(
    type = fct_relevel(factor(type), "sample", "standard deviation", "standard error"),
    label = case_when(
      type == "sample" ~ NA_character_,
      type == "standard deviation" ~ "+/- standard deviation",
      type == "standard error" ~ "+/- standard error",
      TRUE ~ as.character(type) #paste0("mean +/- ", type)
    )
  )

label_x <- filter(CI_df, type == "standard deviation")$conf.high + 0.04

ggplot(CI_df, aes(estimate, type)) + 
  geom_point(
    data = cacao_single, aes(rating, "sample"), 
    position = position_jitter(height = 0.6, width = 0.02, seed = 7843),
    color = point_color,
    size = 0.3
  ) +
  geom_errorbarh(aes(xmin = conf.low, xmax = conf.high), height = 0.2, na.rm = TRUE) +
  geom_point(size = 2, color = "#D55E00") +
  geom_label(
    aes(label_x, label = label), hjust = 0, nudge_y = 0.01, na.rm = TRUE,
    family = dviz_font_family,
    size = 14/.pt,
    label.size = 0
  ) +
  geom_label(
    data = filter(CI_df, type == "sample"),
    aes(conf.high + 0.06, label = type), hjust = 0, nudge_y = 0.01,
    family = dviz_font_family,
    size = 14/.pt,
    label.size = 0
  ) +
  geom_text(
    data = filter(CI_df, type == "sample"),
    aes(estimate, label = "mean"), hjust = 0.2, vjust = 0, nudge_y = 0.2,
    family = dviz_font_family,
    size = 14/.pt
  ) +
  scale_x_continuous(
    limits = c(1.95, 4.1),
    expand = c(0, 0),
    name = "chocolate flavor rating"
  ) +
  scale_y_discrete(
    name = NULL,
    limits = rev(levels(CI_df$type)),
    expand = expand_scale(add = c(0.6, 0.8)),
    breaks = NULL
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_vgrid(14, rel_small = 1) +
  theme(
    plot.margin = margin(3, 82, 3, 1.5),
    axis.line.x = element_line(),
    axis.ticks.x = element_line(color = "black"),
    axis.title.x = element_text(hjust = 1)
  )

```


```{block type='rmdtip', echo=TRUE}
Whenever you visualize uncertainty with error bars, you must specify what quantity and/or confidence level the error bars represent.
```

The standard error is approximately given by the sample standard deviation divided by the square root of the sample size, and confidence intervals are calculated by multiplying the standard error with small, constant values. For example, a 95% confidence interval extends approximately two times the standard error in either direction from the mean. Therefore, larger samples tend to have narrower standard errors and confidence intervals, even if their standard deviation is the same. We can see this effect when we compare ratings for chocolate bars from Canada to ones from Switzerland (Figure \@ref(fig:cocoa-CI-vs-n)). The mean rating and sample standard deviation are comparable between Canadian and Swiss chocolate bars, but we have ratings for 125 Canadian bars and only 38 Swiss bars, and consequently the confidence intervals around the mean are much wider in the case of Swiss bars.

(ref:cocoa-CI-vs-n) Confidence intervals widen with smaller sample size. Chocolate bars from Canada and Switzerland have comparable mean ratings and comparable standard deviations (indicated with simple black error bars). However, over three times as many Canadian bars were rated as Swiss bars, and therefore the confidence intervals (indicated with error bars of different colors and thickness drawn on top of one another) are substantially wider for the mean of the Swiss ratings than for the mean of the Canadian ratings. Data source: Brady Brelinski, Manhattan Chocolate Society

```{r cocoa-CI-vs-n, fig.width = 5*6/4.2, fig.asp = 0.46, warning = FALSE, message = FALSE, fig.cap = '(ref:cocoa-CI-vs-n)'}
cacao_CA <- filter(cacao, location == "Canada")
cacao_CH <- filter(cacao, location == "Switzerland")

fit_CA <- lm(rating ~ 1, data = cacao_CA)
fit_CH <- lm(rating ~ 1, data = cacao_CH)

CI_CA <- data.frame(level = c(0.99, 0.95, 0.8)) %>%
  mutate(df = map(level, ~tidy(emmeans(fit_CA, ~ 1, options = list(level = .x))))) %>%
  unnest() %>%
  select(level, estimate, conf.low, conf.high) %>%
  mutate(
    level = paste0(signif(100*level, 2), "%"),
    type = "CI",
    location = "Canada"
  )

CI_CH <- data.frame(level = c(0.99, 0.95, 0.8)) %>%
  mutate(df = map(level, ~tidy(emmeans(fit_CH, ~ 1, options = list(level = .x))))) %>%
  unnest() %>%
  select(level, estimate, conf.low, conf.high) %>%
  mutate(
    level = paste0(signif(100*level, 2), "%"),
    type = "CI",
    location = "Switzerland"
  )

CI_df <- rbind(CI_CA, CI_CH)

sd_df <- rbind(
  data.frame(
    level = NA,
    estimate = mean(cacao_CA$rating),
    conf.low = mean(cacao_CA$rating) - sd(cacao_CA$rating),
    conf.high = mean(cacao_CA$rating) + sd(cacao_CA$rating),
    type = "original data",
    location = "Canada"
  ),
  data.frame(
    level = NA,
    estimate = mean(cacao_CH$rating),
    conf.low = mean(cacao_CH$rating) - sd(cacao_CH$rating),
    conf.high = mean(cacao_CH$rating) + sd(cacao_CH$rating),
    type = "original data",
    location = "Switzerland"
  )
)

#label_x <- filter(CI_df, type == "standard deviation")$conf.high + 0.04

ggplot(rbind(CI_df, sd_df), aes(estimate, interaction(location, type))) + 
  geom_point(
    data = cacao_CA, # draw two separate layers to get jittering right relative to previous figure
    aes(rating, interaction(location, "original data")),
    position = position_jitter(height = 0.6, width = 0.02, seed = 7843),
    color = point_color,
    size = 0.3
  ) +
  geom_point(
    data = cacao_CH,
    aes(rating, interaction(location, "original data")),
    position = position_jitter(height = 0.6, width = 0.02, seed = 7844),
    color = point_color,
    size = 0.3
  ) +
  geom_errorbarh(
    data = sd_df,
    aes(y = interaction(location, "original data"), xmin = conf.low, xmax = conf.high),
    height = 0.2
  ) +
  geom_errorbarh( # error bar without cap
    data = CI_df,
    aes(y = interaction(location, "CI"), xmin = conf.low, xmax = conf.high, color = level, size = level),
    height = 0
  ) +
  geom_errorbarh( # error bar with cap
    data = CI_df,
    aes(y = interaction(location, "CI"), xmin = conf.low, xmax = conf.high, color = level),
    height = 0.2
  ) +
  geom_point(size = 2, color = "#D55E00") +
  geom_label(
    data = data.frame(
      estimate = 4.06,
      location = c("Canada", "Switzerland"),
      type = "original data",
      label = c(
        paste0("Canada,\nn = ", nrow(cacao_CA)),
        paste0("Switzerland,\nn = ", nrow(cacao_CH))
      )
    ),
    aes(label = label), hjust = 0, vjust = 0.5, nudge_y = 0.01,
    family = dviz_font_family,
    size = 14/.pt,
    label.size = 0
  ) +
  scale_x_continuous(
    limits = c(1.95, 4.1),
    expand = c(0, 0),
    name = "chocolate flavor rating"
  ) +
  scale_y_discrete(
    name = NULL,
    limits = rev(c("Canada.original data", "Canada.CI", "dummy", "Switzerland.original data", "Switzerland.CI")),
    expand = expand_scale(add = c(1, 0.8)),
    breaks = NULL
  ) +
  scale_fill_manual(
    aesthetics = c("color", "fill"),
    name = "confidence level",
    values = c(
     `80%` = desaturate(darken("#0072B2", .2), .3),
     `95%` = desaturate(lighten("#0072B2", .2), .3),
     `99%` = desaturate(lighten("#0072B2", .4), .3)
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  scale_size_manual(
    name = "confidence level",
    values = c(
     `80%` = 2.25,
     `95%` = 1.5,
     `99%` = .75
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_vgrid(14, rel_small = 1) +
  theme(
    plot.margin = margin(3, 82, 3, 1.5),
    axis.line.x = element_line(),
    axis.ticks.x = element_line(color = "black"),
    axis.title.x = element_text(hjust = 1),
    legend.position = c(0, 0.01),
    legend.justification = c(0, 0),
    legend.key.height = grid::unit(7, "pt"),
    legend.key.width = grid::unit(35, "pt"),
    legend.spacing.x = grid::unit(7, "pt"),
    legend.spacing.y = grid::unit(3.5, "pt"),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.spacing = grid::unit(0, "pt"),
    legend.title.align = 0.5
  )

```

In Figure \@ref(fig:cocoa-CI-vs-n), I am showing three different confidence intervals at the same time, using darker colors and thicker lines for the intervals representing lower confidence levels. I refer to these visualizations as *graded error bars*. The grading helps the reader perceive that there is a range of different possibilities. If I showed simple  error bars (without grading) to a group of people, chances are at least some of them would perceive the error bars deterministically, for example as representing minimum and maximum of the data. Alternatively, they might think the error bars delineate the range of possible parameter estimates, i.e., the estimate could never fall outside the error bars. These types of misperception are called *deterministic construal errors.* The more we can minimize the risk of deterministic construal error, the better our visualization of uncertainty.

Error bars are convenient because they allow us to show many estimates with their uncertainties all at once. Therefore, they are commonly used in scientific publications, where the primary goal is usually to convey a large amount of information to an expert audience. As an example of this type of application, Figure \@ref(fig:mean-chocolate-ratings) shows mean chocolate ratings and associated confidence intervals for chocolate bars manufactured in six different countries.

(ref:mean-chocolate-ratings) Mean chocolate flavor ratings and associated confidence intervals for chocolate bars from manufacturers in six different countries. Data source: Brady Brelinski, Manhattan Chocolate Society

```{r mean-chocolate-ratings, fig.width = 5*6/4.2, fig.asp = 0.5, fig.cap = '(ref:mean-chocolate-ratings)'}
#countries <- c("Austria", "Belgium", "Canada", "Peru", "Switzerland")
#
cacao_small <- 
  cacao_small %>% 
  mutate(location = fct_recode(location, US = "U.S.A.")) # change to O'Reilly style

fit <- lm(rating ~ location, data = cacao_small)

conf_df <- data.frame(level = c(0.99, 0.95, 0.8)) %>%
  mutate(df = map(level, ~tidy(emmeans(fit, ~location, options = list(level = .x))))) %>%
  unnest() %>%
  select(level, location, estimate, std.error, conf.low, conf.high) %>%
  mutate(level = paste0(signif(100*level, 2), "%"))

ggplot(conf_df, aes(estimate, reorder(location, estimate))) + 
  geom_errorbarh(
    aes(xmin = conf.low, xmax = conf.high, color = level, size = level),
    height = 0
  ) +
  geom_errorbarh(
    aes(xmin = conf.low, xmax = conf.high, color = level),
    height = 0.2
  ) +
  geom_point(data = filter(conf_df, level == "80%"), size = 2.5, color = "#D55E00") +
  scale_x_continuous(
    limits = c(2.6, 3.6),
#    expand = c(0, 0),
    name = "mean rating"
  ) +
 scale_y_discrete(
#    position = "right",
#    limits = rev(c("Canada.original data", "Canada.CI", "Switzerland.original data", "Switzerland.CI")),
#    breaks = NULL,
   name = NULL
  ) +
  scale_color_manual(
    name = "confidence level",
    values = c(
     `80%` = desaturate(darken("#0072B2", .2), .3),
     `95%` = desaturate(lighten("#0072B2", .2), .3),
     `99%` = desaturate(lighten("#0072B2", .4), .3)
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  scale_size_manual(
    name = "confidence level",
    values = c(
     `80%` = 2.25,
     `95%` = 1.5,
     `99%` = 0.75
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(14, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.ticks.x = element_line(color = "black"),
    axis.title.x = element_text(hjust = 1),
    legend.position = c(1, 0.01),
    legend.justification = c(1, 0),
    legend.key.height = grid::unit(7, "pt"),
    legend.key.width = grid::unit(35, "pt"),
    legend.spacing.x = grid::unit(7, "pt"),
    legend.spacing.y = grid::unit(3.5, "pt"),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.spacing = grid::unit(0, "pt"),
    legend.title.align = 0.5
  )

```

When looking at Figure \@ref(fig:mean-chocolate-ratings), you may wonder what it tells us about the differences in mean ratings. The mean ratings of Canadian, Swiss, and Austrian bars are higher than the mean rating of U.S. bars, but given the uncertainty in these mean ratings, are the differences in means *significant*? The word "significant" here is a technical term used by statisticians. We call a difference significant if with some level of confidence we can reject the assumption that the observed difference was caused by random sampling. Since only a finite number of Canadian and U.S. bars was rated, the raters could have accidentally considered more of the better Canadian bars and fewer of the better U.S. bars, and this random chance might look like a systematic rating advantage of Canadian over U.S. bars.

Assessing significance from Figure \@ref(fig:mean-chocolate-ratings) is difficult, because both the mean Canadian rating and the mean U.S. rating have uncertainty. Both uncertainties matter to the question whether the means are different. Statistics textbooks and online tutorials sometimes publish rules of thumb of how to judge significance from the extent to which error bars do or don't overlap. However, these rules of thumb are not reliable and should be avoided. The correct way to assess whether there are differences in mean rating is to calculate confidence intervals for the differences. If those confidence intervals exclude zero, then we know the difference is significant at the respective confidence level. For the chocolate ratings dataset, we see that only bars from Canada are significantly higher rated than bars from the U.S. (Figure \@ref(fig:chocolate-ratings-contrasts)). For bars from Switzerland, the 95% confidence interval on the difference just barely includes the value zero. Thus, the difference between the mean ratings of U.S. and Swiss chocolate bars is barely not significant at the 5% level. Finally, there is no evidence at all that Austrian bars have systematically higher mean ratings that U.S. bars.

(ref:chocolate-ratings-contrasts) Mean chocolate flavor ratings for manufacturers from five different countries, relative to the mean rating of U.S. chocolate bars. Canadian chocolate bars are significantly higher rated that U.S. bars. For the other four countries there is no significant difference in mean rating to the U.S. at the 95% confidence level. Confidence levels have been adjusted for multiple comparisons using Dunnett's method. Data source: Brady Brelinski, Manhattan Chocolate Society

```{r chocolate-ratings-contrasts, fig.width = 5*6/4.2, fig.asp = 0.5, fig.cap = '(ref:chocolate-ratings-contrasts)'}
# need reference grid for contrasts
fit_rg <- ref_grid(fit)

contrasts_dunnettx <- data.frame(level = c(0.99, 0.95, 0.8)) %>%
  mutate(
    df = map(
      level,
      ~data.frame(confint(contrast(fit_rg, method = "trt.vs.ctrl1"), level = .x))
    )
  ) %>%
  unnest() %>%
  select(level, contrast, estimate, std.error = SE, conf.low = lower.CL, conf.high = upper.CL) %>%
  mutate(
    level = paste0(signif(100*level, 2), "%"),
    contrast = stringr::str_extract(as.character(contrast), "[a-zA-Z]+")
  )

ggplot(contrasts_dunnettx, aes(x = estimate, y = reorder(contrast, estimate))) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  geom_errorbarh(
    aes(xmin = conf.low, xmax = conf.high, color = level, size = level),
    height = 0
  ) +
  geom_errorbarh(
    aes(xmin = conf.low, xmax = conf.high, color = level),
    height = 0.2
  ) +
  geom_point(data = filter(contrasts_dunnettx, level == "80%"), size = 2.5, color = "#D55E00") +
  scale_x_continuous(
    name = "difference in mean rating",
    sec.axis = dup_axis(
      name = NULL,
      breaks = 0,
      labels = "US mean rating"
    )
  ) +
  scale_y_discrete(
    name = NULL
  ) +
  scale_color_manual(
    name = "confidence level",
    values = c(
     `80%` = desaturate(darken("#0072B2", .2), .3),
     `95%` = desaturate(lighten("#0072B2", .2), .3),
     `99%` = desaturate(lighten("#0072B2", .4), .3)
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  scale_size_manual(
    name = "confidence level",
    values = c(
     `80%` = 2.25,
     `95%` = 1.5,
     `99%` = 0.75
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(14, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.line.x.top = element_blank(),
    axis.ticks.x = element_line(color = "black"),
    axis.ticks.x.top = element_line(color = "gray50"),
    #axis.title.x = element_text(hjust = 1),
    legend.position = c(1, 0.02),
    legend.justification = c(1, 0),
    legend.key.height = grid::unit(7, "pt"),
    legend.key.width = grid::unit(35, "pt"),
    legend.spacing.x = grid::unit(7, "pt"),
    legend.spacing.y = grid::unit(3.5, "pt"),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.spacing = grid::unit(0, "pt"),
    legend.title.align = 0.5
  )

```

In the preceding figures, I have used two different types of error bars, graded and simple. More variations are possible. For example, we can draw error bars with or without a cap at the end (Figure \@ref(fig:confidence-visualizations)a, c versus Figure \@ref(fig:confidence-visualizations)b, d). There are advantages and disadvantages to all these choices. Graded error bars highlight the existence of different ranges corresponding to different confidence levels. However, the flip side of this additional information is added visual noise. Depending on how complex and information-dense a figure is otherwise, simple error bars may be preferable to graded ones. Whether to draw error bars with or without cap is primarily a question of personal taste. A cap highlights where exactly an error bar ends (Figure \@ref(fig:confidence-visualizations)a, c), whereas an error bar without cap puts equal emphasis on the entire range of the interval (Figure \@ref(fig:confidence-visualizations)b, d). Also, again, caps add visual noise, so in a figure with many error bars omitting caps may be preferable.

(ref:confidence-visualizations) Mean chocolate flavor ratings for manufacturers from four different countries, relative to the mean rating of U.S. chocolate bars. Each panel uses a different approach to visualizing the same uncertainty information. (a) Graded error bars with cap. (b) Graded error bars without cap. (c) Single-interval error bars with cap. (d) Single-interval error bars without cap. (e) Confidence strips. (f) Confidence distributions.

```{r confidence-visualizations, fig.width = 5.5*6/4.2, fig.asp = 0.75, fig.cap = '(ref:confidence-visualizations)'}
cacao_smaller <- filter(cacao_small, location != "Switzerland")

fit <- lm(rating ~ location, data = cacao_smaller)
fit_rg <- ref_grid(fit)

contrasts_dunnettx <- data.frame(level = c(0.99, 0.95, 0.8)) %>%
  mutate(
    df = map(
      level,
      ~data.frame(confint(contrast(fit_rg, method = "trt.vs.ctrl1"), level = .x))
    )
  ) %>%
  unnest() %>%
  select(level, contrast, estimate, std.error = SE, conf.low = lower.CL, conf.high = upper.CL) %>%
  mutate(
    level = paste0(signif(100*level, 2), "%"),
    contrast = stringr::str_extract(as.character(contrast), "[a-zA-Z]+")
  )

p1 <- ggplot(contrasts_dunnettx, aes(x = estimate, y = reorder(contrast, estimate))) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  geom_errorbarh(
    aes(xmin = conf.low, xmax = conf.high, color = level, size = level),
    height = 0
  ) +
  geom_errorbarh(
    aes(xmin = conf.low, xmax = conf.high, color = level),
    height = 0.2
  ) +
  geom_point(data = filter(contrasts_dunnettx, level == "80%"), size = 2.5, color = "#D55E00") +
  scale_x_continuous(
    name = "difference in mean rating",
    limits = c(-.65, .47),
    expand = c(0, 0)
  ) +
  scale_y_discrete(name = NULL) +
  scale_color_manual(
    name = "confidence level",
    values = c(
     `80%` = desaturate(darken("#0072B2", .2), .3),
     `95%` = desaturate(lighten("#0072B2", .2), .3),
     `99%` = desaturate(lighten("#0072B2", .4), .3)
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  scale_size_manual(
    name = "confidence level",
    values = c(
     `80%` = 1.5,
     `95%` = 1,
     `99%` = 0.5
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.line.x.top = element_blank(),
    axis.ticks.x = element_line(color = "black"),
    axis.ticks.x.top = element_line(color = "gray50"),
    #axis.title.x = element_text(hjust = 1),
    legend.position = "none",
    #legend.position = c(1, .01),
    legend.justification = c(1, 0),
    legend.key.height = grid::unit(6, "pt"),
    legend.key.width = grid::unit(30, "pt"),
    legend.spacing.x = grid::unit(6, "pt"),
    legend.spacing.y = grid::unit(3, "pt"),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.spacing = grid::unit(0, "pt"),
    legend.title.align = 0.5
  )

p2 <- ggplot(filter(contrasts_dunnettx, level == "95%"),
             aes(x = estimate, y = reorder(contrast, estimate))) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  geom_errorbarh(
    aes(xmin = conf.low, xmax = conf.high),
    height = 0.2
  ) +
  geom_point(size = 2.5, color = "#D55E00") +
  scale_x_continuous(
    name = "difference in mean rating",
    limits = c(-.65, .47),
    expand = c(0, 0)
  ) +
  scale_y_discrete(name = NULL) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.line.x.top = element_blank(),
    axis.ticks.x = element_line(color = "black"),
    axis.ticks.x.top = element_line(color = "gray50")
  )

p3 <- ggplot(contrasts_dunnettx, aes(x = estimate, y = reorder(contrast, estimate))) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  geom_errorbarh(
    aes(xmin = conf.low, xmax = conf.high, color = level, size = level),
    height = 0
  ) +
  geom_point(data = filter(contrasts_dunnettx, level == "80%"), size = 2.5, color = "#D55E00") +
  scale_x_continuous(
    name = "difference in mean rating",
    limits = c(-.65, .47),
    expand = c(0, 0)
  ) +
  scale_y_discrete(name = NULL) +
  scale_color_manual(
    name = "confidence level",
    values = c(
     `80%` = desaturate(darken("#0072B2", .2), .3),
     `95%` = desaturate(lighten("#0072B2", .2), .3),
     `99%` = desaturate(lighten("#0072B2", .4), .3)
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  scale_size_manual(
    name = "confidence level",
    values = c(
     `80%` = 1.5,
     `95%` = 1,
     `99%` = 0.5
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.line.x.top = element_blank(),
    axis.ticks.x = element_line(color = "black"),
    axis.ticks.x.top = element_line(color = "gray50"),
    #axis.title.x = element_text(hjust = 1),
    legend.position = "none",
    #legend.position = c(1, .01),
    legend.justification = c(1, 0),
    legend.key.height = grid::unit(6, "pt"),
    legend.key.width = grid::unit(24, "pt"),
    legend.spacing.x = grid::unit(6, "pt"),
    legend.spacing.y = grid::unit(3, "pt"),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.spacing = grid::unit(0, "pt"),
    legend.title.align = 0.5
  )

p4 <- ggplot(filter(contrasts_dunnettx, level == "95%"),
             aes(x = estimate, y = reorder(contrast, estimate))) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  geom_errorbarh(
    aes(xmin = conf.low, xmax = conf.high),
    height = 0
  ) +
  geom_point(size = 2.5, color = "#D55E00") +
  scale_x_continuous(
    name = "difference in mean rating",
    limits = c(-.65, .47),
    expand = c(0, 0)
  ) +
  scale_y_discrete(name = NULL) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.line.x.top = element_blank(),
    axis.ticks.x = element_line(color = "black"),
    axis.ticks.x.top = element_line(color = "gray50")
  )


p5 <- ggplot(filter(contrasts_dunnettx, level == "95%"),
             aes(x = estimate, y = reorder(contrast, estimate))) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  stat_confidence_density(
    aes(moe = conf.high-estimate, fill = stat(ndensity)),
    height = 0.7,
    confidence = 0.95,
    alpha = NA
  ) +
  geom_segment(
    aes(
      x = estimate, xend = estimate,
      y = as.integer(reorder(contrast, estimate)) - 0.35,
      yend = as.integer(reorder(contrast, estimate)) + 0.35
    ),
    size = 2, color = "#D55E00"
  ) +
  scale_fill_gradient(low = "#81A7D600", high = "#345A7FD0") +
  scale_x_continuous(
    name = "difference in mean rating",
    limits = c(-.65, .47),
    expand = c(0, 0)
  ) +
  scale_y_discrete(name = NULL) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.line.x.top = element_blank(),
    axis.ticks.x = element_line(color = "black"),
    axis.ticks.x.top = element_line(color = "gray50")
  )


p6 <- ggplot(filter(contrasts_dunnettx, level == "95%"),
             aes(x = estimate, y = reorder(contrast, estimate))) +
  geom_vline(xintercept = 0, linetype = 2, color = "gray50") +
  stat_confidence_density(
    aes(moe = conf.high-estimate, height = stat(density)),
    geom = "ridgeline",
    confidence = 0.95,
    scale = 0.2,
    min_height = 0.001,
    alpha = 0.7,
    fill = "#81A7D6",
    size = 0.4
  ) +
  geom_segment(
    aes(
      x = estimate, xend = estimate,
      y = as.integer(reorder(contrast, estimate)) - 0.2,
      yend = as.integer(reorder(contrast, estimate)) + 0.5
    ),
    size = 2, color = "#D55E00"
  ) +
  scale_fill_gradient(low = "#81A7D600", high = "#345A7FD0") +
  scale_x_continuous(
    name = "difference in mean rating",
    limits = c(-.65, .47),
    expand = c(0, 0)
  ) +
  scale_y_discrete(name = NULL) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.line.x.top = element_blank(),
    axis.ticks.x = element_line(color = "black"),
    axis.ticks.x.top = element_line(color = "gray50"),
    axis.text.y = element_text(vjust = 0)
  )

plot_grid(
  p1, NULL, p3,
  NULL, NULL, NULL,
  p2, NULL, p4,
  NULL, NULL, NULL,
  p5, NULL, p6,
  ncol = 3,
  labels = c("a", "", "b", "", "", "", "c", "", "d", "", "", "", "e", "", "f"),
  rel_widths = c(1, .02, 1),
  rel_heights = c(1, .02, 1, .02, 1)
)
```

As an alternative to error bars we could draw confidence strips that gradually fade into nothing (Figure \@ref(fig:confidence-visualizations)e). Confidence strips better convey how probable different values are, but they are difficult to read. We would have to visually integrate the different shadings of color to determine where a specific confidence level ends. From Figure \@ref(fig:confidence-visualizations)e we might conclude that the mean rating for Peruvian chocolate bars is significantly lower than that of U.S. chocolate bars, and yet this is not the case. Similar problems arise when we show explicit confidence distributions (Figure \@ref(fig:confidence-visualizations)f). It is difficult to visually integrate the area under the curve and to determine where exactly a given confidence level is reached. This issue can be somewhat alleviated, however, by drawing quantile dotplots as in Figure \@ref(fig:election-quantile-dot).

For simple 2D figures, error bars have one important advantage over more complex displays of uncertainty: They can be combined with many other types of plots. For nearly any visualization we may have, we can add some indication of uncertainty by adding error bars. For example, we can show amounts with uncertainty by drawing a bar plot with error bars (Figure \@ref(fig:butterfat-bars)). This type of visualization is commonly used in scientific publications. We can also draw error bars along both the *x* and the *y* direction in a scatter plot (Figure \@ref(fig:median-age-income)). 

(ref:butterfat-bars) Mean butterfat contents in the milk of four cattle breeds. Error bars indicate +/- one standard error of the mean. Visualizations of this type are frequently seen in the scientific literature. While they are technically correct, they represent neither the variation within each category nor the uncertainty of the sample means particularly well. See Figure \@ref(fig:butterfat-densitites) for the variation in butterfat contents within individual breeds. Data Source: Canadian Record of Performance for Purebred Dairy Cattle

```{r butterfat-bars, fig.width = 5, fig.cap = '(ref:butterfat-bars)'}
cows %>% filter(breed != "Canadian") %>%
  mutate(breed = as.character(breed)) %>%
  group_by(breed) %>%
  summarize(
    mean = mean(butterfat),
    se = sd(butterfat)/sqrt(n())
  ) %>%
  mutate(breed = fct_reorder(factor(breed), desc(mean)))-> cow_means

ggplot(cow_means, aes(x = breed, y = mean, ymin = mean - 1.96*se, ymax = mean + 1.96*se)) +
  geom_col(fill = "#56B4E9", alpha = 0.7, width = 0.7) +
  #geom_linerange(size = 0.75) +
  geom_errorbar(size = 0.5, width = 0.1) +
  scale_x_discrete(
    labels = c("Jersey", "Guernsey", "Ayrshire", "Holstein-\nFriesian"),
    name = NULL
  ) +
  scale_y_continuous(
    name = "mean butterfat contents", expand = c(0, 0),
    labels = scales::percent_format(accuracy = 1, scale = 1)
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid() +
  theme(axis.line.x = element_blank())
```

(ref:median-age-income) Median income versus median age for 67 counties in Pennsylvania. Error bars represent 90% confidence intervals. Data source: 2015 Five-Year American Community Survey

```{r median-age-income, fig.width = 5, fig.asp = 3/4, fig.cap = '(ref:median-age-income)'}
filter(US_income_age, state == "Pennsylvania") %>%
  ggplot(aes(age, income)) + 
  geom_errorbarh(
    aes(xmin = age - age.moe, xmax = age + age.moe),
    height = 0,
    color = "#0072B2"
  ) +
  geom_errorbar(
    aes(ymin = income - income.moe, ymax = income + income.moe),
    width = 0,
    color = "#0072B2"
  ) +
  geom_point(color = "#0072B2", size = 0.7) +
  scale_x_continuous(name = "median age (years)") +
  scale_y_continuous(
    name = "median income (USD)",
    labels = scales::dollar
  ) +
  theme_dviz_open()
```

Let's return to the topic of frequentists and Bayesians. Frequentists assess uncertainty with confidence intervals, whereas Bayesians calculate *posterior distributions* and *credible intervals.* The Bayesian posterior distribution tells us how likely specific parameter estimates are given the input data. The credible interval indicates a range of values in which the parameter value is expected with a given probability, as calculated from the posterior distribution. For example, a 95% credible interval corresponds to the center 95% of the posterior distribution. The true parameter value has a 95% chance of lying in the 95% credible interval.

If you are not a statistician you may be surprised by my definition of a credible interval. You may have thought that it was actually the definition of a confidence interval. It is not. A Bayesian credible interval tells you about where the true parameter likely is and a frequentist confidence interval tells you about where the true parameter likely not is. While this distinction may seem like semantics, there are important conceptual differences between the two approaches. Under the Bayesian approach, you use the data and your prior knowledge about the system under study (called the *prior*) to calculate a probability distribution (the posterior) that tells you where you can expect the true parameter value to lie. By contrast, under the frequentist approach, you first make an assumption that you intend to disprove. This assumption is called the *null hypothesis*, and it is often simply the assumption that the parameter equals zero (e.g., there is no difference between two conditions). You then calculate the probability that random sampling would generate data similar to what was observed if the null hypothesis were true. The confidence interval is a representation of this probability. If a given confidence interval excludes the parameter value under the null hypothesis (i.e., the value zero), then you can reject the null hypothesis at that confidence level. Alternatively, you can think of a confidence interval as an interval that captures the true parameter value with the specified likelihood under repeated sampling (Figure \@ref(fig:ci-frequentist-expl)). Thus, if the true parameter value were zero, a 95% confidence interval would only exclude zero in 5% of the samples analyzed.

(ref:ci-frequentist-expl) Frequency interpretation of a confidence interval. Confidence intervals (CIs) are best understood in the context of repeated sampling. For each sample, a specific confidence interval either includes or excludes the true parameter, here the mean. However, if we sample repeatedly, then the confidence intervals (shown here are 68% confidence intervals, corresponding to sample mean +/- standard error) include the true mean approximately 68% of the time.

```{r ci-frequentist-expl, fig.asp = 1, fig.cap = '(ref:ci-frequentist-expl)'}
set.seed(85439)
df <- data.frame(y = rnorm(10000))
df_cont <- data.frame(
  x = c(seq(-3, 3, by = 0.02), -3)
) %>%
  mutate(
    y = dnorm(x),
    ny = y/max(y)
  )

spl <- sampler(10, 5, seed = 9382)

ggplot(df, aes(y = y)) +
  geom_polygon(data = df_cont, aes(1.5*ny + 10.5, x), fill = "lightblue") +
  geom_hline(yintercept = 0, linetype = 2, size = 0.5) +
  geom_point(data = spl, aes(x = .draw), size = 2,
             shape = 21, fill = "lightblue") +
  stat_summary(
    data = spl, aes(x = .draw, color = ifelse(stat(ymax) < 0 | stat(ymin) > 0, "outside", "inside")),
    geom = "errorbar", width = 0.2, size = 0.75,
    position = position_nudge(x = -.15)
  ) +
  stat_summary(
    data = spl, aes(x = .draw, color = ifelse(stat(ymax) < 0 | stat(ymin) > 0, "outside", "inside")),
    geom = "point", size = 2.5, position = position_nudge(x = -.15)
  ) +
  annotate(x = 11.7, y = 1, geom = "text", label = "population", family = dviz_font_family,
           size = 14/.pt, hjust = 0) +
  annotate(x = 11.1, y = 0.05, geom = "text", label = "mean", family = dviz_font_family,
           size = 12/.pt, hjust = 0) +
  annotate(x = 10.04, y = -Inf, geom = "text", label = "samples", family = dviz_font_family,
           size = 14/.pt, hjust = 0, vjust = 0) +
  annotate(x = 6.9, y = -0.02, geom = "text", label = "CI excludes true mean",
           family = dviz_font_family, size = 12/.pt, hjust = 1, vjust = 1) +
  annotate(x = 8.87, y = 0.23, geom = "text", label = "CI includes true mean",
           family = dviz_font_family, size = 12/.pt, hjust = 0, vjust = 1) +
  scale_color_manual(values = c("#009E73", "#D55E00"), guide = "none") +
  coord_flip() +
  scale_x_continuous(breaks = 1:10, labels = as.character(10:1), name = NULL, expand = c(0.02, 0)) +
  scale_y_continuous(breaks = NULL, name = NULL, limits = c(-3, 3), expand = c(0, 0)) +
  theme_dviz_hgrid() +
  theme(axis.text.y = element_text(vjust = 0.5))
```

To summarize, a Bayesian credible interval makes a statement about the true parameter value and a frequentist confidence interval makes a statement about the null hypothesis. In practice, however, Bayesian and frequentist estimates are often quite similar (Figure \@ref(fig:bayes-vs-ols)). Once conceptual advantage of the Bayesian approach is that it emphasizes thinking about the magnitude of an effect, whereas the frequentist thinking emphasizes a binary perspective of an effect either existing or not. 

(ref:bayes-vs-ols) Comparison of frequentist confidence intervals and Bayesian credible intervals for mean chocolate ratings. We see that both approaches yield similar but not exactly identical results. In particular, the Bayesian estimates display a small amount of shrinkage, which is an adjustment of the most extreme parameter estimates towards the overall mean. (Note how the Bayesian estimate for Switzerland is slightly moved to the left and the Bayesian estimate for Peru is slightly moved to the right relative to the respective frequentist estimates.) The frequentist estimates and confidence intervals shown here are identical to the results for 95% confidence shown in Figure \@ref(fig:mean-chocolate-ratings). 

```{r bayes-vs-ols, fig.width = 5*6/4.2, fig.asp = 0.6, fig.cap = '(ref:bayes-vs-ols)'}
linear_results <-
  lm(rating ~ location, data = cacao_small) %>% 
  emmeans(~ location) %>% 
  tidy() %>%
  mutate(model = "OLS")

# store these separately, will need them again
bayes_emmeans <- cacao_mcmc %>%
  emmeans(~ location) %>%
  gather_emmeans_draws()

bayes_results <- bayes_emmeans %>%
  mean_qi() %>%
  to_broom_names() %>%
  mutate(model = "Bayes")

bind_rows(linear_results, bayes_results) %>%
  mutate(model = factor(model, levels = c("Bayes", "OLS"))) %>%
  ggplot(aes(x = estimate, y = reorder(location, estimate), xmin = conf.low, xmax = conf.high, color = model)) +
  ggplot2::geom_errorbarh(position = ggstance::position_dodgev(height = .7), height = 0.2) +
  geom_point(position = ggstance::position_dodgev(height = .7), size = 2.5) +
  scale_x_continuous(
    limits = c(2.6, 3.6),
#    expand = c(0, 0),
    name = "mean rating"
  ) +
  scale_y_discrete(name = NULL) +
  scale_color_manual(
    name = NULL,
    values = c(
      OLS = "grey40",
      Bayes = desaturate(darken("#0072B2", .2), 0.1)
    ),
    breaks = c("OLS", "Bayes"),
    labels = c("95% confidence interval", "95% credible interval")
    #guide = guide_legend(direction = "horizontal")
  ) +
  coord_cartesian(clip = "off") +
  theme_dviz_hgrid(14, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.ticks.x = element_line(color = "black"),
    axis.title.x = element_text(hjust = 1),
    legend.position = c(1, 0.013),
    legend.justification = c(1, 0),
    legend.key.height = grid::unit(15, "pt"),
    legend.key.width = grid::unit(28, "pt"),
    legend.spacing.x = grid::unit(3.5, "pt"),
    legend.spacing.y = grid::unit(14, "pt"),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.spacing = grid::unit(0, "pt"),
    legend.title.align = 0.5
  )
```


```{block type='rmdtip', echo=TRUE}
A Bayesian credible interval answers the question: "Where do we expect the true parameter value to lie?" A frequentist confidence interval answers the question: "How certain are we that the true parameter value is not zero?" 
```

The central goal of Bayesian estimation is to obtain the posterior distribution. Therefore, Bayesians commonly visualize the entire distribution rather than simplifying it into a credible interval. In terms of data visualization, therefore, all the approaches to visualizing distributions discussed in Chapters \@ref(histograms-density-plots), \@ref(ecdf-qq), and \@ref(boxplots-violins) are applicable. Specifically, histograms, density plots, boxplots, violins, and ridgeline plots are all commonly used to visualize Bayesian posterior distributions. Since these approaches have been discussed at length in their specific chapters, I will here show only one example, using a ridgeline plot to show Bayesian posterior distributions of mean chocolate ratings (Figure \@ref(fig:bayes-ridgeline)). In this specific case, I have added shading under the curve to indicate defined regions of posterior probabilities. As alternative to shading, I could also have drawn quantile dotplots, or I could have added graded error bars underneath each distribution. Ridgeline plots with error bars underneath are called half eyes, and violin plots with error bars are called eye plots (Chapter \@ref(directory-uncertainty)).

(ref:bayes-ridgeline) Bayesian posterior distributions of mean chocolate bar ratings, shown as a ridgeline plot. The red dots represent the medians of each posterior distribution. Because it is difficult to convert a continuous distribution into specific confidence regions by eye, I have added shading under each curve to indicate the center 80%, 95%, and 99% of each posterior distribution.

```{r bayes-ridgeline, fig.width = 5*6/4.2, fig.asp = 0.6, fig.cap = '(ref:bayes-ridgeline)'}
bayes_emmeans %>%
  ggplot(aes(.value, reorder(location, .value))) +
  geom_density_ridges_gradient(
    aes(fill = stat(quantile)),
    quantiles = c(0.005, 0.025, 0.1, 0.9, 0.975, 0.995),
    calc_ecdf = TRUE,
    scale = 1.5,
    color = darken("#0072B2", .4),
    size = 0.3,
    rel_min_height = 0.005
  ) +
  geom_point(
    stat = ggstance:::StatSummaryh,
    fun.x = median,
    size = 2.5, color = "#D55E00"
  ) +
  scale_x_continuous(
#    expand = c(0, 0),
    name = "mean rating"
  ) +
  scale_y_discrete(name = NULL) +
  scale_fill_manual(
    name = "posterior prob.",
    values = c(
      "#00000000",
      "#81A7D6A0", # desaturate(lighten("#0072B2", .4), .3)
      "#508CC6A0", # desaturate(lighten("#0072B2", .2), .1)
      "#035B8FA0", # darken("#0072B2", .2)
      "#508CC6A0",
      "#81A7D6A0",
      "#00000000"
    ),
    breaks = c(4, 3, 2),
    labels = c("80%", "95%", "99%"),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom",
      override.aes = list(color = NA)
    )
  ) +
  coord_cartesian(xlim = c(2.6, 3.6), clip = "off") +
  theme_dviz_hgrid(14, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.ticks.x = element_line(color = "black"),
    axis.title.x = element_text(hjust = 1),
    legend.position = c(1, 0.013),
    legend.justification = c(1, 0),
    legend.key.height = grid::unit(14, "pt"),
    legend.key.width = grid::unit(35, "pt"),
    legend.spacing.x = grid::unit(7, "pt"),
    legend.spacing.y = grid::unit(3.5, "pt"),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.spacing = grid::unit(0, "pt"),
    legend.title.align = 0.5
  )
```


## Visualizing the uncertainty of curve fits {#uncertainty-curve-fits}

In Chapter \@ref(visualizing-trends), we discussed how to show a trend in a dataset by fitting a straight line or curve to the data. These trend estimates also have uncertainty, and it is customary to show the uncertainty in a trend line with a confidence band (Figure \@ref(fig:blue-jays-male-conf-band)). The confidence band provides us with a range of different fit lines that would be compatible with the data. When students encounter a confidence band for the first time, they are often surprised that even a perfectly straight line fit produces a confidence band that is curved. The reason for the curvature is that the straight line fit can move in two distinct directions: it can move up and down (i.e., have different intercepts), and it can rotate (i.e., have different slopes). We can visually show how the confidence band arises by drawing a set of alternative fit lines randomly generated from the posterior distribution of the fit parameters. This is done in Figure \@ref(fig:blue-jays-male-fitted-draws), which shows 15 randomly chosen alternative fits. We see that even though each line is perfectly straight, the combination of different slopes and intercepts of each line generates an overall shape that looks just like the confidence band.

(ref:blue-jays-male-conf-band) Head length versus body mass for male blue jays, as in Figure \@ref(fig:blue-jays-scatter-line). The straight blue line represents the best linear fit to the data, and the gray band around the line shows the uncertainty in the linear fit. The gray band represents a 95% confidence level. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-male-conf-band, fig.width = 5.5, fig.asp = 3/4, fig.cap = '(ref:blue-jays-male-conf-band)'}
blue_jays_male <- filter(blue_jays, KnownSex == "M")

fit <- gam(Head ~ Mass, data = blue_jays_male, method = "REML")

newdata <- data.frame(Mass = seq(59, 82, length.out = 100))

sample_df <- sample_outcomes(fit, newdata, 15, unconditional = TRUE)
ci_df <- confidence_band(fit, newdata, unconditional = TRUE)

ggplot(blue_jays_male, aes(Mass, Head)) + 
  geom_ribbon(data = ci_df, aes(ymin = lo, ymax = hi), fill="grey70", color = NA, alpha = 1/2) +
  geom_point(color = "grey60", size = 1.5) +
  #geom_line(data = sample_df, aes(group = .draw), color = "#0072B2", size = 0.3) +
  geom_line(data = ci_df, color = "#0072B2", size = 1) +
  scale_x_continuous(
    limits = c(59, 82),
    expand = c(0, 0),
    name = "body mass (g)") +
  scale_y_continuous(
    limits = c(52, 61),
    expand = c(0, 0),
    name = "head length (mm)"
  ) +
  theme_dviz_open()
```

(ref:blue-jays-male-fitted-draws) Head length versus body mass for male blue jays. In contrast to Figure  \@ref(fig:blue-jays-male-conf-band), the straight blue lines now represent equally likely alternative fits randomly drawn from the posterior distribution. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-male-fitted-draws, fig.width = 5.5, fig.asp = 3/4, fig.cap = '(ref:blue-jays-male-fitted-draws)'}

ggplot(blue_jays_male, aes(Mass, Head)) + 
  geom_ribbon(data = ci_df, aes(ymin = lo, ymax = hi), fill="grey70", color = NA, alpha = 1/2) +
  geom_point(color = "grey60", size = 1.5) +
  geom_line(data = sample_df, aes(group = .draw), color = "#0072B2", size = 0.3) +
  #geom_line(data = ci_df, color = "#0072B2", size = 1) +
  scale_x_continuous(
    limits = c(59, 82),
    expand = c(0, 0),
    name = "body mass (g)") +
  scale_y_continuous(
    limits = c(52, 61),
    expand = c(0, 0),
    name = "head length (mm)"
  ) +
  theme_dviz_open()

```

To draw a confidence band, we need to specify a confidence level, and just as we saw for error bars and posterior probabilities, it can be useful to highlight different levels of confidence. This leads us to the graded confidence band, which shows several confidence levels at once (Figure \@ref(fig:blue-jays-male-graded-conf-band)). A graded confidence band enhances the sense of uncertainty in the reader, and it forces the reader to confront the possibility that the data might support different alternative trend lines.

(ref:blue-jays-male-graded-conf-band) Head length versus body mass for male blue jays. As in the case of error bars, we can draw graded confidence bands to highlight the uncertainty in the estimate. Data source: Keith Tarvin, Oberlin College

```{r blue-jays-male-graded-conf-band, fig.width = 5.5, fig.asp = 3/4, fig.cap = '(ref:blue-jays-male-graded-conf-band)'}
ci_99_df <- confidence_band(fit, newdata, level = 0.99, unconditional = TRUE)
ci_80_df <- confidence_band(fit, newdata, level = 0.80, unconditional = TRUE)

ggplot(blue_jays_male, aes(Mass, Head)) + 
  geom_ribbon(
    data = ci_99_df,
    aes(ymin = lo, ymax = hi, fill = "99%"),
    color = NA
  ) +
  geom_ribbon(
    data = ci_df,
    aes(ymin = lo, ymax = hi, fill = "95%"),
    color = NA
  ) +
  geom_ribbon(
    data = ci_80_df,
    aes(ymin = lo, ymax = hi, fill = "80%"),
    color = NA
  ) +
  geom_point(color = "grey40", size = 1.5) +
  geom_line(data = ci_df, color = darken("#0072B2", .4), size = 1) +
  scale_fill_manual(
    name = "confidence level",
    values = c(
     `80%` = desaturate(lighten("#0072B2", .4), .3),
     `95%` = desaturate(lighten("#0072B2", .6), .3),
     `99%` = desaturate(lighten("#0072B2", .8), .3)
    ),
    guide = guide_legend(
      direction = "horizontal",
      title.position = "top",
      label.position = "bottom"
    )
  ) +
  scale_x_continuous(
    limits = c(59, 82),
    expand = c(0, 0),
    name = "body mass (g)"
  ) +
  scale_y_continuous(
    limits = c(52, 61),
    expand = c(0, 0),
    name = "head length (mm)"
  ) +
  theme_dviz_open() +
  theme(
    legend.position = c(1, .01),
    legend.justification = c(1, 0),
    legend.key.height = grid::unit(12, "pt"),
    legend.key.width = grid::unit(30, "pt"),
    legend.spacing.x = grid::unit(6, "pt"),
    legend.spacing.y = grid::unit(3, "pt"),
    legend.box.background = element_rect(fill = "white", color = NA),
    legend.box.spacing = grid::unit(0, "pt"),
    legend.title = element_text(size = 12, hjust = 0.5)
  )
```

We can also draw confidence bands for non-linear curve fits. Such confidence bands look nice but can be difficult to interpret (Figure \@ref(fig:mpg-uncertain)). If we look at Figure \@ref(fig:mpg-uncertain)a, we may think that the confidence band arises by moving the blue line up and down and maybe deforming it slightly. However, as Figure \@ref(fig:mpg-uncertain)b reveals, the confidence band represents a family of curves that are all quite a bit more wiggly than the overall best fit shown in part (a). This is a general principle of non-linear curve fits. Uncertainty corresponds not just to a movement of the curve up and down but also to increased wiggliness. 

(ref:mpg-uncertain) Fuel efficiency versus displacement, for 32 cars (1973–74 models). Each dot represents one car, and the smooth lines were obtained by fitting a cubic regression spline with 5 knots. (a) Best fit spline and confidence band. (b) Equally likely alternative fits drawn from the posterior distribution. Data source: Motor Trend, 1974.

```{r mpg-uncertain, fig.width = 5.5*6/4.2, fig.asp = 3/8, fig.cap='(ref:mpg-uncertain)'}
set.seed(8692282)

fit <- gam(mpg ~ s(disp, bs = 'cr', k = 5), data = mtcars, method = "REML")
#fit <- gam(mpg ~ s(disp, bs = 'ts', k = 6), data = mtcars, method = "REML")

newdata <- data.frame(disp = seq(min(mtcars$disp), max(mtcars$disp), length.out = 100))
sample_df <- sample_outcomes(fit, newdata, 10, unconditional = TRUE)
ci_df <- confidence_band(fit, newdata, unconditional = TRUE)

cars_base <- ggplot(mtcars, aes(x = disp, y = mpg)) + 
  scale_x_continuous(
    name = "displacement (cu. in.)",
    expand = c(0.01, 0)
  ) +
  ylab("fuel efficiency (mpg)") +
  theme_dviz_grid(12) +
  theme(plot.margin = margin(6, 1.5, 3, 1.5))

p1 <- cars_base +
  geom_ribbon(data = ci_df, aes(ymin = lo, ymax = hi), fill="grey70", color = NA, alpha = 1/2) +
  geom_point(color = "grey60") +
  geom_line(data = ci_df, color = "#0072B2", size = 1)

p2 <- cars_base +
  geom_ribbon(data = ci_df, aes(ymin = lo, ymax = hi), fill="grey70", color = NA, alpha = 1/2) +
  geom_point(color = "grey60") +
  geom_line(data = sample_df, aes(group = .draw), color = "#0072B2", size = 0.3)

plot_grid(
  p1, NULL, p2, align = 'hv',
  nrow = 1,
  labels = c("a", "", "b"),
  rel_widths = c(1, .02, 1)
)

```



## Hypothetical outcome plots

All static visualizations of uncertainty suffer from the problem that viewers may interpret some aspect of the uncertainty visualization as a deterministic feature of the data (deterministic construal error). We can avoid this problem by visualizing uncertainty through animation, by cycling through a number of different but equally likely plots. This kind of a visualization is called a hypothetical outcome plot [@Hullman_et_al_2015] or HOP. While HOPs are not possible in a print medium, they can be very effective in online settings where animated visualizations can be provided in the form of GIFs or MP4 videos. HOPs can also work well in the context of an oral presentation.

To illustrate the concept of a HOP, let's go back once more to chocolate bar ratings. When you are standing in the grocery store thinking about buying some chocolate, you probably don't care about the mean flavor rating and associated uncertainty for certain groups of chocolate bars. Instead, you might want to know the answer to a simpler question, such as: If I randomly pick up a Canadian and a U.S. manufactured chocolate bar, which one of the two should I expect to taste better? To arrive at  an answer to this question, we could randomly select a Canadian and a U.S. bar from the dataset, compare their ratings, record the outcome, and then repeat this process many times. If we did this, we would find that in approximately 53% of the cases the Canadian bar will be ranked higher, and in 47% of the cases either the U.S. bar is ranked higher or the two bars are tied. We can show this process visually by cycling between several of these random draws and showing the relative ranking of the two bars for each draw (Figure \@ref(fig:chocolate-HOP-static)/Figure \@ref(fig:chocolate-HOP-animated)).

(ref:chocolate-HOP-static) (for print edition) Schematic of a hypothetical outcome plot for chocolate bar ratings of Canadian and U.S. manufactured bars. Each vertical green bar represents the rating for one bar, and each panel shows a comparison of two randomly chosen bars, one each from a Canadian manufacturer and a U.S. manufacturer. In an actual hypothetical outcome plot, the display would cycle between the distinct plot panels instead of showing them side-by-side.

```{r chocolate-HOP-static, fig.width = 5.5*6/4.2, fig.asp = 0.6, fig.cap = '(ref:chocolate-HOP-static)'}
cacao %>% 
  filter(location %in% c("Canada", "U.S.A.")) %>%
  mutate(location = fct_recode(location, US = "U.S.A.")) -> cacao_US_CA

set.seed(49323)
n <- 9
CA_ratings <- sample_n(filter(cacao_US_CA, location == "Canada"), n, replace = TRUE) %>%
  mutate(.draw = 1:n())
US_ratings <- sample_n(filter(cacao_US_CA, location == "US"), n, replace = TRUE) %>%
  mutate(.draw = 1:n())

sample_df <- rbind(CA_ratings, US_ratings) %>%
  mutate(location = fct_relevel(location, "US", "Canada"))

ggplot(sample_df) +
  geom_segment(
    aes(
      x = rating, xend = rating,
      y = as.integer(location) - 0.35,
      yend = as.integer(location) + 0.35
    ),
    size = 1.5, color = darken("#009E73", .3)
  ) +
  scale_x_continuous(
    limits = c(1.95, 4.1),
    expand = c(0, 0),
    name = "chocolate flavor rating"
  ) +
  scale_y_discrete(
    name = NULL,
    limits = c("US", "Canada"),
    expand = c(0, 0.25)
  ) +
  theme_dviz_hgrid(12, rel_small = 1) +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.ticks.x = element_line(color = "black"),
    #axis.title.x = element_text(hjust = 1),
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank(),
    strip.text = element_blank()
  ) + 
  panel_border() +
  facet_wrap(~.draw, scales = "free_x")
```

(ref:chocolate-HOP-animated) (for online edition) Hypothetical outcome plot for chocolate bar ratings of Canadian and U.S. manufactured bars. Each vertical green bar represents the rating for one bar. The animation cycles through different cases of two randomly chosen bars, one each from a Canadian manufacturer and a U.S. manufacturer.

```{r chocolate-HOP-animated, eval = animate, gganimate = list(nframes = 200), fig.asp = 0.6, fig.retina = 1, fig.cap = '(ref:chocolate-HOP-animated)'}
library(gganimate)

set.seed(49316)
n <- 30

CA_ratings <- sample_n(filter(cacao_US_CA, location == "Canada"), n, replace = TRUE) %>%
  mutate(.draw = 1:n())
US_ratings <- sample_n(filter(cacao_US_CA, location == "US"), n, replace = TRUE) %>%
  mutate(.draw = 1:n())

sample_df <- rbind(CA_ratings, US_ratings) %>%
  mutate(location = fct_relevel(location, "US", "Canada"))

ggplot(sample_df) +
  geom_segment(
    aes(
      x = rating, xend = rating,
      y = as.integer(location) - 0.35,
      yend = as.integer(location) + 0.35,
      group = .draw
    ),
    size = 1.5, color = darken("#009E73", .3)
  ) +
  scale_x_continuous(
    limits = c(1.95, 4.1),
    expand = c(0, 0),
    name = "chocolate flavor rating"
  ) +
  scale_y_discrete(
    name = NULL,
    limits = c("US", "Canada"),
    expand = c(0, 0.25)
  ) +
  theme_dviz_hgrid() +
  theme(
    axis.line.x = element_line(color = "black"),
    axis.ticks.x = element_line(color = "black"),
    axis.title.x = element_text(hjust = 1),
    axis.line.y = element_blank(),
    axis.ticks.y = element_blank(),
    strip.text = element_blank()
  ) + 
  #panel_border() +
  transition_states(.draw, 1, 2)
```

As a second example, consider the variation in shapes among equally probable trendlines in Figure \@ref(fig:mpg-uncertain)b. Because all trendlines are plotted on top of one another, we primarily perceive the overall area that is covered by trendlines, which is similar to the confidence band. Perceiving individual trendlines is difficult. By turning this figure into a HOP, we can highlight individual trendlines one at a time (Figure \@ref(fig:mpg-uncertain-HOP-static)/Figure \@ref(fig:mpg-uncertain-HOP-animated)).

(ref:mpg-uncertain-HOP-static) (for print edition) Schematic of a hypothetical outcome plot for fuel efficiency versus displacement. Each dot represents one car, and the smooth lines were obtained by fitting a cubic regression spline with 5 knots. Each line in each panel represents one alternative fit outcome, drawn from the posterior distribution of the fit parameters. In an actual hypothetical outcome plot, the display would cycle between the distinct plot panels instead of showing them side-by-side.


```{r mpg-uncertain-HOP-static, fig.width = 5.5*6/4.2, fig.asp = 2/4, fig.cap = '(ref:mpg-uncertain-HOP-static)'}
#set.seed(8692282)
set.seed(8692276)

fit <- gam(mpg ~ s(disp, bs = 'cr', k = 5), data = mtcars, method = "REML")

newdata <- data.frame(disp = seq(min(mtcars$disp), max(mtcars$disp), length.out = 100))
sample_df <- sample_outcomes(fit, newdata, 6, unconditional = TRUE)

ggplot(mtcars, aes(x = disp, y = mpg)) + 
  scale_x_continuous(
    name = "displacement (cu. in.)",
    expand = c(0.03, 0)
  ) +
  scale_y_continuous(
    name = "fuel efficiency (mpg)",
    limits = c(8.5, 35),
    expand = c(0, 0)
  ) +
  geom_point(color = "grey60") +
  geom_line(data = sample_df, color = "#0072B2", size = 0.5) +
  facet_wrap(~.draw, scales = "free") + 
  theme_dviz_grid(12) +
  theme(
    strip.text = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = unit(0, "pt"),
    plot.margin = margin(6, 1.5, 3, 1.5)
  ) +
  panel_border()
```

(ref:mpg-uncertain-HOP-animated) (for online edition) Hypothetical outcome plot for fuel efficiency versus displacement. Each dot represents one car, and the smooth lines were obtained by fitting a cubic regression spline with 5 knots. The animation cycles through different alternative fit outcomes drawn from the posterior distribution of the fit parameters.

```{r mpg-uncertain-HOP-animated, eval = animate, fig.width = 5.5, fig.asp = 3/4, fig.retina = 1, fig.cap = '(ref:mpg-uncertain-HOP-animated)'}
library(gganimate)

#set.seed(8692282)
set.seed(8692276)

fit <- gam(mpg ~ s(disp, bs = 'cr', k = 5), data = mtcars, method = "REML")

newdata <- data.frame(disp = seq(min(mtcars$disp), max(mtcars$disp), length.out = 100))
sample_df <- sample_outcomes(fit, newdata, 20, unconditional = TRUE)

ggplot(mtcars, aes(x = disp, y = mpg)) + 
  scale_x_continuous(
    name = "displacement (cu. in.)",
    expand = c(0.03, 0)
  ) +
  scale_y_continuous(
    name = "fuel efficiency (mpg)",
    limits = c(8.5, 35),
    expand = c(0, 0)
  ) +
  geom_point(color = "grey60") +
  geom_line(data = sample_df, aes(group = .draw), color = "#0072B2", size = 0.5) +
  theme_dviz_grid() +
  theme(
    strip.text = element_blank(),
    axis.ticks = element_blank(),
    axis.ticks.length = unit(0, "pt"),
    plot.margin = margin(7, 1.5, 3.5, 1.5)
  ) +
  panel_border() +
  transition_states(.draw, 0, 1)
```

When preparing a HOP, you may wonder whether it is better to make a hard switch between different outcomes (as in a slide projector) or rather smoothly animate from one outcome to the next (e.g., slowly deform the trendline for one outcome until it looks like the trendline for another outcome). While this is to some extent an open question that continues to be researched, some evidence indicates that smooth transitions make it harder to judge about the probabilities represented [@Kale_et_al_2018]. If you consider animating between outcomes, you may want to at least make these animations very fast, or choose an animation style where outcomes fade in and out rather than deform from one to the other. 

There is one critical aspect we need to pay attention to when preparing a HOP: We need to make sure that the outcomes we do show are representative of the true distribution of possible outcomes. Otherwise, our HOP could be rather misleading. For example, going back to the case of chocolate ratings, if I randomly selected ten outcome pairs of chocolate bars and among those the U.S. bar was rated higher than the Canadian bar in seven cases, then the HOP would erroneously create the impression that U.S. bars tend to be higher rated than Canadian bars. We can prevent this issue either by choosing a very large number of outcomes, so sampling biases are unlikely, or by verifying in some form that the outcomes that are shown are appropriate. When making Figure \@ref(fig:chocolate-HOP-static)/Figure \@ref(fig:chocolate-HOP-animated), I verified that the number of times the Canadian bar was shown winning was close to the true percentage of 53%.

<!--chapter:end:visualizing_uncertainty.Rmd-->

