10 min read

List Iteration-Part-02


iterating over a list in R
Iterating and extracting from a list

View raw source for this post

Summary

This post is about how to iterate over and extract from a list. It begins with the function lapply() since many people are familiar with it and then expounds on functions in the purrr package.

Table of Contents

Overview

This post addresses a common task of iterating over a list. Standard iteration relies heavily on the “apply” family of functions in the base package. The method for iterating over a list in the base package is base::lapply() and the newer, tidyverse method is purrr::map(). The two are similar but purrr allows for more flexibility and its syntax is more concise and consistent.

The post will show how to iterate over a list, convert a list to a vector, extract elements from a list by both name and position, flatten a list, transpose a list and simplify a list. The emphasis and goal is to incorporate the newer purrr functions into future workflows. (After package install, see ?purrr::map for more information).

One quote to remember as you begin this tutorial is from “R for Data Science”: “[Y]ou should never feel bad about using a for loop instead of a map function. The map functions are a step up a tower of abstraction, and it can take a long time to get your head around how they work.”

Key terms

In addition to the key terms in List Basics-Part 01, the following terms are also useful:

  • a named function is one of the form:
addition_function <- function(x) {
    x + 1
}
  • lambda or anonymous functions are not important enough to be named but instead are created “on-the-fly” like function(.x) .x + 1. Lambda/anonymous functions are considered to be verbose in R.

  • a purrr-style lambda function, formula is a convenience or shortcut function taking the form ~ .x + 1. It is available for use in the purrr package. This form should be reserved for short and simple functions.

map() distinguished from lapply()

While map() and lapply() are similar , the map() function accepts a purrr-style lambda function where the lapply() function does not.

The lapply() method

# apply function
list(letters[5:9]) |> lapply(paste0, "<==")
[[1]]
[1] "e<==" "f<==" "g<==" "h<==" "i<=="
# apply anonymous function
list(letters[5:9]) |> lapply(function(x) paste0(x, "<=="))
[[1]]
[1] "e<==" "f<==" "g<==" "h<==" "i<=="
# apply purr-style lambda function
list(letters[5:9]) |> lapply(~ paste0(x, "<=="))
Error in match.fun(FUN): '~paste0(x, "<==")' is not a function, character or symbol

a purrr-style lambda function will not work in lapply().

The map() method

A basic template for map() usage is map(YOUR_LIST, YOUR_FUNCTION) or if you’re piping, it would be modified to YOUR_LIST |> map(YOUR_FUNCTION).

# map function
list(letters[5:9]) |> map(paste0, "<==")
[[1]]
[1] "e<==" "f<==" "g<==" "h<==" "i<=="
# map anonymous function
list(letters[5:9]) |> map(function(x) paste0(x, "<=="))
[[1]]
[1] "e<==" "f<==" "g<==" "h<==" "i<=="
# map purr-style lambda function
list(letters[5:9]) |> map(~ paste0(.x, "<=="))
[[1]]
[1] "e<==" "f<==" "g<==" "h<==" "i<=="

Having shown the similarity between map() and lapply(), the balance of the post is dedicated to the purrr package.

List to vector

map() always returns a list. After iterating over a list with the map() function, the result can be returned as a user-defined data type, using the map_lgl (logical), map_chr() (character), map_int() (integer), map_dbl() (double), map_raw() (raw), map_df() (data frame), map_dfr() (data frame), or map_dfc() (data frame) functions.

Output – Double

1:3 |>
    map(~ rnorm(.x, n = 100, sd = .5)) |> # output a list
    map_dbl(mean) # output an atomic vector
[1] 1.074261 2.117823 2.942117

Output – Character

goldilocks <- list(mamma_bear = "bed_1", papa_bear = "bed_2", baby_bear = "bed_3")
goldilocks |>
    map_chr(~ paste0("Who's been sleeping in ", .x, "?"))
                     mamma_bear                       papa_bear 
"Who's been sleeping in bed_1?" "Who's been sleeping in bed_2?" 
                      baby_bear 
"Who's been sleeping in bed_3?" 

Output – Dataframe

The “c” in map_dfc() stands for “column”, the “r” in map_dfr() stands for “row”. (Conversion of lists to data frames and vice versa is the topic of the next post).

list(a = c(1, 2), b = c(3, 4)) |>
    map_dfc(~ as.data.frame(.)) |>
    setNames(c("a", "b"))
  a b
1 1 3
2 2 4

Extract element(s) from list

Single nested list

# create
l1 <- list(list(a = 1), list(a = NULL, b = 2), list(b = 3))
l1
[[1]]
[[1]]$a
[1] 1


[[2]]
[[2]]$a
NULL

[[2]]$b
[1] 2


[[3]]
[[3]]$b
[1] 3

By position

l1 |> map_dbl(2, .default = NA)
[1] NA  2 NA

By name

map(l1, "a", .default = "???")
[[1]]
[1] 1

[[2]]
[1] "???"

[[3]]
[1] "???"
l1 |> map("b", .default = NA)
[[1]]
[1] NA

[[2]]
[1] 2

[[3]]
[1] 3

Multi nested list

l2 <- list(
    list(num = 1:3, let = letters[1:3]),
    list(num = 101:103, let = letters[4:6]),
    list()
)
l2
[[1]]
[[1]]$num
[1] 1 2 3

[[1]]$let
[1] "a" "b" "c"


[[2]]
[[2]]$num
[1] 101 102 103

[[2]]$let
[1] "d" "e" "f"


[[3]]
list()

By position

l2 |> map(c(1, 3))
[[1]]
[1] 3

[[2]]
[1] 103

[[3]]
NULL
l2 |> map(c(2, 2))
[[1]]
[1] "b"

[[2]]
[1] "e"

[[3]]
NULL

By name & position

l2 |> map(list("num", 3))
[[1]]
[1] 3

[[2]]
[1] 103

[[3]]
NULL
l2 |> map(list("let", 1))
[[1]]
[1] "a"

[[2]]
[1] "d"

[[3]]
NULL
l2 |> map_int(list("num", 3), .default = NA)
[1]   3 103  NA

Flatten list

flatten() is similar to unlist(), but it only removes a single layer and returns the same type as the input.

l2 |> flatten()
$num
[1] 1 2 3

$let
[1] "a" "b" "c"

$num
[1] 101 102 103

$let
[1] "d" "e" "f"

Also, a list can be subset and then flattened with map().

# flatten with map
l2 |>
    map(list("let", 1)) |>
    flatten_chr()
[1] "a" "d"

Transpose list

transpose() turns a list-of-lists “inside-out”; it turns a pair of lists into a list of pairs, or a list of pairs into pair of lists.

# create list
x <- rerun(3, x = runif(1), y = runif(5))
x
[[1]]
[[1]]$x
[1] 0.09431897

[[1]]$y
[1] 0.1395099 0.8937823 0.7481023 0.8494439 0.2458480


[[2]]
[[2]]$x
[1] 0.8885299

[[2]]$y
[1] 0.5236346 0.2654153 0.6799881 0.5912315 0.8131719


[[3]]
[[3]]$x
[1] 0.1855539

[[3]]$y
[1] 0.66182346 0.54670796 0.22834060 0.02755153 0.35516621
x |> transpose()
$x
$x[[1]]
[1] 0.09431897

$x[[2]]
[1] 0.8885299

$x[[3]]
[1] 0.1855539


$y
$y[[1]]
[1] 0.1395099 0.8937823 0.7481023 0.8494439 0.2458480

$y[[2]]
[1] 0.5236346 0.2654153 0.6799881 0.5912315 0.8131719

$y[[3]]
[1] 0.66182346 0.54670796 0.22834060 0.02755153 0.35516621

Simplify list

Use simplify_all() to reduce to atomic vectors where possible.

x <- list(list(a = 1, b = 2), list(a = 3, b = 4), list(a = 5, b = 6))
x
[[1]]
[[1]]$a
[1] 1

[[1]]$b
[1] 2


[[2]]
[[2]]$a
[1] 3

[[2]]$b
[1] 4


[[3]]
[[3]]$a
[1] 5

[[3]]$b
[1] 6

Then, transpose the above list.

x |> transpose()
$a
$a[[1]]
[1] 1

$a[[2]]
[1] 3

$a[[3]]
[1] 5


$b
$b[[1]]
[1] 2

$b[[2]]
[1] 4

$b[[3]]
[1] 6

Finally, simplify the list.

## helpful
x |>
    transpose() |>
    simplify_all()
$a
[1] 1 3 5

$b
[1] 2 4 6

Conclusion

You should now know how to iterate over a list, convert a list to a vector, extract elements from a list by both name and position, flatten, transpose, and simplify a list. These are only some of the highlights within the purrr package and there are many more variants and functions available. Rstudio published a “must-have” cheat sheet for purrr.

References

[1]
R Core Team, R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing, 2022 [Online]. Available: https://www.R-project.org/
[2]
Y. Xie, C. Dervieux, and A. Presmanes Hill, Blogdown: Create blogs and websites with r markdown. 2022 [Online]. Available: https://CRAN.R-project.org/package=blogdown
[3]
L. Henry and H. Wickham, Purrr: Functional programming tools. 2020 [Online]. Available: https://CRAN.R-project.org/package=purrr

Disclaimer

The views, analysis and conclusions presented within this paper represent the author’s alone and not of any other person, organization or government entity. While I have made every reasonable effort to ensure that the information in this article was correct, it will nonetheless contain errors, inaccuracies and inconsistencies. It is a working paper subject to revision without notice as additional information becomes available. Any liability is disclaimed as to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause. The author(s) received no financial support for the research, authorship, and/or publication of this article.

Reproducibility

─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.1.3 (2022-03-10)
 os       macOS Big Sur/Monterey 10.16
 system   x86_64, darwin17.0
 ui       X11
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       America/Chicago
 date     2022-04-21
 pandoc   2.14.1 @ /usr/local/bin/ (via rmarkdown)

─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version    date (UTC) lib source
 assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.1.0)
 blogdown    * 1.9        2022-03-28 [1] CRAN (R 4.1.2)
 bookdown      0.25       2022-03-16 [1] CRAN (R 4.1.2)
 brio          1.1.3      2021-11-30 [1] CRAN (R 4.1.0)
 bslib         0.3.1.9000 2022-03-04 [1] Github (rstudio/bslib@888fbe0)
 cachem        1.0.6      2021-08-19 [1] CRAN (R 4.1.0)
 callr         3.7.0      2021-04-20 [1] CRAN (R 4.1.0)
 cli           3.2.0      2022-02-14 [1] CRAN (R 4.1.2)
 colorspace    2.0-3      2022-02-21 [1] CRAN (R 4.1.2)
 crayon        1.5.1      2022-03-26 [1] CRAN (R 4.1.0)
 DBI           1.1.2      2021-12-20 [1] CRAN (R 4.1.0)
 desc          1.4.1      2022-03-06 [1] CRAN (R 4.1.2)
 devtools    * 2.4.3      2021-11-30 [1] CRAN (R 4.1.0)
 digest        0.6.29     2021-12-01 [1] CRAN (R 4.1.0)
 dplyr         1.0.8      2022-02-08 [1] CRAN (R 4.1.2)
 ellipsis      0.3.2      2021-04-29 [1] CRAN (R 4.1.0)
 evaluate      0.15       2022-02-18 [1] CRAN (R 4.1.2)
 fansi         1.0.3      2022-03-24 [1] CRAN (R 4.1.2)
 fastmap       1.1.0      2021-01-25 [1] CRAN (R 4.1.0)
 fs            1.5.2      2021-12-08 [1] CRAN (R 4.1.0)
 generics      0.1.2      2022-01-31 [1] CRAN (R 4.1.2)
 ggplot2       3.3.5      2021-06-25 [1] CRAN (R 4.1.0)
 ggthemes    * 4.2.4      2021-01-20 [1] CRAN (R 4.1.0)
 glue          1.6.2      2022-02-24 [1] CRAN (R 4.1.2)
 gtable        0.3.0      2019-03-25 [1] CRAN (R 4.1.0)
 htmltools     0.5.2      2021-08-25 [1] CRAN (R 4.1.0)
 jquerylib     0.1.4      2021-04-26 [1] CRAN (R 4.1.0)
 jsonlite      1.8.0      2022-02-22 [1] CRAN (R 4.1.2)
 knitr         1.38       2022-03-25 [1] CRAN (R 4.1.0)
 lifecycle     1.0.1      2021-09-24 [1] CRAN (R 4.1.0)
 magrittr      2.0.3      2022-03-30 [1] CRAN (R 4.1.2)
 memoise       2.0.1      2021-11-26 [1] CRAN (R 4.1.0)
 munsell       0.5.0.9000 2021-10-19 [1] Github (cwickham/munsell@e539541)
 pillar        1.7.0      2022-02-01 [1] CRAN (R 4.1.2)
 pkgbuild      1.3.1      2021-12-20 [1] CRAN (R 4.1.0)
 pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.1.0)
 pkgload       1.2.4      2021-11-30 [1] CRAN (R 4.1.0)
 prettyunits   1.1.1      2020-01-24 [1] CRAN (R 4.1.0)
 processx      3.5.3      2022-03-25 [1] CRAN (R 4.1.0)
 ps            1.6.0      2021-02-28 [1] CRAN (R 4.1.0)
 purrr       * 0.3.4      2020-04-17 [1] CRAN (R 4.1.0)
 R6            2.5.1      2021-08-19 [1] CRAN (R 4.1.0)
 remotes       2.4.2      2021-11-30 [1] CRAN (R 4.1.0)
 rlang         1.0.2      2022-03-04 [1] CRAN (R 4.1.2)
 rmarkdown     2.13       2022-03-10 [1] CRAN (R 4.1.2)
 rprojroot     2.0.3      2022-04-02 [1] CRAN (R 4.1.0)
 rstudioapi    0.13       2020-11-12 [1] CRAN (R 4.1.0)
 sass          0.4.1      2022-03-23 [1] CRAN (R 4.1.2)
 scales        1.1.1      2020-05-11 [1] CRAN (R 4.1.0)
 sessioninfo   1.2.2      2021-12-06 [1] CRAN (R 4.1.0)
 stringi       1.7.6      2021-11-29 [1] CRAN (R 4.1.0)
 stringr       1.4.0      2019-02-10 [1] CRAN (R 4.1.0)
 testthat      3.1.3      2022-03-29 [1] CRAN (R 4.1.2)
 tibble        3.1.6      2021-11-07 [1] CRAN (R 4.1.0)
 tidyselect    1.1.2      2022-02-21 [1] CRAN (R 4.1.2)
 usethis     * 2.1.5      2021-12-09 [1] CRAN (R 4.1.0)
 utf8          1.2.2      2021-07-24 [1] CRAN (R 4.1.0)
 vctrs         0.4.0      2022-03-30 [1] CRAN (R 4.1.2)
 withr         2.5.0      2022-03-03 [1] CRAN (R 4.1.0)
 xfun          0.30       2022-03-02 [1] CRAN (R 4.1.2)
 yaml          2.3.5      2022-02-21 [1] CRAN (R 4.1.2)

 [1] /Library/Frameworks/R.framework/Versions/4.1/Resources/library

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────