R and large data sets

Richard J. Telford

University of Bergen

R and Memory

R loads objects into memory.

  • Data must be loaded into RAM, even if only part of it is needed.
  • Data must be stored in RAM, even if it is not used.
  • RAM is limited, data sets can be larger than the available RAM.

Large objects can cause the system to run out of memory.

Bad things happen if you run out of memory.

Buy a better computer

  • RAM is expensive
  • Could never afford enough to deal with the largest datasets

duckdb

Works with larger than memory data

Data processed on disk

duckplyr is a drop-in replacement for dplyr

Will read csv files and other formats

Using duckplyr

library(duckplyr)

penguins_file <- system.file("extdata/penguins.csv",
                             package = "palmerpenguins") 

# import from file
peng_duck <- read_csv_duckdb(path = penguins_file)

peng_duck
# A duckplyr data frame: 8 variables
  species island    bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
  <chr>   <chr>     <chr>          <chr>         <chr>             <chr>      
1 Adelie  Torgersen 39.1           18.7          181               3750       
2 Adelie  Torgersen 39.5           17.4          186               3800       
3 Adelie  Torgersen 40.3           18            195               3250       
4 Adelie  Torgersen NA             NA            NA                NA         
5 Adelie  Torgersen 36.7           19.3          193               3450       
# ℹ more rows
# ℹ 2 more variables: sex <chr>, year <dbl>

Manipulate with dplyr verbs

peng_duck |> 
  mutate(body_mass_g = as.numeric(body_mass_g)) |> 
  summarise(mean_mass = mean(body_mass_g, na.rm = TRUE), .by = species) |> 
  collect()
# A tibble: 3 × 2
  species   mean_mass
* <chr>         <dbl>
1 Adelie        3701.
2 Chinstrap     3733.
3 Gentoo        5076.

duckplyr

Supports most dplyr function.

Will fall-back to dplyr if it has to.

  • group_by() fails - use .by argument instead

Supports many functions within dplyr verbs - each needs a translation

  • sum(), mean(), n()

parquet files

Alternative to csv files

Compressed - smaller file size

Import with duckplyr or arrow packages

Databases

Useful for large, complex datasets.

  site  lat long elevation     species   family functionGroup abundance
1    1 60.5  5.1      2300  Poa alpina  Poaceae     graminoid         5
2    2 62.3  5.7      1500  Poa alpina  Poaceae     graminoid         7
3    1 60.5  5.1      2300 Geum rivale Rosaceae          forb        14
4    2 62.3  5.7      1500 Geum rivale Rosaceae          forb         3

Split into several tables

Sites

  siteID  lat long elevation
1      1 60.5  5.1      2300
2      2 62.3  5.7      1500

Species

  speciesID     species   family functionGroup
1    PoaAlp  Poa alpina  Poaceae     graminoid
2    GeuRiv Geum rivale Rosaceae          forb

Occurrences

  siteID speciesID abundance
1      1    PoaAlp         5
2      2    PoaAlp         7
3      1    GeuRiv        14
4      2    GeuRiv         3

Databases

Separate tables

  • Primary keys - unique

  • Foreign keys

Stand alone database - SQLite, duckDB

Client-server databases - postgres

Connect with DBI

dbplyr makes a bridge to the database - no need to write your own SQL

DBI + dbplyr

con <- DBI::dbConnect(RSQLite::SQLite(), here::here("database/seedclim.sqlite"))

sites_blocks_plots <- tbl(con, "sites") |> 
  inner_join(tbl(con, "blocks"), by = "siteID") |> 
  inner_join(tbl(con, "plots"), by = "blockID") |>  
  select(siteID, annual_precipitation_gridded, summer_temperature_gridded,
          blockID, plotID) |> 
  collect()
DBI::dbDisconnect(con)