Managing the “data life-cycle” within and beyond a project.
Data cleaning is the process of removing incorrect, duplicate, or otherwise erroneous data from a dataset
Data wrangling changing the format to make it more useful for your analysis
Why? What could go wrong?
Error in `select()`:
! unused arguments (species, body_mass)
Turns namespace conflicts into errors
Error:
! [conflicted] select found in 2 packages.
Either pick the one you want with `::`:
• MASS::select
• dplyr::select
Or declare a preference with `conflicts_prefer()`:
• `conflicts_prefer(MASS::select)`
• `conflicts_prefer(dplyr::select)`
[1] "studyName" "Sample Number" "Species"
[4] "Region" "Island" "Stage"
[7] "Individual ID" "Clutch Completion" "Date Egg"
[10] "Culmen Length (mm)" "Culmen Depth (mm)" "Flipper Length (mm)"
[13] "Body Mass (g)" "Sex" "Delta 15 N (o/oo)"
[16] "Delta 13 C (o/oo)" "Comments"
[1] "study_name" "sample_number" "species"
[4] "region" "island" "stage"
[7] "individual_id" "clutch_completion" "date_egg"
[10] "culmen_length_mm" "culmen_depth_mm" "flipper_length_mm"
[13] "body_mass_g" "sex" "delta_15_n_o_oo"
[16] "delta_13_c_o_oo" "comments"
Rows: 344
Columns: 17
$ study_name <chr> "PAL0708", "PAL0708", "PAL0708", "PAL0708", "PAL0708…
$ sample_number <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 1…
$ species <chr> "Adelie Penguin (Pygoscelis adeliae)", "Adelie Pengu…
$ region <chr> "Anvers", "Anvers", "Anvers", "Anvers", "Anvers", "A…
$ island <chr> "Torgersen", "Torgersen", "Torgersen", "Torgersen", …
$ stage <chr> "Adult, 1 Egg Stage", "Adult, 1 Egg Stage", "Adult, …
$ individual_id <chr> "N1A1", "N1A2", "N2A1", "N2A2", "N3A1", "N3A2", "N4A…
$ clutch_completion <chr> "Yes", "Yes", "Yes", "Yes", "Yes", "Yes", "No", "No"…
$ date_egg <date> 2007-11-11, 2007-11-11, 2007-11-16, 2007-11-16, 200…
$ culmen_length_mm <dbl> 39.1, 39.5, 40.3, NA, 36.7, 39.3, 38.9, 39.2, 34.1, …
$ culmen_depth_mm <dbl> 18.7, 17.4, 18.0, NA, 19.3, 20.6, 17.8, 19.6, 18.1, …
$ flipper_length_mm <dbl> 181, 186, 195, NA, 193, 190, 181, 195, 193, 190, 186…
$ body_mass_g <dbl> 3750, 3800, 3250, NA, 3450, 3650, 3625, 4675, 3475, …
$ sex <chr> "MALE", "FEMALE", "FEMALE", NA, "FEMALE", "MALE", "F…
$ delta_15_n_o_oo <dbl> NA, 8.94956, 8.36821, NA, 8.76651, 8.66496, 9.18718,…
$ delta_13_c_o_oo <dbl> NA, -24.69454, -25.33302, NA, -25.32426, -25.29805, …
$ comments <chr> "Not enough blood for isotopes.", NA, NA, "Adult not…
Also skimr package for more detailed summaries
# A tibble: 4 × 2
Day Person
<chr> <chr>
1 Monday Becks
2 Tuesday Amy
3 Wednesday Matt
4 Wednesday Matt
# A tibble: 3 × 2
Day Person
<chr> <chr>
1 Monday Becks
2 Tuesday Amy
3 Wednesday Matt
dplyr & comutate() with
lubridate package for datesstringr package for character vectorsforcats package for factor vectorsreplace_values(), replace_when() for updating vectorsMissing data is normally a problem. Typically as ecologists we sweep missing data under the carpet by using a “complete case” approach to data analysis.
If you have ever written some code like this:
you are removing missing data (NAs) from your dataset.
By throwing away potentially useful data (only including those rows without a NA in them) you
There are three broad categories of missing data:
MCAR - missingness is not related to any measured or unmeasured variables
MAR - missingness is not random but related to other variables and can be accounted for by another complete variable
MNAR - missingness is related to the missing data itself (there is a systematic reason why the data are missing within a particular variable)
Imagine that we are measuring rainfall at weather stations across Norway every year.
| station number | rainfall |
|---|---|
| 1 | 30 |
| 2 | 150 |
| 3 | 75 |
| 4 | 250 |
| 5 | 55 |
| station number | rainfall |
|---|---|
| 1 | 30 |
| 2 | |
| 3 | |
| 4 | 250 |
| 5 | 55 |
| station number | rainfall |
|---|---|
| 1 | 30 |
| 2 | 150 |
| 3 | 75 |
| 4 | 250 |
| 5 | 55 |
| station number | rainfall |
|---|---|
| 1 | |
| 2 | |
| 3 | |
| 4 | 250 |
| 5 | 55 |
Missing data patterns - MAR
| station number | rainfall |
|---|---|
| 1 | 30 |
| 2 | 150 |
| 3 | 75 |
| 4 | 250 |
| 5 | 55 |
| station number | rainfall |
|---|---|
| 1 | 30 |
| 2 | |
| 3 | 75 |
| 4 | |
| 5 | 55 |
Missing data patterns - MNAR
library(missMethods)
# create datasets with levels missingness
penguins_complete <- penguins |>
drop_na()
miss_penguins_MCAR <- delete_MCAR(
penguins_complete,
0.3, "flipper_len"
)
# create a pattern of missingness with censoring
# (missing value in flipper_length if body_mass in below 30% quantile)
miss_penguins_MAR <- delete_MAR_censoring(
penguins_complete,
0.3, "flipper_len", "body_mass"
)
# create a pattern of missingness with censoring
# (missing value in flipper_length if flipper_length is below 30% quantile)
miss_penguins_MNAR <- delete_MNAR_censoring(
penguins_complete,
0.3, "flipper_len"
)
all_data <- bind_rows(
"Full" = penguins_complete,
"MCAR" = miss_penguins_MCAR,
"MAR" = miss_penguins_MAR,
"MNAR" = miss_penguins_MNAR,
.id = "Missingness"
)With MCAR and MAR we can use multiple imputation techniques
# Load mice
library(mice)
# Set seed for reproducibility
set.seed(123)
# Simulate data: location, year, count
data <- data.frame( # Create a dataframe
Location = rep(1:5, each = 5), # Assuming 5 locations,
Year = rep(2009:2023, 5), # Assuming data for 5 years,
Count = rpois(25, lambda = 20) # Simulated count data
)
# Introduce missingness - Missing completely at random (MCAR)
prop_missing <- 0.2 # Example: 20% missingness
missing_indices <- sample(1:nrow(data), prop_missing * nrow(data))
data$Count[missing_indices] <- NAWhen we impute the missing data we use 5 replicates and then take the mean of all 5 datasets.
Using a Structured Equation Model (SEM) we can run a simple linear regression.
library(lavaan)
# we will use the Iris dataset for this example
complete_model <- lm(Petal.Width ~ Sepal.Length + Sepal.Width + Petal.Length, data = iris)
iris_MCAR <- missMethods::delete_MCAR(iris, 0.3, "Sepal.Width")
miss_model1 <- lm(Petal.Width ~ Sepal.Length + Sepal.Width + Petal.Length, data = iris_MCAR)
=====================================================================
Dependent variable:
-------------------------------------------------
Petal.Width
(1) (2)
---------------------------------------------------------------------
Sepal.Length -0.207*** -0.187**
(0.048) (0.057)
Sepal.Width 0.223*** 0.219***
(0.049) (0.057)
Petal.Length 0.524*** 0.516***
(0.024) (0.029)
Constant -0.240 -0.315
(0.178) (0.215)
---------------------------------------------------------------------
Observations 150 105
R2 0.938 0.931
Adjusted R2 0.937 0.929
Residual Std. Error 0.192 (df = 146) 0.204 (df = 101)
F Statistic 734.389*** (df = 3; 146) 454.577*** (df = 3; 101)
=====================================================================
Note: *p<0.05; **p<0.01; ***p<0.001
lavaan 0.7-3 ended normally after 1 iteration
Estimator ML
Optimization method NLMINB
Number of model parameters 5
Used Total
Number of observations 105 150
Number of missing patterns 1
Model Test User Model:
Test statistic 0.000
Degrees of freedom 0
Browne's residual (NT model-based) test
Test statistic 0.000
Degrees of freedom 0
Parameter Estimates:
Standard errors Standard
Information Observed
Observed information based on Hessian
Regressions:
Estimate Std.Err z-value P(>|z|)
Petal.Width ~
Sepal.Length -0.187 0.056 -3.344 0.001
Sepal.Width 0.219 0.056 3.945 0.000
Petal.Length 0.516 0.028 18.133 0.000
Intercepts:
Estimate Std.Err z-value P(>|z|)
.Petal.Width -0.315 0.211 -1.493 0.135
Variances:
Estimate Std.Err z-value P(>|z|)
.Petal.Width 0.040 0.006 7.246 0.000
The argument missing = "ML" to the sem() function estimates a likelihood function for each row based on the variables that are present.
Let all the available data be used.
library(data.validator)
report <- data_validation_report()
between <- function(a, b) {
function(x) {
a <= x & x <= b
}
}
validate(iris, name = "Verifying flower dataset") |>
validate_if(
expr = Sepal.Length > 0,
description = "Sepal length is greater than 0"
) |>
validate_cols(
predicate = between(0, 4),
Sepal.Width,
description = "Sepal width is between 0 and 4"
) |>
add_results(report)
print(report)Validation summary:
Number of successful validations: 1
Number of validations with warnings: 0
Number of failed validations: 1
Advanced view:
|table_name |description |type | total_violations|
|:------------------------|:------------------------------|:-------|----------------:|
|Verifying flower dataset |Sepal length is greater than 0 |success | NA|
|Verifying flower dataset |Sepal width is between 0 and 4 |error | 3|
Validation summary:
Number of successful validations: 1
Number of validations with warnings: 0
Number of failed validations: 1
Advanced view:
|table_name |description |type | total_violations|
|:------------------------|:------------------------------|:-------|----------------:|
|Verifying flower dataset |Sepal length is greater than 0 |success | NA|
|Verifying flower dataset |Sepal width is between 0 and 4 |error | 3|
How would you go about cleaning this dataset?
Date SEX age Species Name lat lon count
1 2023/02/01 male Adult Lagopus matu 9.123826 62.92628 5
2 02/02/23 Female Juvenile Logopus muta 7.836517 61.70676 9
3 2023/02/01 male J Logopus muta 9.387262 63.84034 15
4 01-01-2023 female ADULT Lagopus lagopus 9.609423 61.06334 14
5 01-01-2023 Male ADULT Lagopus matu 8.531654 63.44823 10
6 2023-02-01 Male A Lagopus muta 10.369053 62.60819 10
7 02/02/23 FEMALE JUVENILE Lagopus muta 9.297162 62.71758 11
8 02-01-23 FEMALE Adult Lagopus lagopus 6.479778 61.59151 15
9 2023-02-01 female Juvenile Lagopus muta 7.879991 63.10879 4
10 02/02/23 MALE JUVENILE Lagopus muta 6.794386 62.41039 14
11 2023/02/01 Female ADULT Lagopus matu 10.075820 62.14845 6
12 02-01-23 MALE J Lagopas lagopus 9.368355 64.29196 13
13 2023/02/01 Female ADULT Lagopus matu 10.933222 63.08134 8
14 01-01-2023 Male A Logopus muta 9.508031 61.63579 7
15 02/02/23 MALE JUVENILE Logopus muta 10.925062 62.59583 4
16 02/02/23 female A Lagopus lagopus 9.279532 62.91511 11
17 02/02/23 male J Lagopus lagopus 9.139805 65.96236 10
# A tibble: 1,376 × 8
species island sex year part measure unit value
<fct> <fct> <fct> <int> <chr> <chr> <chr> <dbl>
1 Adelie Torgersen male 2007 bill length mm 39.1
2 Adelie Torgersen male 2007 bill depth mm 18.7
3 Adelie Torgersen male 2007 flipper length mm 181
4 Adelie Torgersen male 2007 body mass g 3750
5 Adelie Torgersen female 2007 bill length mm 39.5
# ℹ 1,371 more rows
# A tibble: 92 × 9
island sex year part measure unit Adelie Gentoo Chinstrap
<fct> <fct> <int> <chr> <chr> <chr> <list> <list> <list>
1 Torgersen male 2007 bill length mm <dbl [7]> <NULL> <NULL>
2 Torgersen male 2007 bill depth mm <dbl [7]> <NULL> <NULL>
3 Torgersen male 2007 flipper length mm <dbl [7]> <NULL> <NULL>
4 Torgersen male 2007 body mass g <dbl [7]> <NULL> <NULL>
5 Torgersen female 2007 bill length mm <dbl [8]> <NULL> <NULL>
# ℹ 87 more rows
No identifier for each observation so R puts all the values in a list. To solve this we need a unique row id.
# A tibble: 1,376 × 10
island sex year part measure unit sample Adelie Gentoo Chinstrap
<fct> <fct> <int> <chr> <chr> <chr> <int> <dbl> <dbl> <dbl>
1 Torgersen male 2007 bill length mm 1 39.1 NA NA
2 Torgersen male 2007 bill depth mm 2 18.7 NA NA
3 Torgersen male 2007 flipper length mm 3 181 NA NA
4 Torgersen male 2007 body mass g 4 3750 NA NA
5 Torgersen female 2007 bill length mm 5 39.5 NA NA
# ℹ 1,371 more rows
One cell should contain one thing
# A tibble: 3 × 2
id x
<int> <chr>
1 1 m-123
2 2 f-455
3 3 f-123
# A tibble: 3 × 3
id sex unit
<int> <chr> <chr>
1 1 m 123
2 2 f 455
3 3 f 123
Have a look at the smallGame dataset.
What format is it in (long or wide)?
How would you convert it to the other format?
Which format do you find easier to use? (there is no “correct” answer to this one!)
With your computer
With your collaborators
“Your closest collaborator is you six months ago but you don’t reply to email.” — Paul Wilson
Need understandable code
A condition of publication in a Nature Portfolio journal is that authors are required to make materials, data, code, and associated protocols promptly available to readers without undue qualifications.
it is a condition for publication of accepted manuscripts at CJFAS that authors make publicly available all data and code needed to reproduce those results (including code to reproduce statistical results, simulation results, and figures) via an online data repository.
The only way to write good code is to write tons of shitty code first. Feeling shame about bad code stops you from getting to good code
— Hadley Wickham (@hadleywickham) 17 April 2015
Make your own style - but be consistent
“There are only two hard things in Computer Science: cache invalidation and naming things.”
— Phil Karlton
TRUE, for, ifcnames_1 = data_1
.columnNames()
.filter((c) => !['Timestamp', 'Score'].includes(c))
.map(d => d.replace(/^.*\[/, '').replace(/]$/, ''));
foldData_1 = data_1
.fold(aq.not('Timestamp', 'Score'), { as: ['question', 'answer'] })
.spread({question: d => op.split(d.question, '[')} , {as: ['question_title', 'question']})
.derive({question: d => op.replace(d.question, ']', '')});Plot.plot({
marginLeft: 170,
marginBottom: 60,
height: 500,
x: {label: 'Frequency', labelOffset: 50},
y: {label: null, domain: cnames_1},
color: {
legend: true,
domain: ["Yes", "No"],
range: ["green", "orange"],
},
style: {fontSize: '25px'},
marks: [
Plot.barX(foldData_1, {y: 'question', x: 1, inset: 0.5, fill: 'answer', sort: 'answer'}),
Plot.ruleX([0])
]
})Å
| camelCase 🐫 | UpperCamelCase | snake_case 🐍 |
|---|---|---|
| billLengthMM | BillLengthMM | bill_length_mm |
| bergenWeather2022 | BergenWeather2022 | bergen_weather_2022 |
| dryMassG | DryMassG | dry_mass_g |
| makeWeatherPlot | MakeWeatherPlot | make_weather_plot |
Place spaces
|>, +, -, <-, )= in function callsHelps you find your way around a script
Repeated code is hard to maintain
Make repeated code into functions.
Single place to maintain
Long scripts become difficult to navigate
Fix by moving parts of the code into different files
For example:
Import with
Number files so they sort alphabetically in order of use.
All in one directory - messy
/var/folders/9f/_rfj1x210xv96zcnkqbkj8dw0000gn/T//Rtmp90XGtd/bad
├── data1.csv
├── data2.csv
├── discussion.qmd
├── introduction.qmd
└── thesis.Rproj
Directories for each type of file - clean
/var/folders/9f/_rfj1x210xv96zcnkqbkj8dw0000gn/T//Rtmp90XGtd/good
├── chapters
│ ├── 1-introduction.qmd
│ └── 2-discussion.qmd
├── data
│ ├── data1.csv
│ └── data2.csv
└── thesis.Rproj
Have a look at the “notReproducible.qmd” file
What is wrong with the code?
Some badly formatted code in “bad_style.R”
# Spacing
average<-mean(feet/12 + inches,na.rm=TRUE)
sqrt(x ^ 2 + y ^ 2)
x <- 1 : 10
base :: get
# Indenting
if (y < 0 && debug)
message('Y is negative')
# Assignment
x = 5
lintr for static code analysisDetects style issues, syntax errors and fragile code
/Users/audhalbritter/Documents/Research/OS_course/OS-course-teaching-material/data_handling/bad_style.R:1:10: style: [trailing_whitespace_linter] Remove trailing whitespace.
# Spacing
^
/Users/audhalbritter/Documents/Research/OS_course/OS-course-teaching-material/data_handling/bad_style.R:2:8: style: [infix_spaces_linter] Put spaces around all infix operators.
average<-mean(feet/12 + inches,na.rm=TRUE)
^~
/Users/audhalbritter/Documents/Research/OS_course/OS-course-teaching-material/data_handling/bad_style.R:2:19: style: [infix_spaces_linter] Put spaces around all infix operators.
average<-mean(feet/12 + inches,na.rm=TRUE)
^
/Users/audhalbritter/Documents/Research/OS_course/OS-course-teaching-material/data_handling/bad_style.R:2:32: style: [commas_linter] Put a space after a comma.
average<-mean(feet/12 + inches,na.rm=TRUE)
^
/Users/audhalbritter/Documents/Research/OS_course/OS-course-teaching-material/data_handling/bad_style.R:2:37: style: [infix_spaces_linter] Put spaces around all infix operators.
average<-mean(feet/12 + inches,na.rm=TRUE)
^
/Users/audhalbritter/Documents/Research/OS_course/OS-course-teaching-material/data_handling/bad_style.R:8:0: style: [indentation_linter] Indentation should be 2 spaces but is 0 spaces.
message('Y is negative')
^~
/Users/audhalbritter/Documents/Research/OS_course/OS-course-teaching-material/data_handling/bad_style.R:8:9: style: [quotes_linter] Only use double-quotes.
message('Y is negative')
^~~~~~~~~~~~~~~
/Users/audhalbritter/Documents/Research/OS_course/OS-course-teaching-material/data_handling/bad_style.R:10:3: style: [assignment_linter] Use one of <-, <<- for assignment, not =.
x = 5
^
styler to fix your styleStyling 1 files:
bad_style.R ℹ
────────────────────────────────────────
Status Count Legend
✔ 0 File unchanged.
ℹ 1 File changed.
✖ 0 Styling threw an error.
────────────────────────────────────────
Please review the changes carefully!
# Spacing
average <- mean(feet / 12 + inches, na.rm = TRUE)
sqrt(x^2 + y^2)
x <- 1:10
base::get
# Indenting
if (y < 0 && debug) {
message("Y is negative")
}
# Assignment
x <- 5
It runs (on your computer)
It runs (on my computer - without me having to do anything/much)
It does what you expect it to do (even after 5 years)
It is documented in some way
If the first line of your R script is
I will come into your office and SET YOUR COMPUTER ON FIRE 🔥.
If the first line of your R script is
I will come into your office and SET YOUR COMPUTER ON FIRE 🔥.
Comments
Use # to start comments.
Help you and others to understand what you did
Comments should explain the why, not the what.
Try to make code self-documenting with descriptive object names