SOCR ≫ DSPA ≫ DSPA3 Topics ≫ 

library(ggplot2)
library(dplyr)
library(tidyr)
library(patchwork)   # panel composition, replaces gridExtra / par(mfrow)
library(plotly)      # interactive figures and ALL 3-D graphics
library(DT)

How this chapter uses graphics

Every two-dimensional figure is drawn with ggplot2 and rendered statically. Immediately after each one you will find the equivalent plot_ly() code in a chunk marked eval=FALSE, echo=TRUE, the code is printed in these notes but not executed, so you can copy it into a live session for an interactive demonstration without inflating the size of this document.

Every three-dimensional figure, surfaces, 3-D scatter, image stacks, volumetric neuroimaging, interactive dendrograms and scatterplot matrices — is drawn with plot_ly() and evaluated, because rotation and zoom carry information no static image can. Those figures are live in the rendered HTML.

The rationale is perceptual, not aesthetic: static images are reproducible, printable, and diffable; interaction earns its cost only when the reader must navigate rather than read (§2.10).

1 Learning objectives

After completing this chapter you will be able to:

  1. Ingest tabular, compressed, foreign-format (SPSS), spreadsheet, and HTML-table data, and verify what you loaded via checksum and structure.
  2. Choose an appropriate measure of centrality and dispersion by reasoning about the breakdown point and the AM–GM–HM inequality, rather than by habit.
  3. Derive the bias–variance tradeoff that determines optimal histogram bin width and kernel bandwidth, and apply Sturges, Scott, Freedman–Diaconis, and Silverman’s rules.
  4. Analyze a contingency table correctly: test statistic, cell contributions, standardized residuals, effect size, and validity conditions.
  5. Classify missingness as MCAR / MAR / MNAR using formal notation, and prove why single mean imputation deflates variance.
  6. Implement EM imputation, state its convergence property, and analyze its computational complexity.
  7. Pool multiple imputations using Rubin’s rules and interpret the between- vs. within-imputation variance decomposition.
  8. Rebalance an imbalanced cohort without leaking information from the test set, and quantify the optimism that leakage induces.
  9. Select a visualization for a given analytic question using the Cleveland–McGill accuracy ordering.
  10. Fit univariate and mixture distribution models by maximum likelihood and EM, and assess fit with methods valid under estimated parameters.

Estimated time: 8–10 hours including exercises. Prerequisites: Chapter 1, in particular the reproducibility conventions (§1.6.6) and the dspa_read() loader (§1.6.7), both of which this chapter reuses.

1.1 Chapter 2 Live Webapps

The DSPA3 Labs include a number of interactive webapps, which are build as portable, self-contained, and browser accessible applications. These apps demonstrate various concepts discussed in this chapter, including:

2 PART I, DATA HANDLING, QUALITY, AND MISSINGNESS

3 Ingesting data

Every analysis begins by moving bytes into memory and losing information in the process. The question is only whether you know which information you lost.

3.1 Native R serialization

data("iris")
class(iris)
#> [1] "data.frame"
# Chapter 1 convention: never write into the working directory.
rds_path <- file.path(tempdir(), "iris.rds")
saveRDS(iris, rds_path)
iris2 <- readRDS(rds_path)
identical(iris, iris2)
#> [1] TRUE