Exercise 1

For this initial exercise we will be trying to identify errors in a data frame containing information about the US presidents. First we install the packages needed to get the dataset that we will be working with. If you have already installed dataMaid from home, you can skip this step.

install.packages("dataMaid")

Now we have installed the dataMaid package which contains the raw dataset, presiData. We can see the first couple of rows of the dataset with the head command.

library("dataMaid")
load(url("http://biostatistics.dk/eRum2018/data/presiData.rda"))
head(presiData)

##     lastName firstName orderOfPresidency   birthday dateOfDeath
## 10     Tyler      John                10 1790-03-29  1862-01-18
## 2      Adams      John                 2 1735-10-30  1826-07-04
## 25  McKinley   William                25 1843-01-29  1901-09-14
## 32 Roosevelt  Franklin                32 1882-01-30  1945-04-12
## 14    Pierce  Franklin                14 1804-11-03  1869-10-08
## 24 Cleveland    Grover                24 1837-03-18  1908-06-24
##     stateOfBirth assassinationAttempt  sex ethnicity presidencyYears
## 10      Virginia                    0 Male Caucasian               3
## 2  Massachusetts                    0 Male Caucasian               3
## 25          Ohio                    1 Male Caucasian               4
## 32      New York                    1 Male Caucasian              12
## 14 New Hampshire                    0 Male Caucasian               4
## 24    New Jersey                    0 Male Caucasian               4
##    ageAtInauguration favoriteNumber
## 10                51           1+0i
## 2                 61           4+0i
## 25                54           3+0i
## 32                51           6+0i
## 14                48           4+0i
## 24                55           2+0i

We are being told by our research collaborators that the dataset should contain information on 11 variables relates to the US presidents. The variables are

• lastName The surname of the president.
• firstName The first name of the president.
• orderOfPresidency A factor variable indicating the order of the presidents (with George Washington as number 1 and Donald Trump as number 45).
• birthday The date of birth of the president.
• dateOfDeath The date of death of the president.
• stateOfBirth A character variable with the state in which the president was born.
• assassinationAttempt A text variable indicating whether there was an assassination attempt (1) or not (0) on the president.
• sex A factor variable with the sex of the president.
• ethnicity A factor variable with the ethnicity of the president.
• presidencyYears A numeric variable with the duration of the presidency, in years.
• ageAtInauguration the age at inauguration.
• favoriteNumber A complex type variable with a fictional favorite number for each president.

Try to find and document as many errors are you can in the presiData. In particular focus on validity (conforms to the correct syntax with the right format, type, and range), on consistency, accuracy, and uniqueness.

[Hint: If you are new to R then you can use the functions str, print and the $ operator to extract variables from the data frame.]

Exercise 2

Here we will use the dataMaid package makeDataReport() to screen data for obvious and non-obvious errors in the dataset. We will continue working on the presiData from the dataMaid package in order to find errors that we previously missed.

Note that exercise 2 contains a lot of exercises. We do not expect you to be able to solve all of them in the time given. Remember the primary focus is to identify potential errors in the dataset.

1. Run the makeDataReport() function to make your own report for the presiData and look through it. What legitimate errors are found? What warnings are unnecessary?

2. Start by verifying the validity of the data (format, type, and range). When dataMaid encounters a variable type that is unknown then it prints a note and skips the variable. If we want to perform specific checks for a variable with an unknown type then we must fix the class in the data frame (when relevant) or alternatively force dataMaid to consider a specific class as if it were another class.

   Handle the unusual variable class used for the variables lastName and firstName by changing their classes to something more standard. This can be done using the code

   class(presiData$firstName) <- "character"
   class(presiData$lastName) <- "character"

   Try calling help(makeDataReport) and look at the documentation for the argument treatXasY. Use this argument in a new call to makeDataReport() so that for example the favoriteNumber variable will be handled like a numeric variable.

   [Hint: you may need to use the replace=TRUE argument too to overwrite any previous reports.]

Note how everything is documented and can be used in collaboration with a research partner.

The argument checks can be used to control which checks are performed for variables of each supported class. The helper function setChecks() makes it easy to add or remove certain checks for certain variable classes while allCheckFunctions() returns an overview of all available checkFunctions to choose from.

The following bit of code shows how to change the choice of checks for numeric variables to not include a check for outliers (i.e. the checkFunction identifyOutliers):

makeDataReport(presiData, replace = TRUE,
               checks = setChecks(numeric = defaultNumericChecks(remove="identifyOutliers")))

#or, by without the use of defaultNumericChecks():
makeDataReport(presiData, replace  = TRUE,
               checks = setChecks(numeric = "identifyMissing"))

Now, it’s your turn to make a makeDataReport() call that does not check whether or not character variables have less than 5 unique levels. You can proceed as follows:

3. Use allCheckFunctions() to find out which checkFunction does the check for less than 5 unique values. Check that this function is indeed among the default checkFunctions for character variables by calling defaultCharacterChecks().

4. Use the checks argument, setChecks() and defaultCharacterChecks() to remove the “less than 5 unique values”-check from the checks performed for character variables. Look at the overview table on the first page of your report. Can you find a difference?

The default plots used in a dataMaid report are made using the ggplot2 package. However, you can change the plot style to base R graphics if you please by using the visuals argument in makeDataReport().

5. Use allVisualFunctions() to identify the name of the visualFunction that you need to use if you want base R graphics style plots.

6. Look at the documentation for setVisuals() e.g. by calling ?setVisuals. Here, you see an argument called all. Call setVisuals() in the console using the all argument to specify the graphics style plots. Try also calling setVisuals() with the new function in the factor and character arguments only.

7. Use makeDataReport()’s visuals argument and setVisuals() to obtain a data report where character and factor variables use base R graphics plots, while all other variable classes use ggplot2 plots.

8. The choice of summaries included for each variable class in the report can also be controlled, and the manner in which this is done is very similar to the procedures you have tried for checks and visuals in the above. Experiment with the functions setSummaries(), allSummaryFunctions() and defaultCharacterSummaries() in order to remove “Mode” from the summaries listed for such variables in the data report (using the argument summaries).

9. We would now like a report that only displays the results of checks (no visuals or summaries) and only lists variables that actually were found to have potential problems. But for each problem found, we would like to see all problematic values listed, not just the first 10, as is currently the case. Moreover, we would like to rename the report title (as displayed on the front page) to be “Problem flagging report”, and we would also like to have “problemflagging” appended to the file name of the report, so that we can easily tell it apart from the usual data report.

   Look at the documentation for makeDataReport() and try to produce the report described here.

library(dataMaid)
toyData$pill

##  [1] red  red  red  red  red  red  red  red  red  red  blue blue blue <NA>
## [15] <NA>
## Levels: blue red

refCat <- function(v, ...) {
  val <- #the reference category of the variable v
  res <- val
  summaryResult(list(feature = "Reference category", result = res,
                     value = val))
}

refCat <- summaryFunction(refCat,
  description = #Text describing what the summaryFunction does,
    ,
  classes = c(#[vector of data types that the function is intended for]
    )
  )

makeDataReport(presiData, 
               summaries = setSummaries(factor = defaultFactorSummaries(add = "refCat")),
               vol = "_withRefCat")

identifyNonStartCase <- function(v, nMax, maxDecimals, ...) {
  
  #do the check
  problemValues <- #vector of values in v that are problematic, i.e. that are not 
    #written in start case. If no problem is encountered, it should be set to NULL
  
  problem <- #is there a problem? TRUE/FALSE
  
  problemStatus <- list(problem = problem,
                        problemValues = problemValues)

  problemMessage <- #Message that is printed prior to listing
                    # problem values in the dataMaid output,
                    # ending with a colon
    
  outMessage <- messageGenerator(problemStatus, problemMessage, nMax)

  checkResult(list(problem = problem,
    message = outMessage,
    problemValues = problemValues))
}

identifyNonStartCase <- checkFunction(identifyNonStartCase,
  description = #Some text describing the checkFunction
    ,
  classes = c(#[the data types that this function is intended to be used for]
    )
)

Exercise 3