Data Collection

Data Collection Questionable Research Practices (Wicherts et al, 2016, Table 1)

  1. Failing to randomly assign participants to conditions.
  2. Insufficient blinding of participants and/or experimenters.
  3. Correcting, coding, or discarding data during data collection in a non-blinded manner.
  4. Determining the data collection stopping rule on the basis of desired results or intermediate significance testing. Proviso – unless using a valid dynamic stopping approach (discussed previously).

Guidance:

With respect to random assignment, it is not only important to commit to true random assignment when possible (i.e., by using a random number generator), but it is also important to keep in mind that the goal of random assignment (i.e., equivalence of participants between conditions on all nuisance factors) can be achieved only with much higher sample sizes than previously thought (see article and reflection). The threats to experimental integrity by not blinding participants and/or experimenters are well known (i.e., demand character and inadvertent direction of participants’ responding). Performing anyoperation on data in a non-blinded manner could inflate the Type I error rate. As Wagenmakers and others have made clear, it is paramount that there be no possibility of stopping based on observed results. Doing so willinflate the Type I error rate – unless appropriate mitigation procedures are used (e.g., p-augmented, sequential analysis, or by using a Bayesian statistical approach).

Student Check List 3 of 5: Data Collection

_____  The student thoroughly discussed design considerations and reasoning with the committee.

_____  The student agreed to create power estimates following data collection (see previous section)

_____  The student discussed a data sharing plan with the committee.

_____  The ethics application form appropriately reflects the data sharing strategy on the Informed Consentpage (if relevant).