Standardization cause of poor reproducibility?

In the last few years Web-based experiments have become an attractive alternative to lab-based experiments. Next to the advantages of versatility and the ecological validity of the results, Web-based experiments can potentially reach a much larger, more varied and intrinsically motivated participant pool. Especially in the domain of music perception and cognition it is important to probe a wide variety of participants, with different levels of training and cultural backgrounds.

Nevertheless, to get research published that takes advantage of the Internet is not straightforward. An important reason for the conservatism held by some journals in publishing results obtained with Web-based experiments is the issue of replicability. Especially in the fields of experimental psychology and psychophysics there are serious concerns about the (apparent) lack of control one has in Web experiments as opposed to those performed in the laboratory. Where in the lab most relevant factors, including all technical issues, are under control of the experimenter (i.e. have a high internal validity) it is argued that Web experiments lack this important foundation of experimental psychology. As a result of the first issue, it often proves to be problematic to convince University Review Panels to give permission when there is little insight in the environment in which participants tend to do these experiments. As a result of the second issue, some high-impact journals made it a policy decision not to publish Web-based studies, as such discouraging Web experiments to be performed (cf. Honing & Ladinig, 2008; Honing & Reips, 2008). Nevertheless, it is important to stress that if an effect is found - despite the limited control in Web-based experiments over the home environment and the technological variance caused by the Internet - then the argument for that effect and its generalizability is even stronger.

The latter issue was recently discussed in an issue of Nature Methods by researchers from the Universities of Giessen and Münster, Germany (see reference below and [modified] figure above). In fact, the authors make the opposite argument! They argue that standardization should be seen as a cause of, rather than a cure for, poor reproducibility of experimental outcomes. Their study showed that environmental standardization can contribute to spurious and conflicting findings in the literature. Würbel and colleagues conclude that to generate results that are most likely going to be reproducible in other laboratories, the strategies to standardize environmental conditions in an experiment should be minimized.

As such the variance caused by Web-based setups (as discussed above) might actually amount to experimental results with a much higher external validity than thought before.

Richter, S., Garner, J., Auer, C., Kunert, J., & Würbel, H. (2010). Systematic variation improves reproducibility of animal experiments. Nature Methods, 7 (3), 167-168. 10.1038/nmeth0310-167


Honing, H., & Reips, U.-D. (2008). Web-based versus lab-based studies: a response to Kendall (2008). Empirical Musicology Review, 3 (2), 73-77.


Honing, H., & Ladinig, O. (2008). The potential of the Internet for music perception research: A comment on lab-based versus Web-based studies. Empirical Musicology Review, 3 (1), 4-7.