Positive Controls for Psychology – My pitch for a SIPS project
Positive controls are one of the most useful tools for ensuring interpretable and fruitful research. Strangely, though, positive controls are rarely used in psychological research. That’s a shame, but also an opportunity–it would be an easy but substantial improvement for psychological researchers to start using regularly using positive controls. I (Bob) am currently at SIPS 2018; I’ll be giving a lighting talk about positive controls and hopefully developing some resources to encourage the use of positive controls. To kick things off, I’ve started this OSF page on positive controls: https://osf.io/n5yx9/
What is a positive control? A positive control (aka an active control) is a research condition that has a known effect in the research domain; it’s a research condition that ought to work if the research is conducted properly. For example, a researcher might be studying how much a new drug affects alertness. She will administer either the new drug or placebo and then measure alertness with an odd-ball task. A positive control would be adding a third group that receives caffeine (blinded, of course), a drug well known to produce a modest increase in alertness on this task.
Why use positive controls? There are several potential benefits:
- Positive controls help indicate the sensitivity and integrity of the experiment. If the experiment is conducted properly and with sufficient data then the positive control ought to show the expected effect. If the experiment does not, then the research will know that something may have gone on and will be able to investigate. Positive controls are especially useful for interpreting “negative” results. From the example above, if the researcher finds that the new drug does not influence alertness, she may wonder about the result: was enough data collected and was the procedure administered correctly? Checking that the positive control came out as expected gives reassurance that the research was conducted properly and was sensitive to the desired range of effects.
- Positive controls can be used as a training tool–new researchers can run positive controls to ensure procedural proficiency before collecting real data (and while collecting real data)
- Screening for outliers and/or non-compliant responding – for some positive controls there is a clear range of valid responses even at the individual level. In these cases, positive controls provide an additional way to screen for outliers and unusual responses.
How to select a positive control? To aid interpretation, a positive control should be well-matched to the experimental question. The ideal positive control:
- Is from the same research domain
- Has a well-characterized effect size that is similar to what is expected for the research question (or a set of positive controls can be used to test sensitivity to small, medium, and large effects)
- Is sensitive to the factors that could ruin the effect of interest
- Is easy/short to administer
Can positive controls really help in psychology? Yes. I (Bob) have been using positive controls extensively in my replication research. These have been essential in demonstrating the quality and sensitivity of the replication research. For some examples, see:
Calin-Jageman RJ. 2018. Direct replications of Ottati et al. (2015): The earned dogmatism effect occurs only with some manipulations of expertise. J Exp Soc Psychol 1–10. http://linkinghub.elsevier.com/retrieve/pii/S0022103117302913.
Lehmann GK, Calin-Jageman RJ. 2017. Is Red Really Romantic? Soc Psychol (Gott) 48: 174–183. http://econtent.hogrefe.com/doi/10.1027/1864-9335/a000296.
Sanchez C, Sundermeier B, Gray K, Calin-Jageman RJ. 2017. Direct replication of Gervais & Norenzayan (2012): No evidence that analytic thinking decreases religious belief ed. M. Van Elk. PLoS One 12: e0172636. PMID: 28234942. http://dx.plos.org/10.1371/journal.pone.0172636.
Moery E, Calin-Jageman RJ. 2016. Direct and conceptual replications of Eskine (2013): Organic food exposure has little to no effect on moral judgments and prosocial behavior. Soc Psychol Personal Sci 7: 312–319. http://spp.sagepub.com/cgi/doi/10.1177/1948550616639649.
Cusack M, Vezenkova N, Gottschalk C, Calin-Jageman RJ. 2015. Direct and Conceptual Replications of Burgmer & Englich (2012): Power May Have Little to No Effect on Motor Performance ed. J.M. Haddad. PLoS One 10: e0140806. PMID: 26536592. http://dx.plos.org/10.1371/journal.pone.0140806.
So how do I get started? I (Bob) have started an OSF page on positive controls. I’m hoping to use some of my time at SIPS 2018 to populate the page and start some research to show they are worth using. Here’s the page (still in development): https://osf.io/n5yx9/