Open Science Practices: Progress, Yes, but Must Do Better
Giofrè, D., Boedker, I., Cumming, G., Rivella, C., & Tressoldi, P. (2022). The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices: Five years later. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01993-3
Open Science practices improved in two leading psychology journals, in some cases greatly, during 2013-2015. From 2016-2020, as we now report, there was good progress on open materials and data, but little further advance on CI use or interpretation, and NHST remains ubiquitous. More progress required.
A fuller summary
Judging from Psychological Science (PS) and the Journal of Experimental Psychology: General (JEPG), during 2013-2020:
- Reporting of Confidence Intervals (CIs) increased markedly from 2013 to 2015 🙂 but has plateaued at around 60% of articles since then 🙁
- CIs are still rarely explicitly used to inform interpretation of results 🙁 🙁
- Use of Effect Sizes (ESs) to inform interpretation has increased steadily to around 50% 🙂
- Provision of Open Data and Analysis Code, and Open Materials has increased steadily, from near zero to around 50% 🙂
- Use of Preregistration has increased from near zero to around 30% 🙂
- Use of NHST remains almost universal 🙁
- Overall, only 1.9% of articles reported replications 🙁 and there were no registered reports.
- Larger changes in PS than JEPG suggest strong journal-specific policies and offering Open Science badges can be effective 🙂 🙂
Conclusion: Open Science has made enormous strides since 2013, but there is still a long way to go. Keep at it!
Much more detail in my earlier post.