Open Science Practices: Progress, Yes, but Must Do Better

I posted about our preprint a while back, but now it’s refereed and online, open access. The citation:

Giofrè, D., Boedker, I., Cumming, G., Rivella, C., & Tressoldi, P. (2022). The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices: Five years later. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01993-3

Take-home message

Open Science practices improved in two leading psychology journals, in some cases greatly, during 2013-2015. From 2016-2020, as we now report, there was good progress on open materials and data, but little further advance on CI use or interpretation, and NHST remains ubiquitous. More progress required.

A fuller summary

Judging from Psychological Science (PS) and the Journal of Experimental Psychology: General (JEPG), during 2013-2020:

• Reporting of Confidence Intervals (CIs) increased markedly from 2013 to 2015 🙂 but has plateaued at around 60% of articles since then 🙁
• CIs are still rarely explicitly used to inform interpretation of results 🙁 🙁 
• Use of Effect Sizes (ESs) to inform interpretation has increased steadily to around 50% 🙂
• Provision of Open Data and Analysis Code, and Open Materials has increased steadily, from near zero to around 50% 🙂
• Use of Preregistration has increased from near zero to around 30% 🙂
• Use of NHST remains almost universal 🙁
• Overall, only 1.9% of articles reported replications 🙁 and there were no registered reports.
• Larger changes in PS than JEPG suggest strong journal-specific policies and offering Open Science badges can be effective 🙂 🙂 

Conclusion: Open Science has made enormous strides since 2013, but there is still a long way to go. Keep at it!

Much more detail in my earlier post.

Geoff