Simon Gandevia, a giant in the research integrity community, assembled an international A-list of incredible speakers–plus me–for this Conference (IRIC) recently in Sydney.
Diversity of disciplines, but a common focus
There were more than 100 registrants, so the presentation room was usually packed. A striking feature was our enormous diversity of disciplines: medicine, law, neuroscience, statistics, psychology, journalism, education… and more. Our common focus was the integrity of research, Open Science, and meta-research (research on how research is done). Most presentations to the conference were either an alarming assessment of how the research literature has been polluted, or a (sometimes) encouraging evaluation of efforts to improve integrity.
We felt strongly about so many things, with emotions swinging wildly. Mainly we felt enormous pride in group cohesion across such a diversity of backgrounds, and conviction that our mission–the very trustworthiness of research–is enormously important.
Paper mills, fakery, and more
But then we’d despair when a speaker documented the extend of deadly pollution spreading in the scientific literature, from old-fashioned fakery to plagiarism and fake manuscripts sold by paper mills. Anyone wanting to bulk up their cv can buy an AI-generated manuscript and submit it to a dozen journals. The best journals will identify the fakery, these days using AI to help, but some ‘vanity’ journal will publish the manuscript after a bogus peer review process–of course for a hefty payment. What criminality! What waste, especially of the time and effort of expert editors!
The memorable conference dinner was held on the roof deck of the stunning new Museum of Contemporary Art. Champagne in hand, sun setting on the harbour bridge to the left, opera house to the right: what’s not to like?
Significance Roulette released to the world
I’m delighted to say my brief presentation was very well received. Most importantly, I released to the world for the first time Bradley Dean‘s Significance Roulette that you can download, save, and run in your browser. Or simply go to esci web and select the significance roulette menu item to see and run the roulette wheel.
From here you can download my powerpoint slides, also a pdf summary, and the .html file you can save to your computer and run in your browser to enjoy significance roulette.
Read more about the rationale for significance roulette here.
The New Statistics: Why and How (abstract below) explained the advantages of moving on from NHST to the new statistics (estimation and meta-analysis) and the need for better practices to improve research integrity. I’m delighted that an award from Sage indicates the article seems to be helping researchers improve what they do. Next: Can ITNS2 help the next generation do even better?
The article appeared online in late 2013, so was considered when Sage examined the citation numbers of all articles appearing in any of the 400+ journals Sage published back in 2013. It was one of the top three most cited, so has been given a Sage 10-Year Impact Award. Sage’s announcement is here. Sage has just published a blog post about it with headline:
Now for the abstract:
The article was commissioned by Eric Eich, then editor-in-chief of Psychological Science, to appear immediately following his famous editorial Business Not As Usual. This opened the Journal’s first issue of 2014 and announced sweeping changes in the journal’s submission requirements, which, for many psychologists, marked the arrival of Open Science.
Interview
Sage’s blog post includes an email interview with me. Here’s a brief summary:
What was it in your own background that led to your article?
When I was a teenager my father gave me a simple explanation of significance testing. I said something like “That’s weird, sort of backwards. And why .05?” He replied “I agree, but that’s the way we do it.”
Over decades of teaching I became ever more dissatisfied with NHST, and focused ever more on confidence intervals (CIs).
Was there an article that had a particularly strong influence on you?
Frank Schmidt (1996) wrote: “It is now possible to use meta-analysis to show that reliance on significance testing retards the development of cumulative knowledge.” A revelation!
What did Schmidt’s article lead to?
About 2003 I started using an Excel forest plot to give a simple explanation of meta-analysis in my intro course. I was delighted: Students told me it just made sense. Of course, for meta-analysis you need a CI from each study, while p values are irrelevant, even misleading.
Figure: Dances of means, confidence intervals, and p values.
In 2009 I uploaded a video of the dance of the p values. I became passionate about advocating the new statistics (estimation and meta-analysis). I wrote Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis (UTNS, 2012).
What was happening in psychology at about that time?
Ioannidis (2005) explained how reliance on NHST was a major cause of the replication crisis. Largely in response to that crisis, Open Science arrived—perhaps the most important advance in how science is done for a very long time.
Eric Eich’s famous editorial Business Not As Usual in the January 2014 issue of Psychological Science marked the arrival of Open Science in psychology. Months earlier Eric had invited me to write a tutorial article to support the changes he wanted. This was The New Statistics: Why and Howand was published immediately following his editorial.
What has been the reception of the article?
Mainly very positive. Some have felt I went too far in advising that in most cases it’s better not to use NHST at all. Some Bayesians have been unhappy with the focus on confidence intervals.
Revisiting that article, what would you have done differently?
I used the term ‘research integrity’, but ‘Open Science’ was coming into use and I soon realized that was way better. Reading the article today, for ‘research integrity’ read ‘Open Science’.
Otherwise, I think the article has held up well, including all 25 guidelines in Table 1.
What has happened since
Psychological Science has continued to lead in the adoption of Open Science practices.
Meta-science, also known as meta-research, has emerged and now thrives as a highly multi-disciplinary field. It applies the scientific method to improve that method—wonderful!
What have you been doing since?
I teamed with Robert Calin-Jageman to write the first intro statistics textbook based on the new statistics and with Open Science all through. The second edition has just come out: Introduction to The New Statistics: Estimation, Open Science, and Beyond, 2nd edition (ITNS2, 2024). It has much improved software, as we explain in Calin-Jageman & Cumming (2024), which is on open access.
We believe this book can sweep the world—we’ll see! To read the Preface and Chapter 1 go to www.thenewstatistics.com. In the second para is a link to the book’s Amazon site. Click ‘Read sample’.
References
Calin-Jageman, R., & Geoff Cumming, G. (2024). From significance testing to estimation and Open Science: How esci can help. International Journal of Psychology, https://doi.org/10.1002/ijop.13132
Cumming, G. (2012). The New Statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1(2), 115-129. https://doi.org./10.1037/1082-989X.1.2.115
Originally from the Czech Republic (Czechia), Petra has lived, studied, and worked all over the world, as you can see at her site. Her doctorate is from Oxford. Her TEDx talk outlines some of her research interests.
She arrived in Australia a couple of years ago as a post doc at Griffith University, Queensland. Soon after, she launched into organising what became a three-day online Workshop on Good Statistical Practice in Archaeology–open to anyone of any discipline.
I first learned of her enthusiasm for statistical reform and Open Science when she kindly invited me to speak at that workshop. I gave a talk on the new statistics, and another on using Bob’s new esci software in jamovi–in archaeological science, as in any other discipline. I posted about the workshop here.
Earlier this year Petra took up a lectureship at ANU in Canberra, and enthusiastically took on teaching statistics and topics in archaeological science to both undergraduate and postgraduate students.
She expressed keen interest in our second edition, even volunteering to help. Over many months last year she worked through final drafts of most chapters, picking up many errors and infelicities. Later she worked painstakingly through the proofs of many chapters, picking up elusive tiny errors. She made an immense contribution to ITNS2, as Bob and I acknowledge on p. xxvii.
So last week Petra, Fiona and I had much to discuss. Petra will be at AIMOS in November (more details here), no doubt meeting many of the good folks who make the Meta-science scene in Australia so lively and multi-disciplinary.
As an associate editor of the Journal of Archaeological Science she is helping develop guidelines for that journal to encourage reproducibility and Open Science practices.
I discovered years ago that archaeological science has Open Science lessons for us all–see my post here. I wish Petra all strength for her continuing efforts towards statistical reform and Open Science!
00:00: Brian’s early interest in improving science 15:24: How the Center for Open Science got funded (by John and Laura Arnold) 26:08: How long is COS financed into the future? 29:01: What if COS isn’t benefitting science anymore? 35:42: Is Brian a scientist or an entrepreneur? 40:58: The future of the Center for Open Science 51:13: A book or paper more people should read 54:42: Something Brian wishes he’d learnt sooner 58:53: Advice for PhD students/postdocs
Here’s a depressing new paper that surveys reporting practices in The Journal of Neurophysiology after it instituted new reporting guidelines for authors (Héroux et al., 2023). Why depressing? Because the updated guidelines seem to have done little to improve reporting, and this was true even for practices the editors had said would be required. Was that because the editors went to far, asking authors to jump through arduous new hoops? No. About 2/3 of authors defied (with impunity) a requirement to consistently report the sample size and statistical test used in their figure captions. Really? Yes, really. And, in general, poor statistical practices abound and have shown little change: about 60% of recent papers interpreted non-significant results as trending towards significance, reporting of exact p values was sporadic, and reporting of effect sizes with confidence intervals has increased but remains rare (30%).
This makes me (Bob) sad. The Journal of Neurophysiology is a strong journal that is published by a renowned professional society (just check out these beauties: (Calin-Jageman, Tunstall, Mensh, Katz, & Frost, 2007) and (Sakurai, Calin-Jageman, & Katz, 2007). The editors have clearly worked to take steps to improve statistical reporting, with updated policies in 2016 and 2018. None of the new requirements were especially out there, either — this was a pretty conservative effort for reform. Even in this conducive context, it is clear that the inertia of poor practices is overwhelming — both from authors who won’t adapt and reviewers/editors unequipped and/or unmotivated to consistently appy new policies.
I guess this is only surprising if you’re not yet fully cynical. Blech.
Calin-Jageman, R. J., Tunstall, M. J., Mensh, B. D., Katz, P. S., & Frost, W. N. (2007). Parameter Space Analysis Suggests Multi-Site Plasticity Contributes to Motor Pattern Initiation inTritonia. American Physiological Society. doi: 10.1152/jn.00572.2007
Héroux, M., Diong, J., Bye, E., Fisher, G., Robertson, L., Butler, A., & Gandevia, S. (2023). Poor statistical reporting, inadequate data presentation and spin persist despite Journal awareness and updated Information for Authors. F1000 Research Ltd. doi: 10.12688/f1000research.142841.1
Sakurai, A., Calin-Jageman, R. J., & Katz, P. S. (2007). Potentiation Phase of Spike Timing-Dependent Neuromodulation by a Serotonergic Interneuron Involves an Increase in the Fraction of Transmitter Release. American Physiological Society. doi: 10.1152/jn.00702.2007
“A clear and accessible introduction to statistics, perfect for beginners. This book covers both the old and the new – giving students the fundamentals they need to understand their field, while equipping them with a more sophisticated understanding of the pros and cons of those established practices. The focus on open science and integration with statistical tools (e.g., JAMOVI) makes the book particularly useful for training future researchers.“
That’s the endorsement of our forthcoming second edition by Simine Vazire. Bob and I are, of course, enormously appreciative of her generous words. I recently posted (here) about her appointment as the incoming Editor-in-Chief of Psychological Science–a fantastic development imho.
The bunch of balloons on the right sketch Simine’s main research interests. Who better to write about Open Science and what’s needed to help us all–and our students–do better science? Thanks Simine.
“If I were teaching introductory statistics to undergraduates, this is the textbook I’d use. The things that make it distinctive are first, the focus on estimation rather than p-values (though the latter are covered), second, the link with free open-source software that allows users to explore analyses and visualisations, and third an emphasis on open science practices, coupled with red flags and examples of DFY (Don’t Fool Yourself!). There are plenty of exercises, quizzes, and take-home messages, which will bring the material alive even for the most maths-phobic students.”
…so writes Dorothy Bishop about our second edition. Bob and I are hugely grateful, and chuffed, to have her generous words.
Dorothy is Emeritus Professor of Developmental Neuropsychology at Oxford University, and one of the tiny number of scholars so broadly accomplished that they can list FBA, FMedSci, FRS after their name, meaning being a Fellow of The British Academy, The Academy of Medical Sciences, and the Royal Society.
She writes an award-winning blog, numerous tweets (@deevybee), and, as “Deevy Bishop“, several detective novels.
She has long written about meta-science issues. Search her blog and find many interesting reads. She was a co-author of A manifesto for reproducible science. Since “retirement” she has become even more involved in meta-science, as recounted here.
You can see why Bob and I are so happy that she felt able to write so positively about the new edition of ITNS.
Geoff
P.S. Replying to the copy-editor is done, page proofs expected in a couple of weeks. The Routledge page for ITNS2 is here, 21 March the publication date. Note one error: full colour, not “B/W”.
Computational reproducibility is a mouthful but the idea is simple: Use the reported data analysis to get the reported results. Haven’t we always assumed that works? Yes, but.
Search for “computational reproducibility” and find recent discussions in numerous disciplines. Conclusion: It’s often very difficult and we need new practices and tools.
Steve Lindsay is campaigning for psychology journals to do more to ensure that reported results do indeed arise, as claimed, from the data. His guest editorial in Meta-Psychology is here.
Steve, a highly experienced journal editor, has long worked to advance Open Science. (See my post here.) He makes a cogent argument as he discusses practical ways to achieve computational reproducibility without simply expecting already over-worked voluntary expert reviewers to do the checking.
Large version
Open Code badge, small
Good news is arrival of a new Open Science badge to signal that an article provides full details of the data analysis including openly available analysis script or code.
“Is metascience, and promotion of Open Science practices, the most urgent and consequential field for all of science at present? Maybe yes.” I wrote that about the latest AIMOS meeting (here). Now a leading metascientist is the incoming Editor-in-Chief of Psychological Science, arguably psychology’s flagship journal. Rejoice!
Psychological Science leading
Eric Eich probably first brought statistics reform and Open Science to many researchers’ consciousness with his famous editorial Business Not As Usual in Jan 2014. Following the editorial was my The New Statistics: Why and How, commissioned by Eric. Submitting authors suddenly needed to do things very differently–in a good way. Eric kept the advances coming.
Steve Lindsay, next in the job, kept up the progress. He modestly summarized the advances he and Eric had made in Swan Song Editorial at the end of 2019. I posted about that here. Progress has continued since.
Simine
In June 2019 Simine posted had i been editor in chief–her Vision Statement submitted as part of selection of the previous Editor-in-Chief of Psychological Science. It remains a fascinating read. It struck me at the time as about the strongest application anyone could possibly make. I won’t speculate how she might have revised her statement this time round but, happily, it did the job.
Simine now lives in Sydney with her newish husband Alex Holcombe, also a psychologist and metascientist. As noted above, Simine’s appointment is at the University of Melbourne. When in Melbourne she often stays with Fiona Fidler, her co-director of the MetaMelb Lab.
Join with me in wishing Simine all strength in her new role as Cajoler-in-Chief of us all to keep pushing for better.
