Meta-analysis | Introduction to the New Statistics

‘The New Statistics’ (2013) Wins Sage 10-Year Impact Award

The New Statistics: Why and How (abstract below) explained the advantages of moving on from NHST to the new statistics (estimation and meta-analysis) and the need for better practices to improve research integrity. I’m delighted that an award from Sage indicates the article seems to be helping researchers improve what they do. Next: Can ITNS2 help the next generation do even better?

The article appeared online in late 2013, so was considered when Sage examined the citation numbers of all articles appearing in any of the 400+ journals Sage published back in 2013. It was one of the top three most cited, so has been given a Sage 10-Year Impact Award. Sage’s announcement is here. Sage has just published a blog post about it with headline:

Now for the abstract:

The article was commissioned by Eric Eich, then editor-in-chief of Psychological Science, to appear immediately following his famous editorial Business Not As Usual. This opened the Journal’s first issue of 2014 and announced sweeping changes in the journal’s submission requirements, which, for many psychologists, marked the arrival of Open Science.

Interview

Sage’s blog post includes an email interview with me. Here’s a brief summary:

What was it in your own background that led to your article?

When I was a teenager my father gave me a simple explanation of significance testing. I said something like “That’s weird, sort of backwards. And why .05?” He replied “I agree, but that’s the way we do it.”

Over decades of teaching I became ever more dissatisfied with NHST, and focused ever more on confidence intervals (CIs).

Was there an article that had a particularly strong influence on you?

Frank Schmidt (1996) wrote: “It is now possible to use meta-analysis to show that reliance on significance testing retards the development of cumulative knowledge.” A revelation!

What did Schmidt’s article lead to?

About 2003 I started using an Excel forest plot to give a simple explanation of meta-analysis in my intro course. I was delighted: Students told me it just made sense. Of course, for meta-analysis you need a CI from each study, while p values are irrelevant, even misleading.

Figure: Dances of means, confidence intervals, and p values.

In 2009 I uploaded a video of the dance of the p values. I became passionate about advocating the new statistics (estimation and meta-analysis). I wrote Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis (UTNS, 2012).

What was happening in psychology at about that time?

Ioannidis (2005) explained how reliance on NHST was a major cause of the replication crisis. Largely in response to that crisis, Open Science arrived—perhaps the most important advance in how science is done for a very long time.

Eric Eich’s famous editorial Business Not As Usual in the January 2014 issue of Psychological Science marked the arrival of Open Science in psychology. Months earlier Eric had invited me to write a tutorial article to support the changes he wanted. This was The New Statistics: Why and How and was published immediately following his editorial.

What has been the reception of the article?

Mainly very positive. Some have felt I went too far in advising that in most cases it’s better not to use NHST at all. Some Bayesians have been unhappy with the focus on confidence intervals.

Revisiting that article, what would you have done differently?

I used the term ‘research integrity’, but ‘Open Science’ was coming into use and I soon realized that was way better. Reading the article today, for ‘research integrity’ read ‘Open Science’.

Otherwise, I think the article has held up well, including all 25 guidelines in Table 1.

What has happened since

Psychological Science has continued to lead in the adoption of Open Science practices.

Meta-science, also known as meta-research, has emerged and now thrives as a highly multi-disciplinary field. It applies the scientific method to improve that method—wonderful!

What have you been doing since?

I teamed with Robert Calin-Jageman to write the first intro statistics textbook based on the new statistics and with Open Science all through. The second edition has just come out: Introduction to The New Statistics: Estimation, Open Science, and Beyond, 2^nd edition (ITNS2, 2024). It has much improved software, as we explain in Calin-Jageman & Cumming (2024), which is on open access.

We believe this book can sweep the world—we’ll see! To read the Preface and Chapter 1 go to www.thenewstatistics.com. In the second para is a link to the book’s Amazon site. Click ‘Read sample’.

References

Calin-Jageman, R., & Geoff Cumming, G. (2024). From significance testing to estimation and Open Science: How esci can help. International Journal of Psychology, https://doi.org/10.1002/ijop.13132

Cumming, G. (2012). The New Statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.

Cumming, G. (2014) The new statistics: Why and how. Psychological Science. 25(1), 7-29. https://doi.org/10.1177/0956797613504966

Cumming, G., & Calin-Jageman, R. (2024). Introduction to The New Statistics: Estimation, Open Science, & Beyond, 2^nd edition. New York: Routledge.

Eich, E. (2014) Business not as usual. Psychological Science, 25(1), 3–6. https://doi.org/10.1177/0956797613512465

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine 2: e124. https://doi.org/10.1371/journal.pmed.0020124

Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1(2), 115-129. https://doi.org./10.1037/1082-989X.1.2.115

Vale Bob Rosenthal, Statistical Reform Leader and Much Else

I was much saddened to read of the death last month of Bob Rosenthal. See this obituary; and another in the New York Times.

I met him first in 1996 when I called on him at Harvard to discuss statistical reform. What a gentle, encouraging, and thoroughly nice person! What a giant intellect! He loved nothing better than to find innovative solutions to tricky problems.

Considering statistical reform and Open Science:

He was an early proponent of a focus on effect sizes, especially his favourite, Pearson correlation, r.
He was a pioneer of meta-analysis and identified what he called the file-drawer effect.
Rosenthal and Gaito (1963) reported evidence that researchers’ confidence in an effect drops sharply as the p value increases past .05; they labelled this the cliff effect. This was an early example of statistical cognition–the empirical study of how people understand statistical concepts and reports. We still need much more of that, imho.
Around 2009 Jerry Lai wanted to investigate the cliff effect as part of his PhD. He sent Bob a very polite request for any further information about the original study. Promptly, back came an encouraging message to Jerry and a scan of several hand-written pages of the original data. From almost 50 years earlier! A wonderful example of Open Data (well, available data), with no excuses about hard disk crashes and superseded storage formats.
He advocated analysis of well-chosen contrasts as better than the customary reliance on Anova and p values (*, **, ***, or ns) to interpret omnibus main and interaction effects. He stated that “the problem is that omnibus tests … do not usually tell us anything we really want to know”. Contrast Analysis: Focused Comparisons in the Analysis of Variance (1985) by Rosenthal and Rosnow remains an accessible and powerful explanation. UTNS, and both editions of ITNS take this planned contrast approach (these days, with preregistration) to the analysis of complex designs.
In 2008 Fiona Fidler and I were working on Confidence Intervals : Better Answers to Better Questions. We sent a draft to Bob who was working on an accompanying article Effect Sizes : Why, When, and How to Use Them. Bob responded with enthusiasm, saying he loved our article and also offering valuable suggestions.

Bob’s nickname among his students was “Prof ARRRZZZental“, recognising his love of correlation r.

I salute his memory and his enduring contribution to improving how we do things.

Geoff

Geoff’s Stats Passions: A BJKS Podcast

When Benjamin Kuper-Smith kindly invited me to chat with him for his podcast I warned him he’d have trouble shutting me up. Maybe Ben felt that, but I felt we had a pretty interesting chat about lots of great (imho) stats issues. The podcast is here.

There’s an auto-generated transcript, and you can hover just below the moving sound line to see a control of audio replay speed–I find x1.25 or even x1.5 can be good.

Timestamps
0:00:00: A brief history of statistics, p-values, and confidence intervals <what, half an hour is ‘brief‘?>
0:32:02: Meta-analytic thinking
0:42:56: Why do p-values seem so random?
0:45:59: Are p-values and estimation complementary?
0:47:09: How do I know how many participants I need (without a power calculation)?
0:50:27: Problems of the estimation approach (big data)
1:00:08: A book or paper more people should read
1:02:50: Something Geoff wishes he’d learnt sooner
1:04:52: Advice for PhD students and postdocs

You can see my podcast is #82. Ben told me he was about to chat with Brian Nosek, I’m sure that will be a great podcast coming in a week or two. A few others I found especially interesting:

80. Simine Vazire: scientific editing, the purpose of journals, and the future of psychological science

55. Angelika Stefan: p-hacking, simulations, and Shiny Apps

54. Jessica Kay Flake: Schmeasurement, making stats engaging, and the Psychological Science Accelerator

53. Chris Chambers: Registered Reports, scheduled peer-review, and science without journals

13. Joe Hilgard: Scientific fraud, reporting errors, and effects that are too big to be true

Have a browse at BJKS. Happy listening,

Geoff

The Latest on esci: Geoff’s Talk to Turku

Yesterday I greatly enjoyed a zoom talk with the Open Science Community of Turku, Finland. Online were good folks from Finland, The Netherlands and I don’t know where else. Great questions at the end!

My slides and 3 data files (download the lot as a zip) are at osf.io/uhwj2 — you may need to log in to OSF but registering is simple.

The talk is up on YouTube here. I start at about 5.00 and esci at about 39.20.

I moved rapidly through dances, significance roulette, the plausibility curve on a confidence interval, esci web, Open Science, … to what’s new: the second edition of ITNS, due March ’24, AND the latest on esci.

Starting at Slide 61 there’s step-by-step detail of how to instal jamovi, instal esci into jamovi, and see the wide range of analyses esci offers, with examples. Plenty of guidance for anyone to get started using Bob’s great esci application. All open and free.

The Open Science Community of Turku boasts members from many disciplines and has the wonderful mission of “making Open Science the norm in research“. Way to go!

My thanks to Lydia Laninga-Wijnen, also Oskari Lahtinen, for the invitation and hosting.

Geoff

AIMOS: Correcting the Record… Will Keep You Awake at Night!

“It will keep you awake at night!” said Fiona Fidler about the Correcting the Record session. I’d zoomed in to some of AIMOS 2022 (the Meta-science conference in Melbourne last week; my post is here) but had missed that session. The video (here) is now online. It’s very much worth watching. I don’t expect to sleep for days 🙁

Four alarming and impressive speakers are listed above and mentioned below. Or search the AIMOS program (here) for the speaker names for more.

John Loadsman

Editor-in-Chief of Anaesthesia and Intensive Care. Several submitted manuscripts every week show clear signs of fraud. Some are readily identified with a careful read. Plagiarism is rife and can be hard to detect. Chasing fraud now takes a large part of his time. His suggested tools:

Ben Mol

“About 30% of the RCTs in Women’s Health are fabricated” (Weep!) Fabricating authors can be hard to hold to account, and they can attack back. We need better ways to identify and counter fraud. Are the data true? It can be hard or impossible to obtain original data.

Lisa Parker

Fraud is common, alas. Paper mills are companies that generate and sell fake manuscripts. Much but not all fraud is fairly easy to spot. She interviewed researchers to gather signs of fraud and is working towards a simple tool to help identify fraud. Others are using machine learning to develop such a tool.

Jana Christopher

Catching fraudulent and plagiarised images (Western blots, photos of tumors…) Exposing fraud can be a very long and frustrating process. Publishers can be obstructive, legalistic. Her suggested tools:

Finally

Yes, I was horrified at the extent of the problem–an enormous criminal enterprise polluting our research literature–often involving life-or-death issues. Is meta-science, and promotion of Open Science practices, the most urgent and consequential field for all of science at present? Maybe yes.

That video again. I hope you sleep well… eventually,

Geoff

For the Latest Science About Science Hurry to AIMOS 2022

Online and in Melbourne, 28-30 Nov. A vast spread of disciplines. From around the world, numerous young folks–and some oldies–speaking truth to age. Yes, it’s AIMOS 2022. The fourth AIMOS conference! See my posts after the first conference (2019, here) and third (2021, here).

Registration

Register for AIMOS 2022 here. For online only it’s free. Or come along live and the social events alone more than justify the modest cost.

The Program

It’s here–scroll down a bit. It’s still being updated, more goodies to come.

Plenaries, hackathons, discussion groups, lightning talks, workshops, working papers, mini-note panels…

Psychology, neuroscience, economics, medicine, ecology, law, philosophy, evolution, history,…

Open Science, metascience, metaresearch, meta-what-you-will…

IMHO the program already looks fantastic, with more yet to come.

Enjoy,

Geoff

Knowing Statistics Can Lead You To the Heart of Government

…even to being a key financial Minister in Federal Parliament.

After the recent Australian election, Labor took power. (Hooray!) The new Assistant Minister for Treasury is Andrew Leigh. He’s one of the small team responsible for all things economic and financial, at the heart of Government.

I’ve posted about him before–in 2018 when he published Randomistas, and when he spoke at AIMOS in Melbourne in 2019 (see the highlights para).

Leigh is a Harvard-trained economist and former professor of economics. In Randomistas he argues we should use randomised trials much more often to guide public policy. Imagine, policy guided by high quality evidence! He’s well aware of the replication crisis and Open Science practices needed for trustworthy research.

In his AIMOS talk he discussed replication, His best one-liner: “If at first you DO succeed, try, try and try again.” Hey, meta-analysis!

Leigh is quoted in a recent interview as saying that he’s probably “the biggest stats nerd” the Australian Bureau of Statistics has had as its minister in its 116-year history. (Nerd can be good!)

It’s a great time for stats nerds and policy makers: First results from the 2021 Australian census are just being released.

Lots of us will be watching anxiously to see how much influence evidence, statistics, and Leigh will have on Labor’s policy making.

Meanwhile, be encouraged–knowing statistics can lead to great things in life.

Geoff

MRI Workshop Videos, Including Two Short ‘p Values Suck’ Talks

I recently posted about MRI Together. It was a great global zoomfest, and now videos of the talks are online.

The Videos

The YouTube site with all the videos is here, but it’s easiest to scan the program and click on any talk title to go to the video.

It’s clear that many in the MRI community have been working on Open Science issues for a while. It was great to see lots of talks about software for statistical analysis of scans, and the challenges of making analyses reproducible–and of finding good ways to make code and the highly complex data sets open, and readily usable by others.

I’ll mention just a few videos below.

An Intro

Cassandra Gould van Praag gives an engaging intro talk here. She encountered a range of OS issues when her PhD examiners required modifications to her thesis. She bravely tells the story of the work and revisions she had to undertake. She must have taken the lessons to heart: Her current job is to develop and promote the open science infrastructure of Oxford Neuroscience.

Two ‘p Values Suck’ Talks

Valentin Amrhein gave a rousing talk titled ‘P Values and the Replicability of Results‘. The video is here. He has some well-chosen graphics and some striking graphs, for example of the wide range of likely p values in various situations. You won’t be surprised to hear that I heartily agree with almost everything he said.

My talk was titled The New Statistics for Reproducible Science The video is here. My three take-home messages were:

p values are highly unreliable, never trust them
Adopt Open Science practices, planned analyses
esci, Bob’s new software for estimation, meta-analysis, teaching, and more, is available now (in beta), and gaining thousands of installs each month. It includes great graphs with confidence intervals. It’s a pleasure to teach with it.

Brian Nosek Rounds It Off

Talk title: Publishing and Sharing Open Science. The video is here. Brian does a typically neat and persuasive job, of course with lots of evidence.

Lots more gems to discover: Just browse the program.

Geoff

AIMOS: Recorded Session Gems Now Online

Bob tweeted about it, I posted about it, and recently it happened: The AIMOS zoomfest. The program is here.

AIMOS, just 3 years old, advances meta-science, a big part of which is the scholarship and advocacy that advances Open Science. This is the third conference, and now, supported by a generous donation, planning is in full swing for the establishment of a Journal of Metascience–to be open, of course. (I don’t think that journal title has been settled yet.)

Many sessions were recorded and are now online. Below are a few gems–there are many more worth exploring, at the YouTube channel. It’s great to hear of OS advances and advice from law, economics, medicine… and many more.

Brian Nosek: The Story so far here

Brian, ED of the Center for Open Science, describes the massive OS progress these last ten years. His abstract is a neat summary:

Scholarly studies of science are as old as science itself. The last 10 years of metascience has a unique character. Compared with other scholarly treatments, it is more data-driven, collaborative, grounded in the discipline it studies, applied, interventionist, and activist. Metascience studies how the system works, proposes how it should work, develops interventions to change how it works, and evaluates whether those interventions are having the desired effects. The last 10 years reflects the emergence of metascience as a scholarly activity not just to understand science but to improve it.

Rose O’Day: We Need to get on With Getting Stuff Done here

Brian gave us an informative welcome. Rose book-ended the meeting by sending us off energised to actually get real change done. She was given the online equivalent of a standing ovation.

Bob Explaining The New Statistics to Improve Statistics Education

Alas, the recording system slipped up, so no video. Bob did a rapid-fire and very neat job. His slides are here.

Meta-Analysis: the Latest here

Three minitalks, then discussion. Highlights for me included: Tari Turner (1:40 – 14:00) on how ‘live meta-analysis‘ provided constantly updated coronavirus evidence to guide ever-changing public policy these last two years. And Shinichi Nakagawa (14:40 – 29:40) on meta-analysis of variances, with diverse great examples of why this can be vitally important.

Assessing the Risk of Bias in the Studies in a Meta-Analysis

Alas again, no recording for this workshop run by Matt Page. We heard about his ROB-ME tool, and in small groups used it for a couple of examples. My take-away: It’s not possible to avoid considerable subjectivity in assessing bias. The only solution is for all researchers to follow the full Open Science list of better practices (of course). But it’s a tool worth investigating; it may be the best we have. Read about it here and here.

How to Start a Revolution in Your Discipline

A terrific discussion, again with no recording. Lots of hand-wringing about how hard it is to get take-up of OS practices. Dorothy Bishop responded by advocating a focus on small philanthropic funding bodies, for which every dollar really matters. Therefore they, more than any other stakeholder, are likely be most responsive to the argument that OS practices can greatly increase the cost-effectiveness of research.

WikiJournals, and Bridging From Journals to Wikipedia here (see 29:20 – 36:00)

Pages in Wikipedia typically get 1,000 times as many views as a journal article–or more. I hadn’t known about the many efforts to find some sort of integration. Some journals have rules about authors posting an update or new page to Wikipedia. Wikijournals have appeared. The experiments continue. Fascinating stuff from Thomas Shafee, of La Trobe.

What does damage to scientific progress look like? here

Time for three cold showers to enrage us, and energise our future efforts towards OS and research integrity. It’s chaired by the indefatigable David Vaux (Davo), distinguished fraud detective and campaigner for an Australian Research Integrity body–against strong university opposition. (Shame!)

Other AIMOS gems

There are lots of other gems to discover. That YouTube site? It’s here.

Happy digging,

Geoff

MRI Analysis, Now With Open Science

It’s fabulous to see yet one more research field, MRI and fMRI, jumping on board with Open Science. The Workshop runs 13-17 December, 2021, and the site is here.

Recall the dead salmon? (tiny.cc/deadsalmon, ITNS p. 485.) Back then, in 2009, a common way to analyse fMRI data relied on p values for each of many thousands of voxels. Apply this analysis to a dead salmon shown two different types of pictures and find part of its (totally dead) nervous system lit up on the analysis screen! It was a massive Type I error, of course, caused by inadequate correction of p values given by the gazillion voxel comparisons. Make a more appropriate correction and all we see is noise. The study deserved its 2012 IgNobel Prize.

Analysis of MRI data has come a long way since 2009, partly prompted by the dead salmon. More recent discussions, for example here and here, include consideration of the basic Open Science issues, including p-hacking, cherry-picking, unplanned analyses, and lack of replication.

The Timetable (Program)

It’s here. Near the top, select your timezone. There are sessions around the 24 hours, grouped to fit the waking hours in Atlantic, Pacific, Indian, and Caribbean zones. It looks to me like a wonderfully broad take on Open Science and Reproducibility. I recognised only a few of the presenters, including:

Valentin Amrhein: P-values and the replicability of results

Brian Nosek: Publishing and Sharing Open Science

I’m giving a brief (20 min, including Q+A) talk: The new statistics for reproducible science in a session titled Study design and interpretation. The reproducibility crisis, running 11.00 to 13.00 on 15 Dec, those times being UTC+11, which includes Sydney and Melbourne. I’ll post my slides in due course.

Geoff