Category: Teaching

Teaching the Forest Plot–What Do You Think?

I’ve been a bit obsessed with the forest plot for, I’d guess, close to 20 years. Partly because I love pictures, partly because the forest plot can tell us so much. I regard it as the beautiful face of meta-analysis. These days it’s probably the beautiful face of Open Science. Consider this forest plot, which is part of Figure 9.7 from ITNS.

The data are from Calin-Jageman and Caldwell (2014) and we discuss the example on pp. 239-243 of ITNS. The first six studies are from one lab and are estimates of the extent to which various superstitious beliefs can enhance performance. Even a quick glance raises concerns–there seems to be insufficient variability from study to study and, what’s more, they all achieve statistical significance, mostly be a small amount.

The picture of those six studies alerts us to possible p-hacking and/or selection of results. The six results simply look too good, from the traditional NHST perspective in which p values matter.

The last two studies, by Bob’s group, are preregistered replications, carefully designed to be as similar as possible to the top study, but with larger samples. There is a strong suggestion that we have a clear case of failure to replicate, and the effect is actually of negligible size–although we can’t, of course, rule out the possibility of some unknown moderator that accounts for the seeming difference between the first six and the last two studies.

So, with a little extra information (e.g., about preregistration or otherwise) the forest plot can be highly informative, and summarise what a research literature has to say on a research question of interest.

Back around 2001 I had built the first version of ESCI meta-analysis. I decided that meta-analysis is sufficiently important, and the forest plot sufficiently simple, to include it in my intro statistics & design course. My beginning first year undergraduates (i.e., freshmen) thus encountered the forest plot–and meta-analysis–about two months in.

It turned out to be a great teaching moment for me. Not very often (in my experience anyway) do students say ‘ok, that all makes sense, no big deal’. Being newbies, they couldn’t worry that no textbook included meta-analysis, that back then many researchers regarded it as the province of technical experts, and that journals were only beginning to publish meta-analytic reviews.

After developing ESCI and my teaching approach for a couple of years I presented a paper at ICOTS7, the International Conference on Teaching Statistics, in Brazil in 2006. My paper is titled META-ANALYSIS: PICTURES THAT EXPLAIN HOW EXPERIMENTAL FINDINGS CAN BE INTEGRATED and is here.

Reading that short and simple paper now is nostalgic for me. It foreshadows many of the main issues we discuss in Chapter 9 in ITNS, although of course not Open Science and preregistration.

I’m hoping that teaching meta-analysis, probably by way of simple forest plots, is now becoming widespread. The simplest forest plot in ITNS is Figure 1.4 which appears as early in the book as p. 11.

In our symposium at APS next May my contribution is titled Open Science Is Best Practice Science, with an emphasis on teaching. One thing I plan to talk about is the value of the forest plot for presenting an introduction to meta-analysis and Open Science.

I’m therefore keen to hear from anyone with thoughts about, or experience of, using forest plots in the intro statistics and/or design course. If you would care to, please make a comment below to this post. Many thanks!

Geoff

Calin-Jageman, R. J., & Caldwell, T. L. (2014). Replication of the Superstition and Performance Study by Damisch, Stoberock, and Mussweiler (2010). Social Psychology, 45(3), 239–245. https://doi.org/10.1027/1864-9335/a000190

See you in San Francisco?

I’m delighted to report that our symposium proposed for the APS Convention next May has been accepted. (BTW if you visit that site for the Convention, note the happy people in the top centre pic–our panel from the 2017 Convention.)

The Convention will be in San Francisco, 24-27 May 2018.

Our symposium:
Title: Open Science and Its Statistics: What We Need to Teach Now
Chair: Bob Calin-Jageman–who no doubt will say a bit about his own successful teaching experiences with the new statistics.
Speakers:
Susan A. Nolan and Kelly M. Goedert, Seton Hall University:
Transitioning a Traditional Department: Roadblocks and Opportunities for Incorporating the New Statistics and Open Science into Teaching Materials
Tamarah Smith, Cabrini University: Feeling Good about the New Statistics: How the New Statistics Improves the Way Researchers and Students Feel about Statistics
Geoff Cumming, La Trobe University, Australia: Open Science Is Best Practice Science

Symposium Abstract
Open Science and the new statistics promise research that is more ethical and trustworthy, but education needs to change. This change has begun. We will share successes in transitioning to the new statistics from a range of perspectives, with a particular focus on teaching, allowing ample time for discussion.

_____________
The main reason we proposed this symposium is to report and discuss experiences teaching Open Science and the new statistics. We really do hope to hear from people coming to the symposium what their experiences have been. Personally, I’m convinced that it’s way easier and more satisfying to teach the new approaches than the old–especially NHST. Students simply find the new ways more coherent and appealing.

To what extent do others share my experience and opinion? How can we do better? What are the best teaching strategies? What materials provide the best support for students and teachers?

I’m hanging out for answers to questions like these. I’m hoping next May to hear lots of interesting things at our symposium. You might consider coming along?

Geoff

Open Science and The New Statistics: Be Happy!

In May this year at the APS Convention in Boston, Bob and I ran a symposium with the title From NHST to The New Statistics: How do We Get There? It was great fun. There is a summary here.

Next year the APS Convention will be in San Francisco in May. The invitation for submissions is here. Go to that page and see this pic of four happy people:

Would you know it–that’s a pic of our symposium panel at APS this last May! From the left: Steve Lindsay, Editor-in-chief of Psychological Science; Susan Nolan, author of a number of textbooks, including some fine statistics texts; Tamarah Smith, statistics teaching guru; and Bob. I’m not in the pic–I’m probably giving my presentation, perhaps making some sweeping outrageous remarks about NHST or p values at the moment here frozen in time?

I’m thinking that this pic is evidence that APS judges our symposium to be the happiest moment of the 2017 Convention!

Indeed, a big reason for jumping in to teach the new ways is that it’s simply more fun–as well as more successful and satisfying–than teaching as we’ve done for decades past.

Geoff
P.S. We have submitted a proposal for a follow-up symposium for APS next May to discuss case studies of changing to the new ways. Still under review. Watch this space.

Video – Getting started with the New Statistics and Open Science

This fall I (Bob) was invited to give a talk at Indiana University as part of a series on good science and statistical practice organized by the university’s Social Science Research Commons (SSRC).  The SSRC is like a core facility for getting advice on statistics and experimental design…what a cool thing for a university to have!

I really enjoyed my visit (thanks Emily, Cami, and Patricia)–good conversation with fascinating people in a beautiful setting.  The series has a video archive, so my talk is now posted online as a video and as a powerpoint. Here’s the link–take a look if you want to know more about how to get started using Open Science practices and the New Statistics:  https://media.dlib.indiana.edu/media_objects/gt54kp23k

The New Statistics for Neuroscientists

This summer I (Bob) was asked to write a series of perspective pieces on statistical issues for the Journal of Undergraduate Neuroscience.

My first effort has just been published (Calin-Jageman, 2017)–it is a call for neuroscience education to shift away from p values, and an explanation of the basic principles of the New Statistics with an example drawn from neuroscience.

It turns out that the paper was published just before the annual meeting of the Society for Neuroscience, which I am currently attending.  It’s been very gratifying to see the paper is already sparking some discussion.

Here’s the key figure from the paper comparing/contrasting the NHST approach with the New Statistics approach with data from a paper in Nature Neuroscience.

References

Calin-Jageman, R. (2017). After p Values: The New Statistics for Undergraduate Neuroscience Education. Journal of Undergraduate Neuroscience Education, 16(1), 1–4. Retrieved from http://www.funjournal.org/wp-content/uploads/2017/10/june-16-e1.pdf

Beyond p values – Dispatches from the ASA symposium on statistical inference

The next couple of posts will be about my experience at the ASA conference on statistical inference: A World Beyond p < .05.

The first session featured Steve Goodman and John Ioannidis (who Skyped in from Australia).  One highlight was Goodman’s explanation of why p values continue to be so prevalent.  He argued that p values are like a currency–we can trade them in for useful things (grants, papers, promotions).  Their value lies primarily in our common belief in them, more than from their specific mathematical underpinnings.  Good analysis.

 

The joy of many disciplines

One of the great things about working in psychology, or statistics, or–just imagine!– both, is that you can get to play in the backyards of many other folks. As science becomes more and more fragmented, and many researchers feel that their best strategy is to aim for expertise in some highly specialised sub-field, it’s worth taking a moment to enjoy cross- or multi-disciplinary research.

I’ve been lucky enough, over the decades, to publish journal articles with colleagues in computer science, linguistics, education, cell biology, philosophy, ecology, artificial intelligence, statistics, history & philosophy of science, health sciences, and maybe one or two other disciplines that don’t spring to mind right now.

In UTNS, my first book about the new statistics, I included boxed examples from numerous disciplines to illustrate the main argument that anyone using NHST, in whatever discipline, should think hard about making the change to TNS. Maybe as a result I still get emails–almost always positive, and often highly enthusiastic–from folks from a stunning breadth of backgrounds. Recent examples include someone working on assessment of risk in financial markets and another in a ministry for justice.

In ITNS, our introductory book, we mainly focus on psychology–including numerous sub-fields–and education, but we’ve included some examples from wider afield. We certainly believe that our arguments and methods are widely applicable across many disciplines. So we’re delighted when teachers in other disciplines express interest in ITNS.

My most recent example is archaeology. I gave an invited talk to the Archaeology Department at La Trobe University. My slides for the talk are here.

It was a lucky fluke that I had recently posted about archaeology and Open Science, so I could include some archaeology examples. I’m happy to say that the response was highly positive. I was asked specifically about chi-square, so could immediately open the Two proportions page of ESCI intro chapters 10-16, plug in the numbers for an archaeology example, then display the chi-square analysis and our recommended better way based on proportions.

Bob and I dearly hope that ITNS will be of use to folks in lots of disciplines. We love hearing from anyone interested in using ITNS, and particularly so from teachers or researchers outside psychology.

Geoff

Pictures of uncertainty: Dancing with Pierre in Paris

A while back I wrote a post about Pierre Dragicevic, an HCI researcher in Paris who for years has been working to persuade researchers in his field to adopt better statistical methods.

I wrote about his wonderful talk that presents lots of different dances–not only of means, but p values, and of CIs, dichotomous decisions, and several others things. (At that link, scroll down a little to ‘Materials’ for download of the slides, list of references, and more.) Each dance is a picture of uncertainty or, rather, a movie of how uncertainty is represented for successive samples in a simulation.

Click here to see all the dances in action.

He recently let me know that he’d given the talk again. You can see this latest talk here. At about 4.50, note the great quote from Andrew Gelman:

“Statistics has been described as the science of uncertainty. But, paradoxically, statistical methods are often used to create a sense of certainty where none should exist.”

That’s the heart of the NHST vs estimation question: a p value can easily give a seductive but illusory sense of certainty, but a CI puts the extent of uncertainty in our faces.

After watching the talk again, I mentioned to him that he might have brought out more strongly the advantage of CIs over p values that a CI gives us a useful indication of its dance–CI length indicates the general width of the dance–whereas a p values says almost nothing about the dance from which it comes. He agreed, but noted that the implications of dances need to be learned and, in his experience, students can have difficulty building good intuitions about dances.

That’s a good point. In ITNS Bob and I discuss CI dances, and interpretation of a CI in terms of how its length tells about what’s likely to happen on replication. We have found this approach to work very well with students, but I’d love to see empirical investigations of how effective our approach to teaching the new statistics is, in practice. Anyone up for the challenge of doing that?

Geoff

From NHST to the New Statistics — How do we get there?

APS just wrapped up.  Geoff and I were privileges to help host a symposium on making progress moving the field away from p values towards the New Statistics.  Our co-conspirators were fellow text-book author Susan Nolan, Psychological Science editor Stephen Lindsay, and stats teaching wizard Tamarah Smith.  We each offered our perspectives on some of the road-blocks to abandoning the safety blanket of the p value.  The session was lively, with great audience feedback and discussion.  I’ve posted here each speakers’ slides:

  • Introduction by Geoff Cumming — a quick recap of the long history of calling for the end of the p value, and encouragement to make this time the time we really make the change. Slides are here.
  • The Textbook Writers Perspective by Susan Nolan — Susan reviewed some of the inertia holding back substantive change in statistics textbooks (both in her excellent textbook and for others).  She also offered hopeful insight from the sea-change that has occurred in the teaching of projective texts in clinical psychology.   Slides are here.
  • The Instructors’ Perspective by Tamarah Smith and Bob Calin-Jageman.  Tamarah and I considered the surprising complexities of making changes in the undergraduate statistics curriculum.  We reviewed some promising software tools for making the change (JASP, Jamovi, and a new set of extensions for SPSS) and discussed some of the strategies we had found helpful in incorporating the New Statistics into the classroom.  Slides are here.
  • The Editor’s Perspective by Stephen Lindsay.  Stephen discussed some of the very substantive policy changes implemented at Psych Science beginning with prior editor Eric Eich.  These include an emphasis on effect sizes, requirements for sample-size justification, and badges for different open-science practices.  Although these stringent requirements reduced submissions somewhat, they are having a real impact, hopefully towards better replicability.  Slides are here.

Some useful resources and papers discussed during the symposium:

  • Getting Started in the New Statistics – An set of links, guides, and resources for teaching the New Statistics.  Hosted via the Open-Science Framework.  A crowd-sourced effort; request to be an editor to join in.  https://osf.io/muy6u/
  • JASP – free, open-source alternative to SPSS.  Focuses on Bayesian statistics, but makes it pretty easy to use confidence intervals with most analyses and graphs.  https://jasp-stats.org/
  • JAMOVI – another free, open-source alternative to SPSS.  Also does pretty well with confidence intervals on some types of analyses.  https://www.jamovi.org/
  • ESPSS – a set of extensions to get SPSS to do New Statistics well. Still under development, but feel free to live on the bleeding edge.  Currently supports independent samples t-test, paired samples t-test, and correlations.  https://github.com/rcalinjageman/ESPSS
  • A fascinating paper Geoff contributed to examining the impact of the APS guidelines on different publication practices: (Giofrè, Cumming, Fresc, Boedker, & Tressoldi, 2017).  (full link below)
  • Stephen’s most recent editorial enjoining new standards for data sharing in Psychological Science (Lindsay, 2017) (BRAVO!) (full link below)
  • Stephen’s earlier editorial about replication and standards at Psychological Science (Lindsay, 2015).
  • The famous “Business not as usual” editorial from Eric Eich that got things moving in a great direction at Psychological Science (Eich, 2014). (full link below)
  • Susan Nolan’s excellent undergrad statistics textbook with Richard Heinzen: http://www.macmillanlearning.com/catalog/Product/statisticsforthebehavioralsciences-thirdedition-nolan
  • The APA’s 2.0 revision of Undergraduate Learning Goals for Psychology majors, which describes goals related to statistical/scientific thinking for psychology majors: http://www.apa.org/ed/precollege/about/psymajor-guidelines.pdf

References

Eich, E. (2014, January). Business Not as Usual. Psychological Science. SAGE Publications. https://doi.org/10.1177/0956797613512465
Giofrè, D., Cumming, G., Fresc, L., Boedker, I., & Tressoldi, P. (2017, April 17). The influence of journal submission guidelines on authors’ reporting of statistics and use of open research practices. (J. M. Wicherts, Ed.), PLOS ONE. Public Library of Science (PLoS). https://doi.org/10.1371/journal.pone.0175583
Lindsay, D. S. (2015, December). Replication in Psychological Science. Psychological Science. SAGE Publications. https://doi.org/10.1177/0956797615616374
Lindsay, D. S. (2017, April 17). Sharing Data and Materials in Psychological Science. Psychological Science. SAGE Publications. https://doi.org/10.1177/0956797617704015

from the APS Convention in Boston

Bob and I are in Boston this weekend for the annual APS Convention. It’s great to catch up, and discuss a million things about ITNS and this blog, and our future plans. Our publisher told us yesterday that early signs from the field are super-encouraging, which is great. It can of course be a big job to persuade your colleagues to adopt a different approach to intro statistics, but people are taking on this challenge. So the future for teaching OS and the new statistics is bright.

This year there aren’t as many sessions on Open Science issues as in the past 3 years, but still lots of goodies. There’s a real buzz about MPPS (or AIMPIPS!), the new APS journal, which should really help push OS forward.

Bob and I ran a symposium yesterday. The room was packed–standing room only–by close to 100 people. Our focus was on the challenges of putting TNS and OS into practice, and strategies that can help. After our 4 brief presentations we had more than 30 minutes of excellent discussion. Many, many people see the need and are working on how they can, in practice, get real change.

Below is the outline. Bob will post the 4 sets of slide shortly. Watch this space…

Geoff