Some Questions–Would You Care to Comment?
Blogs sometimes either elicit lots of comments, or they don’t. While writing, I’m always wondering how readers might react, what they (you) might be thinking. In my most recent post I asked about experiences or thoughts readers might have about teaching meta-analysis, especially in the intro course and especially using the forest plot.
On average something approaching 100 people a day come to this site. So it’s a good bet that there are some interesting experiences and observations out there. Please feel encouraged to share, if you would care to, using the comment box below. You could react to any of our posts, or raise some new topic, or even ask your own question(s).
Here are just a few of the issues I’d love to hear about:
1. As I mentioned, the teaching of the forest plot.
2. Experiences with ITNS and the ITNS materials in the classroom, or for self-teaching.
3. JASP and/or jamovi. Each of these is a free, open source software package, currently in development but available for use now. Each aims to be highly usable, offer a wide range of data analysis facilities, and serve as a free replacement for SPSS. I’d love to hear of any experiences with either of these, especially along with ITNS. (We are in touch with the developers of both JASP and jamovi. We hope that jamovi, at least, will be developed to include new-statistics facilities, as in ITNS. Anyone interested in helping with this?)
Hoping to hear from you; thanks!
Gwen, you are very welcome, and thanks for your enthusiasm! You may pick up that Bob and I are pretty passionate, not to say obsessed, with encouraging the world to do better statistics. And in finding ways to make the new statistics visible, intuitive, comfortable, natural…
Alas, these videos are on YouTube, but maybe you would be interested in the dance of the p values, and/or (2 more recent videos) significance roulette. Easiest way to find them is at YouTube to search for those topics. May the book and materials serve you well,
I just happened across one of your videos embedded in another page and ended up here–I almost never watch Youtube if I can avoid it, but I’m stunned by your ESCI files! I thought I had just about seen it all in Excel, but this is something new. I’m excited to poke around in it, thank you for making this publicly available!
See pp 239-243 in ITNS, including Figures 9.7 and 9.8. We conclude that “…the later evidence throws strong doubt on the initial… finding that a mention of luck is likely to give a substantial increase in performance”. The m-a result of around 0.5 looks like an unhelpful mixture of apples and oranges. The fact that the last two studies were preregistered is one strong reason for thinking they are more likely to be giving good estimates of reality.
Thanks for the reply. I was referring to the forest plot shown in your article.
Even Though as you said the results of the 6 studies could be due to tphackingand/or selection of results, the point estimate shows 0.5 – a moderate effect size. So superstitious beliefs really enhance performance??
Thanks for your message and suggestions. Bob and I have been considering whether we should develop a Facebook, and possibly Twitter, presence. Yes, that’s the way the world seems to be moving.
If meta-analysis gives an overall estimated ES of 0.5 then that’s the best estimate of the true ES that the research integrated by the m-a can give us. We could note that, by Cohen’s convention, that’s a medium-sized effect. Better, as we explain in ITNS, is to interpret an effect of that size, in its particular research context. It’s often best to do that interpretation in the original units, not the standardised units of Cohen’s d. Consider the practical import of an effect of that size, compare with other effects if that makes sense. Theoretical import? Of course, interpret the ES in the context also of the CI on that m-a result. How much uncertainty remains?
Much overlap of CIs is good news–the results of the various studies included in the m-a are giving a similar story. However, if there is too much overlap we might suspect p-hacking or selection of the studies we’ve found to include in the m-a.
Hi Dr. Cummings,
You should come up a facebook page. Now most people comment on facebook and less on blogs. Something that I have noticed recently.
And there a few psych groups on facebook which will you give a you a lot of feedback. One I know is PsychMap
The article on meta-analysis: My question is the overall effect size is 0.5. So how do you interpret the overall ES. And the CI of the large studies overlap the small studies. So what does it mean?