Resources from SFN’s “Ask an Expert” event on power and estimation

On 6/4/2020 the Society for Neuroscience (SFN) hosted an an “Ask and Expert” online webinar on power and estimation.  Short presentations on the topic were first posted here as a catalyst to generate questions and comments to discuss during a live session.

This post is just to organize links and resources that might be helpful to those who watched the Q&A (thanks! hope it was helpful).  Before listing those, I (Bob) should say that it was a great honor to have been invited to be part of an event with a giant of meta-science, Katherine Button, and a brave advocate for better science, eNeuro editor Christophe Bernard.  It is also heartening for this to be an SFN initiative.  I still feel like the organization doesn’t do quite enough on these issues (especially at the annual meetings), but this was another positive step forward.

First, here are the handful of slides I put together for the Q&A session.  In these slides I show data from Manouze et al. (2019) .  You can grab this data file yourself in csv format here; this is just the data on anxiety for the socially isolated mice.  Email Christophe for the complete data set.

Second, Christophe mentioned the “Dance of the p values”.  These are simulations Geoff created to help researchers explore the realities of sampling variation.  These now exist in a variety of formats:

• Here’s the Dance of the p values simulation I used in today’s session.  It’s in Excel.  You’ll need to enable macros, and it doesn’t run great on all versions of Excel.  All credit to Geoff-this is the version that accompanied his book Understanding the New Statistics.  If you want a guided tour of this sim, check out Geoff’s short video on YouTube: https://www.youtube.com/watch?v=5OL1RqHrZQ8
• Want an online version?  Here are two versions of the Dance of the Means (a bit simpler than Dance of the p values):
  • A really sharp-looking one created by Kristoffer Magnusson: https://rpsychologist.com/d3/ci/
  • An a fabulous new version by Gordon Moore: https://www.esci.thenewstatistics.com/esci-dances.html
• Want a version for students?  Here’s the Dance of the Means as an Excel spreadsheet with an accompanying set of activities for undergraduate education.

Finally, here’s a list of resources relevant to the questions that were submitted.

Software for Estimation:

• For confidence intervals, a great option is our new esci module for jamovi.
  • Download and install a current version of jamovi (>1.2.21)–it’s free and open source
  • Open jamovi, and in the modules tab, access the jamovi library.  Under “available modules” scroll through until you find esci.  Then click “install” to add the module into jamovi.
  • A new esci menu will appear that allows you to generate estimates for most common study designs.  You can bring raw data into jamovi, or you can often just type in summary data (useful if you’d like to get an estimate from a published paper)
  • More details are here, tutorials and examples are in the works, and please feel free to send feedback and/or feature requests.
• For bootstrapped intervals, DABEST is fantastic.  It is available as an R package, a python package, and as a easy-to-use web app.
• For Bayesian estimation:
  • The best way to get started it to buy Kruschke’s Doing Bayesian Data Analysis.  It is remarkably clear and has extensive examples.
  • the BEST package in R is great.  In addition to R, you’ll need to install jags (but this is pretty easy).  This tutorial is especially helpful.  There is also a package in development called Bayesian First Aid which makes BEST easier to apply to a bunch of common designs–it doesn’t seem to have been updated for a while, but I find it still useful.
  • JASP can help you get some Bayesian estimates and is very easy to use.  You can’t use diffuse priors—which I think the developers would tell you is a feature rather than a bug.

Alternatives to planning for Power – a few sources to get started

• Planning for precision
  • Rothman, K. J., & Greenland, S. (2018). Planning Study Size Based on Precision Rather Than Power. Epidemiology, 29(5), 599–603. https://doi.org/10.1097/EDE.0000000000000876
• Planning for evidence
  • Schönbrodt, F. D., & Wagenmakers, E.-J. (2017). Bayes factor design analysis: Planning for compelling evidence. Psychonomic Bulletin & Review, 1–16. https://doi.org/10.3758/s13423-017-1230-y
• And stay tuned as I (Bob) will hopefully soon have an online course out with lots of resources on sample-size determination.

Getting the lay of the land in statistical inference:

• Frequentist Estimation
  • Cumming, G., & Calin-Jageman, R. J. (2017). Introduction to the new statistics: Estimation, open science, and beyond. New York: Routledge.
• Bootstrap Estimation
  • Hesterberg, T. C. (2015). What Teachers Should Know About the Bootstrap: Resampling in the Undergraduate Statistics Curriculum. American Statistician, 69(4), 371–386. https://doi.org/10.1080/00031305.2015.1089789
• Bayesian Estimation
  • Kruschke, J. K., & Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25(1), 178–206. https://doi.org/10.3758/s13423-016-1221-4
• When testing, do better with inference by interval
  • Lakens, D. (2017). Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses. Social Psychological and Personality Science, 8(4), 355–362. https://doi.org/10.1177/1948550617697177