Get Started

Here are some handy resources to get you started with the New Statistics

Learning

  • Of course, we think Introduction to the New Statistics is a great source for learning the New Statistics approach.  But there are lots of other really good resources out there.
  • To learn the Bayesian approach to the New Statistics (which we heartily recommend):
  • Learn about estimation from one of the masters:
    • Statistics with Confidence (Smithson, 2000)
    • Confidence Intervals (Smithson, 2003)
  • And here are some helpful papers on understanding effect sizes and/or meta-analysis:
    • A great primer on effect sizes, the subtle differences between effect sizes for different designs, etc:  (Lakens, 2013)
    • An excellent tutorial on effect sizes and how to report them (Pek & Flora, 2017)
    • The bible of effect sizes and meta-analysis: (Cooper, Hedges, & Valentine, 2009)

Software:

  • ESCI is a free set of Excel spreadsheets that can help provide New Statistics analysis and figures for many common research designs.  It is available for download directly from this website: ESCI
  • ESPSS is a free set of plugins for SPSS to provide New Statistics output for common research designs.  This project is still early in development, but you can find modules for 2-group designs and correlations.  The GitHub page is here: https://github.com/rcalinjageman/ESPSS
  • R is the Swiss Army Knife of statistics, and you can certainly accomplish New Statistics analysis with R.  It can be a bit tricky, though, to get R to do all you’d like.  Fortunately, David Erceg-Hurn has put together a guide to the New Statistics in R.  You can download the guide here: https://osf.io/rmn8p/
    • The ITNS package by David Erceg-Hurn can make the cool plots that show a difference axis, a nice way to show the effect size of interest for a two-group comparison
    • The userfriendlyscience package in R has lots of cool functions for New Statistics analysis and outpu
  • Jamovi is a promising free, open-source statistics program that provides a user-friendly interface for R.  We don’t have any specific New Stats commands for Jamovi yet, but hope to develop some soon.
  • Jasp is a free, open source statistics program with an emphasis on Bayesian analysis.  It will produce credible intervals for many types of designs, the Baysian version of a confidence interval.

Sample-Size Planning (Planning for Precision)

  • ESCI provides worksheets for planning two-group designs for precision
  • Some other good planning for precision tools are:
  • For the Bayesian approach, see Doing Bayesian Data Analysis (J. Kruschke K., 2014)
  • There is a fantastic set of articles on planning for precision by Ken Kelley and associates
    • A general overview of the approach (Maxwell, Kelley, & Rausch, 2008)
    • For regression analysis (Kelley & Maxwell, 2003)
    • For mediator analysis (Kelley, 2007)

Meta-Analysis:

  • ESCI provides some pretty good tools for meta-analysis.  It enables meta-analysis of raw scores or standardized effect sizes.  It enables subgroup analysis.
  • Open Meta Analyst is a free, open source GUI for the metafor package.  It provides meta-analysis for several different types of designs, subgroup analysis, and even meta-regression.  It’s pretty easy to use, though it doesn’t seem to be in development any longer.
  • Meta Essentials is a free Excel workbook that makes many types of meta-analysis straightforward.
  • The metafor package for R is amazing.  It’s not very hard to use, and there is a very lively listserv where you can pose questions.  It’s worth just rolling up your sleeves and learning how to use it.

Open Science:

  • The Open Science Framework is your one-stop shop for pre-registration, data sharing, publishing your file drawer and more.
    • The Open Science Framework has developed a pre-registration template that walks you through the pre-registration process. You can download this template here.
    • If you’d like to see what a completed template looks like, here is an example that plans a replication of a study from Ottati et al. (2015).
    • At of 2017 you can also win $1,000 for pre-registering a study with OSF and then publishing the study in a participating journal.
  • Some of the initiatives towards Open Science going on in cognitive neuroscience (Gilmore, Diaz, Wyble, & Yarkoni, 2017)

 

Copy of ESCI for IU demo: ESCI-intro-chapters-10-16-IU

References

Cooper, H., Hedges, L., V., & Valentine, J., C. (2009). The Handbook of Research Synthesis and Meta-Analysis. Russell Sage Foundation.
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017). Progress toward openness, transparency, and reproducibility in cognitive neuroscience. Annals of the New York Academy of Sciences, 1396(1), 5–18. https://doi.org/10.1111/nyas.13325
Kelley, K. (2007). Sample size planning for the coefficient of variation from the accuracy in parameter estimation approach. Behavior Research Methods, 39(4), 755–766. https://doi.org/10.3758/bf03192966
Kelley, K., & Maxwell, S. E. (2003). Sample Size for Multiple Regression: Obtaining Regression Coefficients That Are Accurate, Not Simply Significant. Psychological Methods, 8(3), 305–321. https://doi.org/10.1037/1082-989x.8.3.305
Kruschke, J., K. (2014). Doing Bayesian Data Analysis. Academic Press.
Kruschke, J. K., & Liddell, T. M. (2017). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-016-1221-4
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4. https://doi.org/10.3389/fpsyg.2013.00863
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample Size Planning for Statistical Power and Accuracy in Parameter  Estimation. Annual Review of Psychology, 59(1), 537–563. https://doi.org/10.1146/annurev.psych.59.103006.093735
Pek, J., & Flora, D. B. (2017). Reporting Effect Sizes in Original Psychological Research: A Discussion and Tutorial. Psychological Methods. https://doi.org/10.1037/met0000126
Smithson, M. (2000). Statistics with Confidence. SAGE.
Smithson, M. (2003). Confidence Intervals. SAGE.