Get Started
Here are some handy resources to get you started with the New Statistics
Learning
- Of course, we think Introduction to the New Statistics is a great source for learning the New Statistics approach. But there are lots of other really good resources out there.
- To learn the Bayesian approach to the New Statistics (which we heartily recommend):
- (J. K. Kruschke & Liddell, 2017)
- Doing Bayesian Data Analysis, 2nd edition (J. Kruschke K., 2014)
- Learn about estimation from one of the masters:
- Statistics with Confidence (Smithson, 2000)
- Confidence Intervals (Smithson, 2003)
- And here are some helpful papers on understanding effect sizes and/or meta-analysis:
- A great primer on effect sizes, the subtle differences between effect sizes for different designs, etc: (Lakens, 2013)
- An excellent tutorial on effect sizes and how to report them (Pek & Flora, 2017)
- The bible of effect sizes and meta-analysis: (Cooper, Hedges, & Valentine, 2009)
Software:
- ESCI is a free set of Excel spreadsheets that can help provide New Statistics analysis and figures for many common research designs. It is available for download directly from this website: ESCI
- ESPSS is a free set of plugins for SPSS to provide New Statistics output for common research designs. This project is still early in development, but you can find modules for 2-group designs and correlations. The GitHub page is here: https://github.com/rcalinjageman/ESPSS
- R is the Swiss Army Knife of statistics, and you can certainly accomplish New Statistics analysis with R. It can be a bit tricky, though, to get R to do all you’d like. Fortunately, David Erceg-Hurn has put together a guide to the New Statistics in R. You can download the guide here: https://osf.io/rmn8p/
- The ITNS package by David Erceg-Hurn can make the cool plots that show a difference axis, a nice way to show the effect size of interest for a two-group comparison
- The userfriendlyscience package in R has lots of cool functions for New Statistics analysis and outpu
- Jamovi is a promising free, open-source statistics program that provides a user-friendly interface for R. We don’t have any specific New Stats commands for Jamovi yet, but hope to develop some soon.
- Jasp is a free, open source statistics program with an emphasis on Bayesian analysis. It will produce credible intervals for many types of designs, the Baysian version of a confidence interval.
Sample-Size Planning (Planning for Precision)
- ESCI provides worksheets for planning two-group designs for precision
- Some other good planning for precision tools are:
- The userfriendlyscience package in R
- This great tool for complex designs by Gerben Mulder
- For the Bayesian approach, see Doing Bayesian Data Analysis (J. Kruschke K., 2014)
- There is a fantastic set of articles on planning for precision by Ken Kelley and associates
- A general overview of the approach (Maxwell, Kelley, & Rausch, 2008)
- For regression analysis (Kelley & Maxwell, 2003)
- For mediator analysis (Kelley, 2007)
Meta-Analysis:
- ESCI provides some pretty good tools for meta-analysis. It enables meta-analysis of raw scores or standardized effect sizes. It enables subgroup analysis.
- Open Meta Analyst is a free, open source GUI for the metafor package. It provides meta-analysis for several different types of designs, subgroup analysis, and even meta-regression. It’s pretty easy to use, though it doesn’t seem to be in development any longer.
- Meta Essentials is a free Excel workbook that makes many types of meta-analysis straightforward.
- The metafor package for R is amazing. It’s not very hard to use, and there is a very lively listserv where you can pose questions. It’s worth just rolling up your sleeves and learning how to use it.
Open Science:
- The Open Science Framework is your one-stop shop for pre-registration, data sharing, publishing your file drawer and more.
- The Open Science Framework has developed a pre-registration template that walks you through the pre-registration process. You can download this template here.
- If you’d like to see what a completed template looks like, here is an example that plans a replication of a study from Ottati et al. (2015).
- At of 2017 you can also win $1,000 for pre-registering a study with OSF and then publishing the study in a participating journal.
- Some of the initiatives towards Open Science going on in cognitive neuroscience (Gilmore, Diaz, Wyble, & Yarkoni, 2017)
Copy of ESCI for IU demo: ESCI-intro-chapters-10-16-IU
References
Cooper, H., Hedges, L., V., & Valentine, J., C. (2009). The Handbook of Research Synthesis and Meta-Analysis. Russell Sage Foundation.
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017). Progress toward openness, transparency, and reproducibility in cognitive neuroscience. Annals of the New York Academy of Sciences, 1396(1), 5–18. https://doi.org/10.1111/nyas.13325
Kelley, K. (2007). Sample size planning for the coefficient of variation from the accuracy in parameter estimation approach. Behavior Research Methods, 39(4), 755–766. https://doi.org/10.3758/bf03192966
Kelley, K., & Maxwell, S. E. (2003). Sample Size for Multiple Regression: Obtaining Regression Coefficients That Are Accurate, Not Simply Significant. Psychological Methods, 8(3), 305–321. https://doi.org/10.1037/1082-989x.8.3.305
Kruschke, J., K. (2014). Doing Bayesian Data Analysis. Academic Press.
Kruschke, J. K., & Liddell, T. M. (2017). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-016-1221-4
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4. https://doi.org/10.3389/fpsyg.2013.00863
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation. Annual Review of Psychology, 59(1), 537–563. https://doi.org/10.1146/annurev.psych.59.103006.093735
Pek, J., & Flora, D. B. (2017). Reporting Effect Sizes in Original Psychological Research: A Discussion and Tutorial. Psychological Methods. https://doi.org/10.1037/met0000126
Smithson, M. (2000). Statistics with Confidence. SAGE.
Smithson, M. (2003). Confidence Intervals. SAGE.