The Multiverse! Dances, and More, From Pierre in Paris

Our Open Science superego tells us that we must preregister our data analysis plan, follow that plan exactly, then emphasise just those results as most believable. Death to cherry-picking! Yay!

The Multiverse

But one of the advantages of open data is that other folks can apply different analyses to our data, perhaps uncovering interesting things. What if we’d like to explore systematically a whole space of analysis possibilities ourselves, to give a fully rounded picture of what our research might be revealing?

The figure below shows (a) traditional cherry-picking–boo!, (b) OCD following of fine Open Science practice–hooray!, and (c) off-the-wall anything goes–hmmm.

That fig is from a recent (in fact, forthcoming) paper by Pierre in Paris and colleagues. The paper is here, and the reference is below at the end (Dragicevic et al., 2019). The abstract below outlines the story.

The Multiverse, Live

Pierre and colleagues not only discuss the multiverse idea in that paper, but here they give neat interactive tools that allow any reader of several example papers to do the exploration themselves. Hover the mouse, or click, to explore the outcome of different analyses.

EMARs

I suggest Sections 4 and 5 in the paper are especially worth reading. Section 4 discusses what’s called explorable multiverse analysis reports (EMARs), with a focus on mapping out just what a rich range of possibilities there often are for alternative analyses.

Then Section 5 grapples with the (large) practical difficulties of building, reviewing, and using EMARs, with the aim of increasing insight into research results. Cherry-picking risks need always to be at the forefront of our thinking. Preregistration of certain proposed uses of an EMAR could be possible, with possibly somewhat reduced cherry-picking risks.

Play Multiverse on Twitter

Matthew Kay, one of the team, gave a great overview in 8 posts to Twitter. See the posts, and a bunch of GIFs in action here.

Enjoy!

Geoff

Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, Fanny Chevalier. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. CHI 2019 – The ACM CHI Conference on Human Factors in Computing Systems, May 2019, Glasgow, United Kingdom. 2019, <10.1145/3290605.3300295>.