A Confidence Interval for the Diamond Ratio: Estimation of Heterogeneity in Meta-Analysis

A while back I posted (here) about the Diamond Ratio (DR), which is our simple visual indicator of the extent of heterogeneity in meta-analysis. (See ITNS, Chapter 9 for more on the DR.) I reported that Max Cairns, a PhD student in statistics at La Trobe University, and his supervisor Luke Prendergast were working on finding a confidence interval (CI) for the DR.

I’m delighted to report that they have now posted a preprint of their results here. We’d love to have your comments and suggestions.

Max explored six approaches to calculating a CI for the DR. He used simulation to investigate their properties, especially coverage, and identified two that give excellent CIs. He provides (here) R code to allow any researcher to calculate the CI on the DR for their own data, for a range of measures. All Max’s simulation materials are available on OSF here, so anyone can recreate or extend Max’s work.

Below is Figure 1 from the preprint, as an example of how the DR and its CI may be reported in a forest plot.

In the figure, DR = 1.40 is reported along with three conventional measures of heterogeneity, all with CIs. Both the RE (Random Effects) and FE (Fixed Effect) diamonds are shown in the forest plot, so it’s easy to eyeball DR, which is simply the length of the RE diamond divided by that of the FE diamond. DR = 1 suggests little or no heterogeneity, and increasing values of DR suggest increasing heterogeneity. One vital message is given by the CI on the DR, which is [0, 3.09], so this meta-analysis, which integrates only 10 studies, can give us only a very imprecise estimate of heterogeneity.

Along with the DR, the figure reports the 95% prediction interval (PI) for true effect sizes as a further estimate of heterogeneity. Borenstein et al. (2017) advocated use of the PI, which is reported here to be 0.285. The red line segment just under the RE diamond pictures that length. Informally, that segment illustrates the likely extent of spread of true effect sizes. The PI is 4 x T, where T is the estimated population SD of true effect sizes. The very long CI reported for T indicates once again a very imprecise estimate of heterogeneity.

In the preprint we conclude that the DR, and its CI, can be valuable for students as they learn about meta-analysis, and for researchers as they interpret and communicate their meta-analyses.

It would be great to have any comments about Max’s work and the preprint. Thanks!

Geoff

Max: mrcairns994@gmail.com Geoff: g.cumming@latrobe.edu.au

Borenstein, M., Higgins, J. P., Hedges, L. V., & Rothstein, H. R. (2017). Basics of meta-analysis: I² is not an absolute measure of heterogeneity. Research Synthesis Methods, 8, 5-18. doi:10.1002/jrsm.1230