The Diamond Ratio (DR), Our Estimate of Heterogeneity: Accepted for Publication

I’m excited to report that Max Cairns’s PhD work on the Diamond Ratio (DR) has been accepted for publication by the British Journal of Mathematical and Statistical Psychology. The preprint of the final accepted version is here.

The Original DR Blog Post

…was in 2018: Measuring Heterogeneity in Meta-Analysis: The Diamond Ratio (DR)

It includes a brief intro to the Fixed Effect (FE) and Random Effects (RE) models for meta-analysis, and to heterogeneity. When there’s heterogeneity, the diamond depicting the 95% CI on the result of an RE meta-analysis is likely to be longer than the FE diamond.

The DR is simply the ratio of those two diamond lengths. If DR = 1 there’s little or no heterogeneity; DR of, say, 1.5 suggests moderate heterogeneity. Larger DR, more heterogeneity.

Bob and I introduced the DR in Chapter 9 of ITNS. We wanted to discuss heterogeneity, while avoiding the complexities of conventional estimates of heterogeneity (Q, I², Τ²). If both the RE and FE diamonds are displayed at the bottom of a forest plot, the DR can easily be eyeballed (see figure below).

A CI on the DR

Last year I posted about Max’s success in developing a very good approximate CI on the DR:

A Confidence Interval for the Diamond Ratio: Estimation of Heterogeneity in Meta-Analysis

The Paper Accepted for Publication

It’s easy to eyeball the DR—the ratio of the lengths of the two red diamonds in this figure from the preprint:

The preprint describes Max’s investigations of seven (!) approaches to calculating a CI for DR. These are all approximations, but, as the preprint reports, Max carried out extensive simulations to evaluate all of these. His main focus was on coverage. He found that the ‘Sub-Q’ approach has excellent coverage (very close to 95%) across a very wide range of situations.

Max’s short description of this CI is that it is the “Substitution CI with the Q-profile τ² interval estimator”. No, that’s not totally clear to me either, but see the preprint for the full story, including the impressive (imho) range of simulation results. There are links to all the data and results, and to R software for calculating the CI on DR.

The red line below the RE diamond in the figure represents the length of the 95% prediction interval (PI) for the population effect sizes estimated by different individual studies. In other words, it indicates the likely extent of spread of these true effect sizes. It’s a further way to visualise the likely extent of heterogeneity. The length of the PI is 0.29, as reported in the figure.

esci in jamovi

Bob has included calculation of the CI on DR in the current beta of esci in jamovi, available here.

Visualisation is Understanding

Well, very often it’s a big help, and not only for beginning students. Visualisation is a focus all through esci and ITNS. We hope DR visualisation will help students and even researchers achieve a better understanding of heterogeneity.

As usual, it’s highly valuable to have a CI as well as a point estimate—in this case, of heterogeneity. Unless k, the number of studies in the meta-analysis, is quite large, the CI on DR is likely to be long, as it is in the figure above. That CI extends from the minimum, 1, to more than 3. With only k = 10 studies, all we can say is that heterogeneity is most likely between zero and very large. In other words, the amount of heterogeneity could be pretty much anything. That’s unfortunate, but it’s better to know this than be misled by any seemingly precise point estimate.

Well done Max! (And supervisor, Luke.)

Geoff