Measuring Heterogeneity in Meta-Analysis: The Diamond Ratio (DR)

This is a post about the Diamond Ratio (DR), a simple measure of the extent of heterogeneity in a meta-analysis. We introduced the DR in ITNS. But first, some background.

Fixed Effect (FE) model for meta-analysis
The diamond at the bottom of the forest plot picturing a meta-analysis reports the overall point estimate and its 95%CI. If there’s not too much study-to-study variation in the results–in other words if the forest plot looks rather like a dance of the CIs arising from exact replications–then we’re done. The simple Fixed Effect (FE) model, which assumes the studies are all estimating the same population effect size, is probably reasonable. If so, we say the studies are homogeneous.

Below is Figure 9.2 of ITNS, showing a FE meta-analysis of 10 studies. (The ESCI Meta-Analysis file used to make this figure is a free download by following the links from here.)

Random Effects (RE) model
However, it’s usually unrealistic to assume the studies are homogeneous. If there looks to be rather more study-to-study variation in the forest plot, then the studies may be heterogeneous and we need the Random Effects (RE) model, which assumes that the different studies may be estimating somewhat differing population effect sizes.

Below is Figure 9.3 of ITNS, showing the same studies as in Figure 9.2 above, but this time a RE meta-analysis. In ESCI, click between the two radio buttons at bottom left to switch between FE and RE models. The most obvious change is that the RE diamond is longer than the FE diamond. In fact about 40% longer. The Diamond Ratio (DR) is the ratio of the RE diamond length to the FE diamond length. Here, the value of the DR is 1.40, as reported centre bottom in both figures.

Measures of heterogeneity
The conventional measures of heterogeneity in a meta-analysis are Q, I-squared, and tau-squared. In Chapter 8 of my first book, UTNS, I give the formulas, explain how the three inter-relate, and try to explain what they mean. It’s all a bit complicated, and I’ve always felt that these measures don’t really give a good intuition of heterogeneity. For ITNS, the introductory book, we needed something much simpler and more intuitive.

The radio buttons in ESCI make it easy to click between FE and RE models, and to watch as the diamond jumps between shorter and longer–when there is heterogeneity. The ratio of these two lengths seemed a simple way to express the amount of difference between the two models, and so ESCI reports the value of the DR. If DR= 1.0 the two models give the same result, suggesting there is little or no heterogeneity–the studies could easily be homogeneous. The larger the DR, the greater the heterogeneity. Values above around 1.5, and especially values approaching or exceeding 2.0, suggest considerable heterogeneity.

Heterogeneity and Moderators
Heterogeneity need not be a nuisance. If we can find a moderator that accounts for a usefully large part of the heterogeneity, then we may have made a discovery–we may be able to answer a research question that no single one of the separate studies in the meta-analysis can address. To take a simple example, maybe some of the studies used only females, and others only males. If gender can account for a large part of the heterogeneity in the results of the separate studies, we may have made a useful discovery.

Moderator analysis can only identify correlation, not causality. But identifying a moderator can lead to theoretical advances, and help guide research fruitfully. Even beginners should be able to appreciate the value of heterogeneity and moderators in meta-analysis. We attempt to explain all that, using simple examples with forest plots, in Chapter 9 of ITNS.

Studying the DR
The DR seemed an appropriately simple and intuitive measure of heterogeneity for ITNS, but what are its properties? I did a few simple simulations, and found the DR seemed to behave sensibly, but I couldn’t get far when I tried to investigate the underlying maths.

Fortunately, my savvy statistics colleague at La Trobe University, Luke Prendergast was interested in the DR. He has been supervising Max Cairns, a PhD student, who has been investigating the DR.

A CI on the DR!
They recently invited me to La Trobe to discuss progress. It turns out that the DR, under another name, has received some (favourable) attention in the tekkie research literature. Max has now taken that previous work notably further. He reports that the DR does seem to behave well, and to be related in a sensible way to the conventional measures of heterogeneity. The big breakthrough is that Max has found a way to calculate a CI on the DR. That’s great news, and a big advance! His simulations suggest that his CI behaves well for a wide variety of situations.

There is more work to be done, then Luke and Max plan to write up their DR findings. We may together prepare a version for a psychology audience. They may also develop an online tool, to make it easy for researchers to enter their data, then get not only the DR but also the CI on that DR. Which would be wonderful!

It will take a while, and the outcomes are not guaranteed, but progress so far is highly promising. Well done Max (and Luke)! Best of luck with the next stages. I shall report further in due course.

Meanwhile we can all read Chapter 9 of ITNS and appreciate that the DR is a legitimate indicator of the extent of heterogeneity. Yay!

Geoff