10.3 Introducing meta-analysis

The preceding section has shown you why vote-counting on the basis of significance is such a bad idea, and probably given you an inkling of how meta-analysis models work. In their simplest form, meta-analysis models combine estimates of the same parameter from multiple datasets in order to produce an overall effect size, that integrates all the available evidence. In the calculation of the overall effect size, estimates are weighted according to their precision, with more precise estimates (generally coming from studies with larger sample sizes) having greater influence than less precise estimates. The model also gives a confidence interval for the overall effect size, which is generally smaller than the confidence interval for the parameter estimate of any of the constituent studies.

Getting an overall effect size is the most basic thing you can do with a meta-analysis. You can also use the model to investigate heterogeneity in effect size (do the studies’ effect sizes differ from one another more than you would expect by chance given their sample sizes?); and also to ask whether any characteristics of the individual studies - their methodology or measurement for example - predict what size of effect they find (see 10.6). Here, we are just going to cover the very simplest use case.

In order to carry out a metaanalysis, all you need is the following two things:

• A vector of effect sizes, one from each of your datasets or studies. The effect sizes can be in a number of formats: parameter estimates (either standardized or not); differences in means (usually standardized); correlation coefficients; or odds ratios. They capture the magnitude of effect or association for the phenomenon of interest from each study. Obviously, they all have to represent measurements on the scale. You cannot compare an effect size in terms of numbers of centimetres growth to another effect size in terms of millimetres of growth. They also all need to be estimates of the same type. If however the different analyses you are drawing from produced different types of estimate (for example, a standardized mean difference (SMD) in one case and a odds ratio in the other), don’t despair. There are formulas available for the conversion of many types of effect sizes into a common currency like the correlation coefficient. The R package effectsize will make many of these conversions for you.

• A vector of precisions, one for each of your effect sizes. The easiest way of capturing precision is the standard error of each parameter estimate (or, the square of the standard error, which is known as the sampling variance). You can find these in the output from lm(), lmer() and glm() objects in R, or reported in papers writing up empirical studies. An alternative way of representing the precision is to know the lower bound and upper bound of the 95% confidence interval for each of your parameter estimates. You can convert between these two formats by remembering that the difference between a parameter estimate and either bound of its confidence intervals is generally 1.96 times its standard error. (Though, be careful when applying this rule of thumb for odds ratios; there it applies on the log odds scale, but ceases to apply when you convert from log odds to odds, as people often do when reporting their results).

Armed with these two pieces of information, it is very easy to run a meta-analysis, as we will see in the following examples. There are a number of contributed R packages, but I recommend the one called metafor. There are two cases where you might use meta-analysis: where you are drawing the different estimates from several papers in the literature; and where you are wanting to integrate the information from several of your own studies. We will work through an example of each.