9.3 The place of sensitivity in your analysis strategy
Specification curve analysis often involves many more specifications than the 24 used in the example above, sometimes even thousands. It is a very handy way of systematically exploring the robustness of conclusions, and being transparent about their limits. The key step in doing a specification curve analysis is working out what your possible universe of specifications is. It can only include specifications that are still tests of the same research question. Sometimes in a study when we report an additional specification, it is not as additional test of the same question, but as an approach to a novel follow up question. For example, we might be wanting to know about a mediator. These kinds of question-changing analysis are not in the same set for specification curve analysis, because they are not tests of exactly the same question.
Also, your set of specifications for a specification curve analysis should only include specifications that actually seem reasonable. There is no interest in showing that hundreds of bad specifications that no reasonable person would ever propose lead to null conclusions. Rather, you are trying to identify the set of specifications that you (or another researcher) might reasonably have thought to try in pursuit of the same question.
There is one school of thought for data analysis, sometimes called multiverse analysis, where your main analysis involves all reasonable specifications, from the get-go (Steegen et al., 2016). Although I absolutely understand the rationale, I don’t usually do this. Quite apart from anything else, it is easier for the reader to understand your analysis if you start with just one model for them to look at. So I generally define a primary analysis in the pre-registration, the one I think a priori would be the right model. I present this as the centrepiece of the Results section. Generally, simplicity is to be preferred for the primary specification. Then, either in a subsequent section of the Results, or in a supplementary appendix, I present a sensitivity analysis, to show to what extent I might have come to a different conclusion had my strategy for the primary analysis been different. In very simple cases (e.g. where there is one alternative specification), you don’t need to use the machinery of specr to do this; but where there are a few choices, the number of specifications rapidly becomes large, and specr becomes a great choice.
It’s important to pre-register what your primary analysis is going to be. Indeed, one of the benefits of pre-registration is it allows for transparency about whether you are presenting just the specification that you have discovered makes your result look best, or whether you genuinely intended to use this particular specification. It’s also good to pre-register alternative specifications you are going to look at in your sensitivity analysis. However, there are very often cases where you only think of these specifications once you are doing the data analysis, or they are pointed out to you by referees. In this case they won’t be in the pre-registration, but you should still include them in the sensitivity analysis anyway, making clear in the text whether they were thought of before or after pre-registration.
In general, I am quite permissive about including extra, non-anticipated specifications into the sensitivity analysis; more sensitivity information is better. On the other hand, I am quite averse to changing my primary analysis (even under pressure from referees). This is because, if you discover that a different specification is going to give you a nice significant p-value, whereas your pre-registered primary does not, you are going to struggle not to make the switch. So to some extent, you just have to bind yourself to the mast of always doing what you said you were going to do. There are exceptions - you discover the data look quite unlike what you expected, or you realise you made a betise in thinking about your strategy. In this case, departure from your primary analysis can be justified (Lakens, 2024), but you must signal transparently in your paper that you are doing it (see section 12.2.1).