DEVELOP PROTOCOL AND SEARCH CRITERIA

Once you have formulated your questions, you need to write up a protocol that will (1) formally specify the questions you intend to ask, (2) specify an objective search strategy, and (3) establish study inclusion criteria (Davis et al., 2021; Gurevitch et al., 2018; Koricheva et al., 2013).

Specifying the questions you intend to ask will include being specific about potential sources of heterogeneity in effect sizes (Côté & Jennions, 2013).

Specifying an objective search strategy entails not biasing data collection toward relevant papers you are already familiar with; otherwise, this could affect the outcome as it is often easier to remember papers with significant results. Primarily, this involves making a list of which electronic databases you will search and what search terms you will use. Secondarily, an objective search strategy will include deciding how much effort is required to search for “gray literature.” For example, will you write to colleagues asking for unpublished data, and if so, who and why?

Once you have conducted a search and compiled a list of potential papers, you must establish study inclusion criteria. These criteria are often fairly obvious and include the following:

1. Does the study fit the scope of your questions?
2. Does the methodology fit with how your question is defined?
3. If so, was it of sufficient magnitude or duration?
4. Does the study contain extractable data? Is there sufficient information to extract an effect size, variance, and the sample size used?
5. Your inclusion criteria will sometimes have to consider study quality. This is far more difficult to assess than the above criteria, but it can be just as important.

It is worth noting, however, that as in primary research, your protocol for searching the literature and extracting effect sizes will almost certainly be modified as you proceed. The reality is that, in many respects, your final protocol will end up describing what you did rather than what you ideally wanted to do. First, you must tell the reader how you collected your data. So, just as in primary research, you provide the reader with enough information on data collection and analysis to allow your review to be repeated and updated in the future. Second, you must have a protocol that forces you to evaluate continually whether your sampling is biased.

A protocol increases the objectivity with which you compile data, but it should not blind you to the reality that the process of meta-analysis involves numerous subjective decisions; these are most apparent when trying to decipher the results of a given paper and deciding whether you can extract the necessary data for your synthesis (Côté & Jennions, 2013). If more than one person collects the data, a well-described and tested protocol ensures uniformity in data extraction and coding decisions about moderators.