DATA COLLECTION

All co-authors must gather data for meta-analysis yourselves; it cannot be outsourced to research assistants. In the future, artificial intelligence (GPT 7?) will be capable of assisting with this time-consuming task. However, at present, there is no alternative to the authors of the meta-analysis, who, as specialists in the field, are uniquely qualified to review each primary study meticulously and methodically construct their dataset manually, one data point at a time (Irsova et al., 2023).
Unlike the authors of most econometric studies, meta-analysts do not take existing data but create new databases*. 

At least two co-authors must gather the data independently. This rigorous process, while time-consuming, is crucial for ensuring the reliability of the meta-analysis. Mistakes can be expected when manually coding studies (which often consist of dozens of pages in PDF format), and having two experts collect the data allows for easy identification and correction of errors. The effect sizes gathered for meta-analysis should be comparable qualitatively and quantitatively. This means that not only should the same estimated sign indicate an effect in the same direction, but it should also be meaningful to compare the actual sizes of the effects across primary studies.

Effect sizes that are quantitatively comparable include correlation coefficients, odd ratios, elasticities, dollar values, and standardised mean differences. Regression coefficients are generally only comparable quantitatively with further transformations because different primary studies can use different units of measurement or functional forms of the independent and dependent variables. An exception is represented by regressions in which variables on both sides are used in logarithms, and, therefore, the regression yields estimated elasticities.

Collecting all estimates reported in the primary studies is imperative. This approach is recommended for five reasons (Irsova et al., 2023):

1. It provides a comprehensive view, ensuring that no information is discarded and eliminating the need for subjective judgment. This comprehensive approach to data collection gives the researchers confidence in the thoroughness of their analysis. You can always present a meta-analysis of the corresponding subsample of the dataset to give greater weight to the estimates preferred by the authors.
2. An exclusive analysis like this can confirm the strength of the results or establish a starting point. However, disregarding other estimates is unjustified even in the latter scenario.
3. When conducting original research, extra verifications are expected to guarantee the accuracy of the findings. Occasionally, the researchers themselves consider these findings less reliable. By incorporating all the findings, it is possible to assess whether the "inferior" results differ consistently from those favoured by the authors.
4. When conducting a best-practice meta-analysis, it is still appropriate to give greater weight to the authors' preferred results. At times, it is challenging to determine objectively which results favour the author. However, collecting and analysing all the findings can empower the researchers to make informed decisions without the need for subjective judgments.

It is important to examine any outliers and influential points in your data. One method is creating a funnel plot, a scatter plot of effect sizes and their precision. Suppose you notice data points that significantly deviate from the main funnel shape or raise concerns in DFBETA (a method for measuring the influence of individual data points on regression analysis)**.

In that case, reviewing the primary studies associated with those data points is advisable. This review will help ensure that there are no errors in the data or the primary studies, and it may also reveal nuances in how the studies were conducted, making their results incomparable to the rest of the research literature. If there are still uncertainties, reaching out to the authors of the primary studies can provide clarity. It is crucial to address any influential or leverage points identified by DFBETA, as they can heavily impact the results of your meta-analysis. This may involve correcting or excluding these points as a last resort. Additionally, it is not just recommended, it's essential that robustness checks be reported to show the impact of removing outliers or employing winsorisationn (replace observations above and below a certain centile with the value of that centile) on the data (Zigraiova et al., 2020). Ultimately, your results should be driven by reliable and influential research findings, and if this is the case, the prominence of these findings should be justified in detail.

Finally, ensure that apart from effect sizes and standard errors, you also gather information on significant differences in the context where the estimated effect sizes were obtained. Most meta-analyses should gather at least ten variables (often dummy binary variables which take the value 0 or 1) reflecting differences in data, methods, and publication characteristics. Depending on the size and complexity of the database, more variables may be needed, but it is advisable to keep the number below 30 for simplicity. For instance, consider if the primary study's experiment focuses on a representative sample of the population or only on a specific group, the country where it was conducted, whether a placebo or an alternative treatment was assigned to the control group, publication date, impact factor of the outlet, and the number of yearly citations received.

Before collecting data, prepare a list of variables to code carefully. This can be the most challenging and creative part of a meta-analysis. The number of potential variables is nearly unlimited, so selecting the most important ones is essential based on discussions in the literature and your expertise. A comprehensive language model can help identify some of the dimensions in which primary studies vary. However, it's crucial to remember that double-checking is vital, as artificial intelligence can sometimes provide misleading results. This caution and attention to detail will ensure the accuracy of your meta-analysis.

Consider including additional information that complements what you collect from primary studies. This comprehensive approach, which goes beyond the primary studies, can provide a more thorough understanding of the research context. For example, if the primary studies were conducted in various countries, including country (or region) characteristics might be valuable as additional variables. Experiment results can be influenced by factors such as temperature, humidity, or the country's financial development, which might be impossible to analyse by individual primary studies alone. Considering and including such additional information can make your meta-analysis more comprehensive and insightful (Irsova et al., 2023).

* Examples of meta-analysis datasets are available at https://www.meta-analysis.cz/.

** See https://blogs.sas.com/content/iml/2019/06/17/influence-regression-dfbeta.html