Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews
- PMID:28420349
- PMCID: PMC5395863
- DOI: 10.1186/s12874-017-0347-z
Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews
Abstract
Background: Systematic reviews (SRs) are an important source of information about healthcare interventions. A key component of a well-conducted SR is a comprehensive literature search. There is limited evidence on the contribution of non-English reports, unpublished studies, and dissertations and their impact on results of meta-analyses.
Methods: Our sample included SRs from three Cochrane Review Groups: Acute Respiratory Infections (ARI), Infectious Diseases (ID), Developmental Psychosocial and Learning Problems (DPLP) (n = 129). Outcomes included: 1) proportion of reviews that searched for and included each study type; 2) proportion of relevant studies represented by each study type; and 3) impact on results and conclusions of the primary meta-analysis for each study type.
Results: Most SRs searched for non-English studies; however, these were included in only 12% of reviews and represented less than 5% of included studies. There was a change in results in only four reviews (total sample = 129); in two cases the change did not have an impact on the statistical or clinical significance of results. Most SRs searched for unpublished studies but the majority did not include these (only 6%) and they represented 2% of included studies. In most cases the impact of including unpublished studies was small; a substantial impact was observed in one case that relied solely on unpublished data. Few reviews in ARI (9%) and ID (3%) searched for dissertations compared to 65% in DPLP. Overall, dissertations were included in only nine SRs and represented less than 2% of included studies. In the majority of cases the change in results was negligible or small; in the case where a large change was noted, the estimate was more conservative without dissertations.
Conclusions: The majority of SRs searched for non-English and unpublished studies; however, these represented a small proportion of included studies and rarely impacted the results and conclusions of the review. Inclusion of these study types may have an impact in situations where there are few relevant studies, or where there are questionable vested interests in the published literature. We found substantial variation in whether SRs searched for dissertations; in most reviews that included dissertations, these had little impact on results.
Keywords: Dissertations; Grey literature; Knowledge synthesis; Language bias; Literature searching; Meta-analysis; Non-English publications; Publication bias; Systematic reviews; Unpublished studies.
Similar articles
- The contribution of databases to the results of systematic reviews: a cross-sectional study.Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B.Hartling L, et al.BMC Med Res Methodol. 2016 Sep 26;16(1):127. doi: 10.1186/s12874-016-0232-1.BMC Med Res Methodol. 2016.PMID:27670136Free PMC article.
- Incorporating data from dissertations in systematic reviews.Vickers AJ, Smith C.Vickers AJ, et al.Int J Technol Assess Health Care. 2000 Spring;16(2):711-3. doi: 10.1017/s0266462300101278.Int J Technol Assess Health Care. 2000.PMID:10932436
- Publication of reviews synthesizing child health evidence (PORSCHE): a survey of authors to identify factors associated with publication in Cochrane and non-Cochrane sources.Hartling L, Shave K, Thomson D, Fernandes RM, Wingert A, Williams K.Hartling L, et al.Syst Rev. 2016 Jun 21;5(1):104. doi: 10.1186/s13643-016-0276-7.Syst Rev. 2016.PMID:27328935Free PMC article.
- Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research.Schmucker CM, Blümle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, von Elm E, Briel M, Meerpohl JJ; OPEN consortium.Schmucker CM, et al.PLoS One. 2017 Apr 25;12(4):e0176210. doi: 10.1371/journal.pone.0176210. eCollection 2017.PLoS One. 2017.PMID:28441452Free PMC article.Review.
- Dissemination and publication of research findings: an updated review of related biases.Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, Hing C, Kwok CS, Pang C, Harvey I.Song F, et al.Health Technol Assess. 2010 Feb;14(8):iii, ix-xi, 1-193. doi: 10.3310/hta14080.Health Technol Assess. 2010.PMID:20181324Review.
Cited by
- Gray Literature in Evaluating Effectiveness in Digital Health and Health and Welfare Technology: A Source Worth Considering.Landerdahl Stridsberg S, Richardson MX, Redekop K, Ehn M, Wamala Andersson S.Landerdahl Stridsberg S, et al.J Med Internet Res. 2022 Mar 23;24(3):e29307. doi: 10.2196/29307.J Med Internet Res. 2022.PMID:35319479Free PMC article.
- Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study.Marshall IJ, Marshall R, Wallace BC, Brassey J, Thomas J.Marshall IJ, et al.J Clin Epidemiol. 2019 May;109:30-41. doi: 10.1016/j.jclinepi.2018.12.015. Epub 2018 Dec 25.J Clin Epidemiol. 2019.PMID:30590190Free PMC article.
- How Often Do Systematic Reviews Exclude Articles Not Published in English?Jackson JL, Kuriyama A.Jackson JL, et al.J Gen Intern Med. 2019 Aug;34(8):1388-1389. doi: 10.1007/s11606-019-04976-x.J Gen Intern Med. 2019.PMID:31011958Free PMC article.Review.No abstract available.
- Children's and adolescents' experiences of healthcare professionals: scoping review protocol.Davison G, Kelly MA, Thompson A, Dornan T.Davison G, et al.Syst Rev. 2020 Mar 7;9(1):51. doi: 10.1186/s13643-020-01298-6.Syst Rev. 2020.PMID:32145750Free PMC article.
- Bridging the gap: aligning economic research with disease burden.Do LA, Synnott PG, Ma S, Ollendorf DA.Do LA, et al.BMJ Glob Health. 2021 Jun;6(6):e005673. doi: 10.1136/bmjgh-2021-005673.BMJ Glob Health. 2021.PMID:34099483Free PMC article.
References
- Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011.
- Sterne JA, Egger M, Moher D. Chapter 10: Addressing reporting biases. In: Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011.
- Higgins JPT, Lasserson T, Chandler J, Tovey D, Churchill R. Methodological Expectations of Cochrane Intervention Reviews (MECIR): standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. Cochrane; 2016 [cited 2017 Jan 3]. Available from:http://methods.cochrane.org/sites/default/files/public/uploads/mecir_pri....
- Institute of Medicine . Finding what works in health care: standards for systematic reviews. Washington, DC: The National Academies Press; 2011. - PubMed
MeSH terms
Related information
LinkOut - more resources
Full Text Sources
Other Literature Sources
Research Materials