Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?
- PMID:11072941
- DOI: 10.1016/S0140-6736(00)02786-0
Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?
Abstract
Background: The inclusion of only a subset of all available evidence in a meta-analysis may introduce biases and threaten its validity; this is particularly likely if the subset of included studies differ from those not included, which may be the case for published and grey literature (unpublished studies, with limited distribution). We set out to examine whether exclusion of grey literature, compared with its inclusion in meta-analysis, provides different estimates of the effectiveness of interventions assessed in randomised trials.
Methods: From a random sample of 135 meta-analyses, we identified and retrieved 33 publications that included both grey and published primary studies. The 33 publications contributed 41 separate meta-analyses from several disease areas. General characteristics of the meta-analyses and associated studies and outcome data at the trial level were collected. We explored the effects of the inclusion of grey literature on the quantitative results using logistic-regression analyses.
Findings: 33% of the meta-analyses were found to include some form of grey literature. The grey literature, when included, accounts for between 4.5% and 75% of the studies in a meta-analysis. On average, published work, compared with grey literature, yielded significantly larger estimates of the intervention effect by 15% (ratio of odds ratios=1.15 [95% CI 1.04-1.28]). Excluding abstracts from the analysis further compounded the exaggeration (1.33 [1.10-1.60]).
Interpretation: The exclusion of grey literature from meta-analyses can lead to exaggerated estimates of intervention effectiveness. In general, meta-analysts should attempt to identify, retrieve, and include all reports, grey and published, that meet predefined inclusion criteria.
Similar articles
- Grey literature in meta-analyses.Conn VS, Valentine JC, Cooper HM, Rantz MJ.Conn VS, et al.Nurs Res. 2003 Jul-Aug;52(4):256-61. doi: 10.1097/00006199-200307000-00008.Nurs Res. 2003.PMID:12867783Review.
- Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research.Schmucker CM, Blümle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, von Elm E, Briel M, Meerpohl JJ; OPEN consortium.Schmucker CM, et al.PLoS One. 2017 Apr 25;12(4):e0176210. doi: 10.1371/journal.pone.0176210. eCollection 2017.PLoS One. 2017.PMID:28441452Free PMC article.Review.
- Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses.Hart B, Lundh A, Bero L.Hart B, et al.BMJ. 2012 Jan 3;344:d7202. doi: 10.1136/bmj.d7202.BMJ. 2012.PMID:22214754
- Effectiveness of interventions that assist caregivers to support people with dementia living in the community: a systematic review.Parker D, Mills S, Abbey J.Parker D, et al.Int J Evid Based Healthc. 2008 Jun;6(2):137-72. doi: 10.1111/j.1744-1609.2008.00090.x.Int J Evid Based Healthc. 2008.PMID:21631819
- Sources of evidence for systematic reviews of interventions in diabetes.Royle PL, Bain L, Waugh NR.Royle PL, et al.Diabet Med. 2005 Oct;22(10):1386-93. doi: 10.1111/j.1464-5491.2005.01645.x.Diabet Med. 2005.PMID:16176201
Cited by
- Are Reading Interventions Effective for At-Risk Readers with ADHD? A Meta-Analysis.Chan ESM, Shero JA, Hand ED, Cole AM, Gaye F, Spiegel JA, Kofler MJ.Chan ESM, et al.J Atten Disord. 2023 Jan;27(2):182-200. doi: 10.1177/10870547221130111. Epub 2022 Oct 22.J Atten Disord. 2023.PMID:36278436Free PMC article.Review.
- The role of guidance in delivering cardiac resynchronization therapy: A systematic review and network meta-analysis.Mehta VS, Ayis S, Elliott MK, Widjesuriya N, Kardaman N, Gould J, Behar JM, Chiribiri A, Razavi R, Niederer S, Rinaldi CA.Mehta VS, et al.Heart Rhythm O2. 2022 Jul 20;3(5):482-492. doi: 10.1016/j.hroo.2022.07.005. eCollection 2022 Oct.Heart Rhythm O2. 2022.PMID:36340494Free PMC article.
- Anti-Xa versus time-guided anticoagulation strategies in extracorporeal membrane oxygenation: a systematic review and meta-analysis.Willems A, Roeleveld PP, Labarinas S, Cyrus JW, Muszynski JA, Nellis ME, Karam O.Willems A, et al.Perfusion. 2021 Jul;36(5):501-512. doi: 10.1177/0267659120952982. Epub 2020 Aug 29.Perfusion. 2021.PMID:32862767Free PMC article.
- Finding studies for inclusion in systematic reviews of interventions for injury prevention the importance of grey and unpublished literature.Blackhall K.Blackhall K.Inj Prev. 2007 Oct;13(5):359. doi: 10.1136/ip.2007.017020.Inj Prev. 2007.PMID:17916897Free PMC article.No abstract available.
- Reporting bias in medical research - a narrative review.McGauran N, Wieseler B, Kreis J, Schüler YB, Kölsch H, Kaiser T.McGauran N, et al.Trials. 2010 Apr 13;11:37. doi: 10.1186/1745-6215-11-37.Trials. 2010.PMID:20388211Free PMC article.Review.
MeSH terms
Related information
LinkOut - more resources
Full Text Sources