Errors in the Conduct of Systematic Reviews of Pharmacological Interventions for Irritable Bowel Syndrome

Ford, Alexander C MD, MRCP¹; Guyatt, Gordon H MD, FRCPC^2,3; Talley, Nicholas J MD, PhD⁴; Moayyedi, Paul PhD, FRCP¹

¹Gastroenterology Division, Health Sciences Center, McMaster University, Hamilton, Ontario, Canada

²Department of Clinical Epidemiology and Biostatistics, Health Sciences Center, McMaster University, Hamilton, Ontario, Canada

³Department of Medicine, Health Sciences Center, McMaster University, Hamilton, Ontario, Canada

⁴Department of Medicine, Mayo Clinic, Jacksonville, Florida, USA

Correspondence: Alexander C. Ford, MD, MRCP, Gastroenterology Division, McMaster University Medical Centre, 1200 Main Street West, Hamilton, Ontario L8N 3Z5, Canada. E-mail:[email protected]

published online 17 November 2009

Received 7 June 2009; accepted 15 October 2009

American Journal of Gastroenterology105(2):p 280-288, February 2010. |DOI:10.1038/ajg.2009.658

Abstract

OBJECTIVES:

Systematic reviews and meta-analyses are integral to evidence-based clinical decision making. Although flawed systematic reviews could compromise optimal decision making, their accuracy has received limited investigation. We assessed conduct of systematic reviews of pharmaceutical interventions for irritable bowel syndrome (IBS).

METHODS:

We searched MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews (up to June 2008) to identify and replicate all published systematic reviews and meta-analyses that examined pharmacological interventions for IBS. We identified trials appropriately and inappropriately included according to the investigators' own eligibility criteria and eligible trials the investigators failed to include, and assessed the accuracy of dichotomous data extraction from all truly eligible trials. We conducted meta-analyses of accurate data from all truly eligible trials, and examined the differences between these accurate estimates and those reported by the authors.

RESULTS:

The search strategy identified 120 citations, and 13 appeared to be relevant. Five systematic reviews did not extract dichotomous data, leaving eight eligible for inclusion. In five of the eight meta-analyses 13–29% of included trials were ineligible according to investigators' criteria, constituting 8–26% of included patients. Six of the meta-analyses missed 17 separate published eligible trials; 3–11% of eligible patients were, as a result, not included. All eight meta-analyses contained errors in dichotomous data extraction, in 29–100% of truly eligible trials, leading to errors in 15 of 16 reported pooled treatment effects. There was a ≥10% relative difference in treatment effects between the reported and recalculated summary statistic in five (31%) cases, and a change in the statistical significance of the recalculated summary statistic in a further four (25%) cases.

CONCLUSIONS:

We found many errors in both application of eligibility criteria and dichotomous data extraction in the eight meta-analyses studied. Independent verification of systematic reviews and meta-analyses may be required for full confidence in their results.