Original ArticleThe reporting quality of meta-analyses improves: a random sampling study
Introduction
The number of meta-analyses of randomized controlled trials (RCTs) has increased drastically in recent years. Meta-analysis, as a quantitative approach to combine results from similar RCTs, has earned a crucial position in providing useful information for evidence-based medicine (EBM) and health care decision making [1], [2], and the results it generates are regarded as the highest level in EBM studies [3].
Like any research enterprise, a meta-analysis can be flawed [4]. The authors of a meta-analysis report not only on the methods they used to review the articles, but also on the research methods in the articles. With this in mind, there has been an increased focus on the quality of reporting of meta-analyses (QUOROMs). A 1987 survey has discussed and explored several aspects of a meta-analysis of randomized trials including the influence of study design, control of bias, statistical analysis, and applicability of the results [5]. In 1996, the QUOROMs conference took place, where individuals with various backgrounds came together to develop guidelines for the reporting of meta-analyses. Their enormous efforts led to the publication of the QUOROM statement in 1999, which consists of a checklist and a flow diagram [6]. The checklist includes 18 items placed in six categories. It is meant to help authors improve their reporting quality and, to some extent, help with the evaluation of quality of meta-analyses.
Since its release, many journal editors and reviewers, including those involved in the Cochrane Collaboration, have pursued compliance with the QUOROM statement to provide sound, clear, and reproducible results [4], [7], [8], [9]. Several articles have evaluated the compliance with the statement [2], [10], [11], [12], [13], but the fields they focused on were limited and very small samples were given, generating results that were not always consistent with each other. In the articles, further research efforts dedicated to the influences of the QUOROM statement were called for. The overall impact of the statement on the design, conduct, and reporting of meta-analyses is yet to be unveiled.
With the exception of a few studies completed by Shea et al. [14] and Jadad et al. [15], little is known about the quality of Cochrane reviews and paper-based articles. To our knowledge, the comparison of reporting quality between Cochrane reviews and paper-based articles using the QUOROM checklist has not been evaluated nor studied.
The main purposes of this study are to determine the overall QUOROMs of RCTs based on the QUOROM statement, and to determine whether the reporting quality of meta-analyses improves with time. We also intend to compare the reporting quality of the electronic Cochrane reviews with that of the paper-based articles.
Section snippets
Search strategy and sample size
A literature search was conducted using Medline from January 2000 to December 2005 to identify meta-analyses of RCTs. The following strategy was used for search: (randomized controlled trial$ .mp. [mp = title, abstract, name of substance word, subject heading word]) AND ((systematic review OR meta analys$) .mp. [mp = title, original title, abstract, name of substance word, subject heading word]). Only articles published in English were accepted. We planned to select a random sample, a minimum
Results
Figure 1 is the flowchart for the selection of eligible articles. A total of 5,218 articles were retrieved, of which 368 were excluded for duplicates. Of the remaining 4,850 articles, 5% were selected as the study sample (n = 243) by random sampling. Of the 243 sampled articles, 82 were excluded, 48 provided no quantitative synthesis, 16 combined data from both RCTs and non-RCTs, 9 were articles of methodological research, 5 were Cochrane reviews updated in 2006, 3 were unclear whether or not the
Discussion
We often tell clinicians, health care policy makers, students, and consumers that systematic reviews represent “the best available evidence.” Unfortunately, there is relatively little empirical data on the QUOROMs or systematic reviews of RCTs. The primary purpose of this study was to determine the overall QUOROMs of RCTs. Although the results demonstrated a basically acceptable reporting quality (mean QUOROM score = 12.3), certain aspects in reporting still need improving.
In terms of complying
References (17)
- et al.
Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on Meta-Analysis
J Clin Epidemiol
(1995) Quality of reporting of meta-analyses: the QUOROM statement. Will it help?
J Hepatol
(2001)CONSORT and QUOROM guidelines for reporting randomized clinical trials and systematic reviews
Am J Orthod Dentofacial Orthop
(2005)- et al.
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses
Lancet
(1999) QUOROM—another step forwards
Ann Oncol
(2000)- et al.
Users' Guides to the Medical Literature: XXV. Evidence-based medicine: principles for applying the Users' Guides to patient care. Evidence-Based Medicine Working Group
JAMA
(2000) - et al.
Meta-analyses of randomized controlled trials
N Engl J Med
(1987) - In: Clarke M, Oxman A, eds. Cochrane reviewers' handbook 4.2.0. Available at...
Cited by (62)
PRISMA 2020 statement: What's new and the importance of reporting guidelines
2021, International Journal of SurgeryHalf of Cochrane reviews were published more than 2 years after the protocol
2020, Journal of Clinical EpidemiologySystematic reviews in orthodontics: Impact of the PRISMA for Abstracts checklist on completeness of reporting
2019, American Journal of Orthodontics and Dentofacial OrthopedicsRisk of bias over time in updates of Cochrane oral health reviews
2019, Journal of DentistrySystematic review and meta-analysis in GI endoscopy: Why do we need them? How can we read them? Should we trust them?
2018, Gastrointestinal EndoscopyQuality of meta-analyses in major leading orthopedics journals: A systematic review
2017, Orthopaedics and Traumatology: Surgery and ResearchCitation Excerpt :It was estimated that at least 2500 new MAs (including systematic reviews, SRs) were indexed in Medline yearly [3]. However, lots of MAs were methodologically poorly reported and could introduce bias and impair the reliability of conclusions although rigorous methodology is a feature of MAs [4,5]. To prevent such incidents, the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PRISMA) was developed in 2009 (Supplemental data 1) [1].