The Resource A process for robust and transparent rating of study quality : phase 1, prepared for Agency for Healthcare Research and Quality U.S. Department of Health and Human Services ; prepared by Tufts Medical Center Evidence-based Practice Center ; investigators, Stanley Ip ... [et al.], (electronic resource)

A process for robust and transparent rating of study quality : phase 1, prepared for Agency for Healthcare Research and Quality U.S. Department of Health and Human Services ; prepared by Tufts Medical Center Evidence-based Practice Center ; investigators, Stanley Ip ... [et al.], (electronic resource)

Label
A process for robust and transparent rating of study quality : phase 1
Title
A process for robust and transparent rating of study quality
Title remainder
phase 1
Statement of responsibility
prepared for Agency for Healthcare Research and Quality U.S. Department of Health and Human Services ; prepared by Tufts Medical Center Evidence-based Practice Center ; investigators, Stanley Ip ... [et al.]
Contributor
Subject
Language
  • eng
  • eng
Summary
BACKGROUND: Critical appraisal of individual studies with a formal summary judgment for methodological quality and subsequent assessment of the strength of a body of evidence addressing a specific question are essential activities of conducting comparative effectiveness reviews (CERs). Uncertainty concerning the optimal approach of quality assessments has given rise to wide variations in practice. A well-defined and transparent methodology to evaluate the robustness of quality assessments is critical for the interpretation of systematic reviews as well as the larger CER process. PURPOSE: To complete the first phase of a project to develop such a methodology, we aimed to examine the extent and potential sources of inter- and intra-rater variations in quality assessments, as conducted in our Evidence-based Practice Center (EPC). METHODS: We conducted three sequential exercises: (1) quality assessment of randomized controlled trials (RCTs) based on the default quality item checklist used in EPC reports without further instruction; (2) quality assessment of RCTs guided by explicit definitions of quality items; and (3) quality assessment of RCTs based on manuscripts stripped of identifying information, and performance of sensitivity analyses of quality items. The RCTs used in these exercises had been included in a previous CER on sleep apnea. Three experienced systematic reviewers participated in these exercises. DATA SYNTHESIS: In exercise 1, an initial set of 11 RCTs was subjected to a quality assessment process without any guidance, conducted in parallel by three independent reviewers. We found that the overall study quality ratings were discordant among the reviewers 64 percent of the time. In exercise 2, quality assessments were performed in a second set of RCTs, guided by explicit quality item definitions. The overall study quality ratings were discordant in 55 percent of the cases. In exercise 3, the provenance (i.e., title, authors, journal, etc.) of the published papers used in exercise 2 were concealed and simultaneously "influential" factors like study dropout rate and blinding were variably modified in a sensitivity analysis scheme. Comparing inter-rater disagreements between exercises 2 and 3, we observed that reviewers were less often in disagreement regarding the overall study quality rating (54.5 percent in exercise 2 vs. 45.5 percent in exercise 3). Anonymization of the papers resulted in increased proportion of disagreements for several items (e.g., "definition of outcomes," "appropriate statistics"). We also observed that for certain items that have a less subjective interpretation (e.g., blinding of outcome assessors or patients), there was a consistent extent of disagreement between exercises 2 and 3. LIMITATIONS: The results presented here are based on a small sample of RCTs, selected from a single CER and assessed by three reviewers from one EPC only. The definitions of the items in our checklist were not evaluated for adequacy and clarity, other than for their face validity assessed by the reviewers of this study. We acknowledge that this default checklist may not be in widespread use across evidence synthesis practices, and is not directly aligned with the current trend to transfer the focus from methodological (and reporting) quality to explicit assessment of the risk of bias of studies. Due to these reasons, the generalizability and the target audience of this research activity may be limited. Furthermore, we did not examine how our quality assessment tool compared with other available tools or how our assessments would differ if applied in a different clinical question. Thus, our findings are preliminary, and no definite conclusions could and should be drawn from this pilot study. CONCLUSIONS: We identified extensive variations in overall study ratings between three experienced reviewers. Discrepancies among reviewers in the assignment of individual items are common. While it may be desirable to have a single rating assessed by multiple reviewers using a process of reconciliation, in the absence of a gold standard method, it may be even more important to report the variations in assessments among different reviewers. A study with large variations in quality assessment may fundamentally be very different from one that has little variations, despite the fact that both of them are assigned the same consensus quality rating. Further investigations are needed to evaluate these hypotheses
Member of
Cataloging source
DNLM
Funding information
Contract No. 290-2007-100551
Government publication
federal national government publication
Illustrations
illustrations
Index
no index present
Literary form
non fiction
Nature of contents
  • dictionaries
  • bibliography
NLM call number
W 20.5
http://library.link/vocab/relatedWorkOrContributorName
  • Ip, Stanley
  • United States
  • Tufts Evidence-based Practice Center
Series statement
  • Methods research report
  • AHRQ publication
Series volume
no. 12-EHC004-EF
http://library.link/vocab/subjectName
  • Qualitative Research
  • Quality Control
  • Randomized Controlled Trials as Topic
  • Comparative Effectiveness Research
  • Evidence-Based Medicine
Label
A process for robust and transparent rating of study quality : phase 1, prepared for Agency for Healthcare Research and Quality U.S. Department of Health and Human Services ; prepared by Tufts Medical Center Evidence-based Practice Center ; investigators, Stanley Ip ... [et al.], (electronic resource)
Instantiates
Publication
Note
"November 2011."
Bibliography note
Includes bibliographical references
Dimensions
unknown
Extent
1 online resource (1 PDF file (vii, 27 p.))
Form of item
online
Other physical details
ill.
Specific material designation
remote
System control number
  • 1581907
  • (DNLM)BKSHLF:NBK82248
Label
A process for robust and transparent rating of study quality : phase 1, prepared for Agency for Healthcare Research and Quality U.S. Department of Health and Human Services ; prepared by Tufts Medical Center Evidence-based Practice Center ; investigators, Stanley Ip ... [et al.], (electronic resource)
Publication
Note
"November 2011."
Bibliography note
Includes bibliographical references
Dimensions
unknown
Extent
1 online resource (1 PDF file (vii, 27 p.))
Form of item
online
Other physical details
ill.
Specific material designation
remote
System control number
  • 1581907
  • (DNLM)BKSHLF:NBK82248

Library Locations

  • African Studies LibraryBorrow it
    771 Commonwealth Avenue, 6th Floor, Boston, MA, 02215, US
    42.350723 -71.108227
  • Alumni Medical LibraryBorrow it
    72 East Concord Street, Boston, MA, 02118, US
    42.336388 -71.072393
  • Astronomy LibraryBorrow it
    725 Commonwealth Avenue, 6th Floor, Boston, MA, 02445, US
    42.350259 -71.105717
  • Fineman and Pappas Law LibrariesBorrow it
    765 Commonwealth Avenue, Boston, MA, 02215, US
    42.350979 -71.107023
  • Frederick S. Pardee Management LibraryBorrow it
    595 Commonwealth Avenue, Boston, MA, 02215, US
    42.349626 -71.099547
  • Howard Gotlieb Archival Research CenterBorrow it
    771 Commonwealth Avenue, 5th Floor, Boston, MA, 02215, US
    42.350723 -71.108227
  • Mugar Memorial LibraryBorrow it
    771 Commonwealth Avenue, Boston, MA, 02215, US
    42.350723 -71.108227
  • Music LibraryBorrow it
    771 Commonwealth Avenue, 2nd Floor, Boston, MA, 02215, US
    42.350723 -71.108227
  • Pikering Educational Resources LibraryBorrow it
    2 Silber Way, Boston, MA, 02215, US
    42.349804 -71.101425
  • School of Theology LibraryBorrow it
    745 Commonwealth Avenue, 2nd Floor, Boston, MA, 02215, US
    42.350494 -71.107235
  • Science & Engineering LibraryBorrow it
    38 Cummington Mall, Boston, MA, 02215, US
    42.348472 -71.102257
  • Stone Science LibraryBorrow it
    675 Commonwealth Avenue, Boston, MA, 02445, US
    42.350103 -71.103784
Processing Feedback ...