An evaluation of DistillerSR's machine learning-based prioritization tool for title/abstract screening - impact on reviewer-relevant outcomes.

Abstract:

BACKGROUND:Systematic reviews often require substantial resources, partially due to the large number of records identified during searching. Although artificial intelligence may not be ready to fully replace human reviewers, it may accelerate and reduce the screening burden. Using DistillerSR (May 2020 release), we evaluated the performance of the prioritization simulation tool to determine the reduction in screening burden and time savings. METHODS:Using a true recall @ 95%, response sets from 10 completed systematic reviews were used to evaluate: (i) the reduction of screening burden; (ii) the accuracy of the prioritization algorithm; and (iii) the hours saved when a modified screening approach was implemented. To account for variation in the simulations, and to introduce randomness (through shuffling the references), 10 simulations were run for each review. Means, standard deviations, medians and interquartile ranges (IQR) are presented. RESULTS:Among the 10 systematic reviews, using true recall @ 95% there was a median reduction in screening burden of 47.1% (IQR: 37.5 to 58.0%). A median of 41.2% (IQR: 33.4 to 46.9%) of the excluded records needed to be screened to achieve true recall @ 95%. The median title/abstract screening hours saved using a modified screening approach at a true recall @ 95% was 29.8 h (IQR: 28.1 to 74.7 h). This was increased to a median of 36 h (IQR: 32.2 to 79.7 h) when considering the time saved not retrieving and screening full texts of the remaining 5% of records not yet identified as included at title/abstract. Among the 100 simulations (10 simulations per review), none of these 5% of records were a final included study in the systematic review. The reduction in screening burden to achieve true recall @ 95% compared to @ 100% resulted in a reduced screening burden median of 40.6% (IQR: 38.3 to 54.2%). CONCLUSIONS:The prioritization tool in DistillerSR can reduce screening burden. A modified or stop screening approach once a true recall @ 95% is achieved appears to be a valid method for rapid reviews, and perhaps systematic reviews. This needs to be further evaluated in prospective reviews using the estimated recall.

journal_name

BMC Med Res Methodol

authors

Hamel C,Kelly SE,Thavorn K,Rice DB,Wells GA,Hutton B

doi

10.1186/s12874-020-01129-1

subject

Has Abstract

pub_date

2020-10-15 00:00:00

pages

256

issue

1

issn

1471-2288

pii

10.1186/s12874-020-01129-1

journal_volume

20

pub_type

杂志文章
  • Comparing denominator degrees of freedom approximations for the generalized linear mixed model in analyzing binary outcome in small sample cluster-randomized trials.

    abstract:BACKGROUND:Small number of clusters and large variation of cluster sizes commonly exist in cluster-randomized trials (CRTs) and are often the critical factors affecting the validity and efficiency of statistical analyses. F tests are commonly used in the generalized linear mixed model (GLMM) to test intervention effect...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-015-0026-x

    authors: Li P,Redden DT

    更新日期:2015-04-23 00:00:00

  • A comparison of the statistical performance of different meta-analysis models for the synthesis of subgroup effects from randomized clinical trials.

    abstract:BACKGROUND:When investigating subgroup effects in meta-analysis, it is unclear whether accounting in meta-regression for between-trial variation in treatment effects, but not between-trial variation in treatment interaction effects when such effects are present, leads to biased estimates, coverage problems, or wrong st...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-019-0831-8

    authors: da Costa BR,Sutton AJ

    更新日期:2019-10-26 00:00:00

  • Power and sample size determination for the group comparison of patient-reported outcomes using the Rasch model: impact of a misspecification of the parameters.

    abstract:BACKGROUND:Patient-reported outcomes (PRO) are important as endpoints in clinical trials and epidemiological studies. Guidelines for the development of PRO instruments and analysis of PRO data have emphasized the need to report methods used for sample size planning. The Raschpower procedure has been proposed for sample...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-015-0011-4

    authors: Blanchin M,Guilleux A,Perrot B,Bonnaud-Antignac A,Hardouin JB,Sébille V

    更新日期:2015-03-15 00:00:00

  • Psychometric properties of an English short version of the Trier Inventory for chronic Stress.

    abstract:BACKGROUND:Although a variety of instruments are available that capture stress experience, the assessment of chronic stress has been hindered by the lack of economical screening instruments. Recently, an English-language version of the Trier Inventory for Chronic Stress (TICS-EN) consisting of 57 items according to a s...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-020-01156-y

    authors: Petrowski K,Braehler E,Schmalbach B,Hinz A,Bastianon C,Ritz T

    更新日期:2020-12-16 00:00:00

  • Protecting patient privacy when sharing patient-level data from clinical trials.

    abstract:BACKGROUND:Greater transparency and, in particular, sharing of patient-level data for further scientific research is an increasingly important topic for the pharmaceutical industry and other organisations who sponsor and conduct clinical trials as well as generally in the interests of patients participating in studies....

    journal_title:BMC medical research methodology

    pub_type: 信件

    doi:10.1186/s12874-016-0169-4

    authors: Tucker K,Branson J,Dilleen M,Hollis S,Loughlin P,Nixon MJ,Williams Z

    更新日期:2016-07-08 00:00:00

  • The new COSMIN guidelines confront traditional concepts of responsiveness.

    abstract::The recently published "COSMIN" guidelines aim to rate properties of outcome instruments and state two issues with regard to responsiveness which is the instrument's ability to detect change over time. These issues are comparison of score changes with change of an external criterion using correlations and the judgemen...

    journal_title:BMC medical research methodology

    pub_type: 评论,信件

    doi:10.1186/1471-2288-11-152

    authors: Angst F

    更新日期:2011-11-18 00:00:00

  • Academic pediatric clinical research: factors associated with study implementation duration.

    abstract:BACKGROUND:The ethical, methodological, and technical aspects of pediatric research, often results in complications and delays in implementation. Our objective was to identify factors associated with the implementation duration of hospital-based pediatric studies. METHODS:All hospital-based pediatric studies sponsored...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-016-0138-y

    authors: Meier-Girard D,Tibi A,Abdoul H,Prot-Labarthe S,Brion F,Bourdon O,Alberti C

    更新日期:2016-03-29 00:00:00

  • Evaluating treatments in health care: the instability of a one-legged stool.

    abstract:BACKGROUND:Both scientists and the public routinely refer to randomized controlled trials (RCTs) as being the 'gold standard' of scientific evidence. Although there is no question that placebo-controlled RCTs play a significant role in the evaluation of new pharmaceutical treatments, especially when it is important to ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-11-65

    authors: Kaplan BJ,Giesbrecht G,Shannon S,McLeod K

    更新日期:2011-05-11 00:00:00

  • Validating the generic quality of life tool "QOL10" in a substance use disorder treatment cohort exposes a unique social construct.

    abstract:BACKGROUND:Generic quality of life (QoL) instruments provide important measures of self-reported wellbeing that can be compared across healthy and clinical populations. The aim of this analysis is to validate the ten-item QoL instrument "QOL10", as well as to confirm the validity of the embedded "QOL5" questionnaire an...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-016-0163-x

    authors: Muller AE,Skurtveit S,Clausen T

    更新日期:2016-05-23 00:00:00

  • Misclassification of incident conditions using claims data: impact of varying the period used to exclude pre-existing disease.

    abstract:BACKGROUND:Estimating the incidence of medical conditions using claims data often requires constructing a prevalence period that predates an event of interest, for instance the diagnosis of cancer, to exclude those with pre-existing conditions from the incidence risk set. Those conditions missed during the prevalence p...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-13-32

    authors: Griffiths RI,O'Malley CD,Herbert RJ,Danese MD

    更新日期:2013-03-06 00:00:00

  • Measurement properties of the Danish version of the Awareness and Beliefs about Cancer (ABC) measure.

    abstract:BACKGROUND:The International Cancer Benchmarking Partnership aims to study international differences in cancer survival and the possible causes. Participating countries are Australia, Canada, Norway, Sweden, Denmark and the UK and a particular focus area is differences in awareness and beliefs about cancer. In this con...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-017-0352-2

    authors: Hvidberg L,Pedersen AF,Wulff CN,Carlsen AH,Vedsted P

    更新日期:2017-04-26 00:00:00

  • Blood spots as an alternative to whole blood collection and the effect of a small monetary incentive to increase participation in genetic association studies.

    abstract:BACKGROUND:Collection of buccal cells from saliva for DNA extraction offers a less invasive and convenient alternative to venipuncture blood collection that may increase participation in genetic epidemiologic studies. However, dried blood spot collection, which is also a convenient method, offers a means of collecting ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-9-76

    authors: Bhatti P,Kampa D,Alexander BH,McClure C,Ringer D,Doody MM,Sigurdson AJ

    更新日期:2009-11-13 00:00:00

  • Understanding the predictive value of continuous markers for censored survival data using a likelihood ratio approach.

    abstract:BACKGROUND:The likelihood ratio function (LR), the ratio of conditional probabilities of obtaining a specific marker value among those with the event of interest over those without, provides an easily interpretable way to quantify the update of the risk prediction due to the knowledge of the marker value. The LR has be...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-019-0721-0

    authors: Smith AM,Christodouleas JP,Hwang WT

    更新日期:2019-05-22 00:00:00

  • The Platino project: methodology of a multicenter prevalence survey of chronic obstructive pulmonary disease in major Latin American cities.

    abstract:BACKGROUND:The prevalence of Chronic Obstructive Pulmonary Disease (COPD) in many developed countries appears to be increasing. There is some evidence from Latin America that COPD is a growing cause of death, but information on prevalence is scant. It is possible that, due to the high frequency of smoking in these coun...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-4-15

    authors: Menezes AM,Victora CG,Perez-Padilla R,PLATINO Team.

    更新日期:2004-06-17 00:00:00

  • The fading of reported effectiveness. A meta-analysis of randomised controlled trials.

    abstract:BACKGROUND:The "real" effect size of a medical therapy is constant over time. In contrast, the effect size reported in randomised controlled trials (RCTs) may change over time because the sum of all kinds of bias influencing the reported effectiveness is not necessarily constant. As this would affect the validity of me...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章,meta分析

    doi:10.1186/1471-2288-6-25

    authors: Gehr BT,Weiss C,Porzsolt F

    更新日期:2006-05-11 00:00:00

  • Dealing with missing data in a multi-question depression scale: a comparison of imputation methods.

    abstract:BACKGROUND:Missing data present a challenge to many research projects. The problem is often pronounced in studies utilizing self-report scales, and literature addressing different strategies for dealing with missing data in such circumstances is scarce. The objective of this study was to compare six different imputatio...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-6-57

    authors: Shrive FM,Stuart H,Quan H,Ghali WA

    更新日期:2006-12-13 00:00:00

  • Heckman-type selection models to obtain unbiased estimates with missing measures outcome: theoretical considerations and an application to missing birth weight data.

    abstract:BACKGROUND:In low-income settings, key outcomes such as biomarkers or clinical assessments are often missing for a substantial proportion of the study population. The aim of this study was to assess the extent to which Heckman-type selection models can create unbiased estimates in such settings. METHODS:We introduce t...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-019-0840-7

    authors: Koné S,Bonfoh B,Dao D,Koné I,Fink G

    更新日期:2019-12-09 00:00:00

  • Suitability of measures of self-reported medication adherence for routine clinical use: a systematic review.

    abstract:BACKGROUND:There is a recognised need to build primary care medication adherence services which are tailored to patients' needs. Continuous quality improvement of such services requires a regular working method of measuring adherence in order to monitor effectiveness. Self report has been considered the method of choic...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章,评审

    doi:10.1186/1471-2288-11-149

    authors: Garfield S,Clifford S,Eliasson L,Barber N,Willson A

    更新日期:2011-11-03 00:00:00

  • Lost productivity due to premature mortality in developed and emerging countries: an application to smoking cessation.

    abstract:BACKGROUND:Researchers and policy makers have determined that accounting for productivity costs, or "indirect costs," may be as important as including direct medical expenditures when evaluating the societal value of health interventions. These costs are also important when estimating the global burden of disease. The ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-12-87

    authors: Menzin J,Marton JP,Menzin JA,Willke RJ,Woodward RM,Federico V

    更新日期:2012-06-25 00:00:00

  • A transdisciplinary team approach to scoping reviews: the case of pediatric polypharmacy.

    abstract:BACKGROUND:Polypharmacy can be either beneficial or harmful to children. We conducted a scoping review to examine the concept of pediatric polypharmacy: its definition, prevalence, extent and gaps in research. In this manuscript, we report our transdisciplinary scoping review methodology. METHODS:After establishing a ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-018-0560-4

    authors: Bakaki PM,Staley J,Liu R,Dawson N,Golchin N,Horace A,Johnson H,Waldron J,Winterstein A,Kleinman LC,Bolen SD

    更新日期:2018-10-04 00:00:00

  • Incorporating nonlinearity into mediation analyses.

    abstract:BACKGROUND:Mediation is an important issue considered in the behavioral, medical, and social sciences. It addresses situations where the effect of a predictor variable X on an outcome variable Y is explained to some extent by an intervening, mediator variable M. Methods for addressing mediation have been available for ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-017-0296-6

    authors: Knafl GJ,Knafl KA,Grey M,Dixon J,Deatrick JA,Gallo AM

    更新日期:2017-03-21 00:00:00

  • Best practice for analysis of shared clinical trial data.

    abstract:BACKGROUND:Greater transparency, including sharing of patient-level data for further research, is an increasingly important topic for organisations who sponsor, fund and conduct clinical trials. This is a major paradigm shift with the aim of maximising the value of patient-level data from clinical trials for the benefi...

    journal_title:BMC medical research methodology

    pub_type: 信件

    doi:10.1186/s12874-016-0170-y

    authors: Hollis S,Fletcher C,Lynn F,Urban HJ,Branson J,Burger HU,Tudur Smith C,Sydes MR,Gerlinger C

    更新日期:2016-07-08 00:00:00

  • The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews.

    abstract:BACKGROUND:In the era of evidence based medicine, with systematic reviews as its cornerstone, adequate quality assessment tools should be available. There is currently a lack of a systematically developed and evaluated tool for the assessment of diagnostic accuracy studies. The aim of this project was to combine empiri...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-3-25

    authors: Whiting P,Rutjes AW,Reitsma JB,Bossuyt PM,Kleijnen J

    更新日期:2003-11-10 00:00:00

  • Pan-Canadian assessment of pandemic immunization data collection: study methodology.

    abstract:BACKGROUND:The collection of individual-level pandemic (H1N1) 2009 influenza immunization data was considered important to facilitate optimal vaccine delivery and accurate assessment of vaccine coverage. These data are also critical for research aimed at evaluating the new vaccine's safety and effectiveness. Systems us...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-10-51

    authors: Pereira JA,Quach S,Heidebrecht C,Foisy J,Quan S,Finkelstein M,Sikora CA,Bettinger JA,Buckeridge DL,McCarthy A,Deeks S,Kwong JC,Public Health Agency of Canada\/Canadian Institutes of Health Research Influenza Research Network Vacc

    更新日期:2010-06-08 00:00:00

  • Getting messier with TIDieR: embracing context and complexity in intervention reporting.

    abstract:BACKGROUND:The Template for Intervention Description and Replication (TIDieR) checklist and guide was developed by an international team of experts to promote full and accurate description of trial interventions. It is now widely used in health research. The aim of this paper is to describe the experience of using TIDi...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-017-0461-y

    authors: Cotterill S,Knowles S,Martindale AM,Elvey R,Howard S,Coupe N,Wilson P,Spence M

    更新日期:2018-01-18 00:00:00

  • Suitability of administrative claims databases for bariatric surgery research - is the glass half-full or half-empty?

    abstract:BACKGROUND:Claims databases are generally considered inadequate for obesity research due to suboptimal capture of body mass index (BMI) measurements. This might not be true for bariatric surgery because of reimbursement requirements and changes in coding systems. We assessed the availability and validity of claims-base...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-020-01106-8

    authors: Li X,Lewis KH,Callaway K,Wharam JF,Toh S

    更新日期:2020-09-07 00:00:00

  • Self-reported measures in health research for people with intellectual disabilities: an inclusive pilot study on suitability and reliability.

    abstract:BACKGROUND:The lack of suitable and reliable scales to measure self-reported health and health behaviour among people with intellectual disabilities (ID) is an important methodological challenge in health research. This study, which was undertaken together with co-researchers with ID, explores possibilities for self-re...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-018-0539-1

    authors: Vlot-van Anrooij K,Tobi H,Hilgenkamp TIM,Leusink GL,Naaldenberg J

    更新日期:2018-07-16 00:00:00

  • Incongruence between test statistics and P values in medical papers.

    abstract:BACKGROUND:Given an observed test statistic and its degrees of freedom, one may compute the observed P value with most statistical packages. It is unknown to what extent test statistics and P values are congruent in published medical papers. METHODS:We checked the congruence of statistical results reported in all the ...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/1471-2288-4-13

    authors: García-Berthou E,Alcaraz C

    更新日期:2004-05-28 00:00:00

  • Translation method is validity evidence for construct equivalence: analysis of secondary data routinely collected during translations of the Health Literacy Questionnaire (HLQ).

    abstract:BACKGROUND:Cross-cultural research with patient-reported outcomes measures (PROMs) assumes that the PROM in the target language will measure the same construct in the same way as the PROM in the source language. Yet translation methods are rarely used to qualitatively maximise construct equivalence or to describe the i...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-020-00962-8

    authors: Hawkins M,Cheng C,Elsworth GR,Osborne RH

    更新日期:2020-05-26 00:00:00

  • Simulation-based estimation of mean and standard deviation for meta-analysis via Approximate Bayesian Computation (ABC).

    abstract:BACKGROUND:When conducting a meta-analysis of a continuous outcome, estimated means and standard deviations from the selected studies are required in order to obtain an overall estimate of the mean effect and its confidence interval. If these quantities are not directly reported in the publications, they must be estima...

    journal_title:BMC medical research methodology

    pub_type: 杂志文章

    doi:10.1186/s12874-015-0055-5

    authors: Kwon D,Reis IM

    更新日期:2015-08-12 00:00:00