Random-effects meta-analysis of the clinical utility of tests and prediction models.

Abstract:

:The use of data from multiple studies or centers for the validation of a clinical test or a multivariable prediction model allows researchers to investigate the test's/model's performance in multiple settings and populations. Recently, meta-analytic techniques have been proposed to summarize discrimination and calibration across study populations. Here, we rather consider performance in terms of net benefit, which is a measure of clinical utility that weighs the benefits of true positive classifications against the harms of false positives. We posit that it is important to examine clinical utility across multiple settings of interest. This requires a suitable meta-analysis method, and we propose a Bayesian trivariate random-effects meta-analysis of sensitivity, specificity, and prevalence. Across a range of chosen harm-to-benefit ratios, this provides a summary measure of net benefit, a prediction interval, and an estimate of the probability that the test/model is clinically useful in a new setting. In addition, the prediction interval and probability of usefulness can be calculated conditional on the known prevalence in a new setting. The proposed methods are illustrated by 2 case studies: one on the meta-analysis of published studies on ear thermometry to diagnose fever in children and one on the validation of a multivariable clinical risk prediction model for the diagnosis of ovarian cancer in a multicenter dataset. Crucially, in both case studies the clinical utility of the test/model was heterogeneous across settings, limiting its usefulness in practice. This emphasizes that heterogeneity in clinical utility should be assessed before a test/model is routinely implemented.

journal_name

Stat Med

journal_title

Statistics in medicine

authors

Wynants L,Riley RD,Timmerman D,Van Calster B

doi

10.1002/sim.7653

subject

Has Abstract

pub_date

2018-05-30 00:00:00

pages

2034-2052

issue

12

eissn

0277-6715

issn

1097-0258

journal_volume

37

pub_type

杂志文章
  • Two-sample rank tests for acceleration in cure models.

    abstract::I derive the locally most powerful rank tests for acceleration against semi-parametric alternatives when some patients are cured of the disease. I consider some particular classes of alternatives and present simulation results to verify the validity of the proposed tests. Real data from clinical trials for childhood l...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780141905

    authors: Lee JW

    更新日期:1995-10-15 00:00:00

  • Non-parametric estimation of the post-lead-time survival distribution of screen-detected cancer cases.

    abstract::The goal of screening programmes for cancer is early detection and treatment with a consequent reduction in mortality from the disease. Screening programmes need to assess the true benefit of screening, that is, the length of time of extension of survival beyond the time of advancement of diagnosis (lead-time). This p...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780142410

    authors: Xu JL,Prorok PC

    更新日期:1995-12-30 00:00:00

  • Analyzing disease recurrence with missing at risk information.

    abstract::When analyzing time to disease recurrence, we sometimes need to work with data where all the recurrences are recorded, but no information is available on the possible deaths. This may occur when studying diseases of benign nature where patients are only seen at disease recurrences or in poorly-designed registries of b...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.6766

    authors: Štupnik T,Pohar Perme M

    更新日期:2016-03-30 00:00:00

  • Human disease cost network analysis.

    abstract::Diseases can be interconnected. In the recent years, there has been a surge of multidisease studies. Among them, HDN (human disease network) analysis takes a system perspective, examines the interconnections among diseases along with their individual properties, and has demonstrated great potential. Most of the existi...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.8472

    authors: Ma C,Li Y,Shia B,Ma S

    更新日期:2020-04-30 00:00:00

  • Semiparametric transformation models for joint analysis of multivariate recurrent and terminal events.

    abstract::Recurrent event data occur in many clinical and observational studies, and in these situations, there may exist a terminal event such as death that is related to the recurrent event of interest. In addition, sometimes more than one type of recurrent events may occur, that is, one may encounter multivariate recurrent e...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4306

    authors: Zhu L,Sun J,Srivastava DK,Tong X,Leisenring W,Zhang H,Robison LL

    更新日期:2011-11-10 00:00:00

  • Independent data monitoring committees: rationale, operations and controversies.

    abstract::Data monitoring committees (DMCs) have become an increasingly common component of randomized clinical trials in recent years. As experience has accumulated, and more individuals and organizations have become involved in such activities, a variety of approaches to the operation of such committees has inevitably arisen....

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.730

    authors: Ellenberg SS

    更新日期:2001-09-15 00:00:00

  • The impact of heterogeneity on the comparison of survival times.

    abstract::We consider several sources of heterogeneity in a clinical trial with patients' survival time as the main response criterion: differences in prognosis which can be attributed to a latent or ignored prognostic factor; differences in treatment efficacy in subgroups of patients, and differences in treatment combinations ...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780060708

    authors: Schumacher M,Olschewski M,Schmoor C

    更新日期:1987-10-01 00:00:00

  • Power and sample size calculation for log-rank test with a time lag in treatment effect.

    abstract::The log-rank test is the most powerful non-parametric test for detecting a proportional hazards alternative and thus is the most commonly used testing procedure for comparing time-to-event distributions between different treatments in clinical trials. When the log-rank test is used for the primary data analysis, the s...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.3501

    authors: Zhang D,Quan H

    更新日期:2009-02-28 00:00:00

  • Using the general linear mixed model to analyse unbalanced repeated measures and longitudinal data.

    abstract::The general linear mixed model provides a useful approach for analysing a wide variety of data structures which practising statisticians often encounter. Two such data structures which can be problematic to analyse are unbalanced repeated measures data and longitudinal data. Owing to recent advances in methods and sof...

    journal_title:Statistics in medicine

    pub_type: 杂志文章,评审

    doi:10.1002/(sici)1097-0258(19971030)16:20<2349::aid-s

    authors: Cnaan A,Laird NM,Slasor P

    更新日期:1997-10-30 00:00:00

  • Comparison of methods for the analysis of longitudinal interval count data.

    abstract::Longitudinal studies are often concerned with estimating the recurrence rate of a non-fatal event. In many cases, only the total number of events occurring during successive time intervals is known. We compared a mixed Poisson-gamma regression method proposed by Thall and a quasi-likelihood method proposed by Zeger an...

    journal_title:Statistics in medicine

    pub_type: 临床试验,杂志文章,随机对照试验

    doi:10.1002/sim.4780121406

    authors: Stukel TA

    更新日期:1993-07-30 00:00:00

  • Cluster detection diagnostics for small area health data: with reference to evaluation of local likelihood models.

    abstract::The focus of this paper is the development of a range of cluster detection diagnostics that can be used to assess the degree to which a clustering method recovers the true clustering behaviour of small area data. The diagnostics proposed range from individual region specific diagnostics to neighbourhood diagnostics, a...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2401

    authors: Hossain MM,Lawson AB

    更新日期:2006-03-15 00:00:00

  • Two-stage residual inclusion for survival data and competing risks-An instrumental variable approach with application to SEER-Medicare linked data.

    abstract::Instrumental variable is an essential tool for addressing unmeasured confounding in observational studies. Two-stage predictor substitution (2SPS) estimator and two-stage residual inclusion (2SRI) are two commonly used approaches in applying instrumental variables. Recently, 2SPS was studied under the additive hazards...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.8071

    authors: Ying A,Xu R,Murphy J

    更新日期:2019-05-10 00:00:00

  • Efficient semiparametric inference for two-phase studies with outcome and covariate measurement errors.

    abstract::In modern observational studies using electronic health records or other routinely collected data, both the outcome and covariates of interest can be error-prone and their errors often correlated. A cost-effective solution is the two-phase design, under which the error-prone outcome and covariates are observed for all...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.8799

    authors: Tao R,Lotspeich SC,Amorim G,Shaw PA,Shepherd BE

    更新日期:2021-02-10 00:00:00

  • Proportional hazards models with frailties and random effects.

    abstract::We discuss some of the fundamental concepts underlying the development of frailty and random effects models in survival. One of these fundamental concepts was the idea of a frailty model where each subject has his or her own disposition to failure, their so-called frailty, additional to any effects we wish to quantify...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.1259

    authors: O'Quigley J,Stare J

    更新日期:2002-11-15 00:00:00

  • The probability of a cancer cluster due to chance alone.

    abstract::We propose to use a very simple model to test whether a cancer cluster is due to chance alone. We focus on the acute childhood leukaemia cluster in Columbus, Ohio. In 1975, 12 leukaemia cases were observed in Columbus while the expected number is 6 cases per year. According to our simple model, the probability of such...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/1097-0258(20000830)19:16<2195::aid-sim522>

    authors: Schinazi RB

    更新日期:2000-08-30 00:00:00

  • Extending the c-statistic to nominal polytomous outcomes: the Polytomous Discrimination Index.

    abstract::Diagnostic problems in medicine are sometimes polytomous, meaning that the outcome has more than two distinct categories. For example, ovarian tumors can be benign, borderline, primary invasive, or metastatic. Extending the main measure of binary discrimination, the c-statistic or area under the ROC curve, to nominal ...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.5321

    authors: Van Calster B,Van Belle V,Vergouwe Y,Timmerman D,Van Huffel S,Steyerberg EW

    更新日期:2012-10-15 00:00:00

  • Rank-based estimating equations with general weight for accelerated failure time models: an induced smoothing approach.

    abstract::The induced smoothing technique overcomes the difficulties caused by the non-smoothness in rank-based estimating functions for accelerated failure time models, but it is only natural when the estimating function has Gehan's weight. For a general weight, the induced smoothing method does not provide smooth estimating f...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.6415

    authors: Chiou S,Kang S,Yan J

    更新日期:2015-04-30 00:00:00

  • An illness-death stochastic model in the analysis of longitudinal dementia data.

    abstract::A significant source of missing data in longitudinal epidemiological studies on elderly individuals is death. Subjects in large scale community-based longitudinal dementia studies are usually evaluated for disease status in study waves, not under continuous surveillance as in traditional cohort studies. Therefore, for...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.1506

    authors: Harezlak J,Gao S,Hui SL

    更新日期:2003-05-15 00:00:00

  • An easy-to-implement approach for analyzing case-control and case-only studies assuming gene-environment independence and Hardy-Weinberg equilibrium.

    abstract::The case-control study is a simple and an useful method to characterize the effect of a gene, the effect of an exposure, as well as the interaction between the two. The control-free case-only study is yet an even simpler design, if interest is centered on gene-environment interaction only. It requires the sometimes pl...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4028

    authors: Lee WC,Wang LY,Cheng KF

    更新日期:2010-10-30 00:00:00

  • Constructing multiple test procedures for partially ordered hypothesis sets.

    abstract::A popular method to control multiplicity in confirmatory clinical trials is to use a so-called hierarchical, or fixed sequence, test procedure. This requires that the null hypotheses are ordered a priori, for example, in order of clinical importance. The procedure tests the hypotheses in this order using alpha-level t...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2905

    authors: Edwards D,Madsen J

    更新日期:2007-12-10 00:00:00

  • Multilevel mixed effects parametric survival models using adaptive Gauss-Hermite quadrature with application to recurrent events and individual participant data meta-analysis.

    abstract::Multilevel mixed effects survival models are used in the analysis of clustered survival data, such as repeated events, multicenter clinical trials, and individual participant data (IPD) meta-analyses, to investigate heterogeneity in baseline risk and covariate effects. In this paper, we extend parametric frailty model...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.6191

    authors: Crowther MJ,Look MP,Riley RD

    更新日期:2014-09-28 00:00:00

  • Estimating age-related trends in cross-sectional studies using S-distributions.

    abstract::Growth trends in children are often based on cross-sectional studies, in which a sample of the population is investigated at one given point in time. Estimating age-related percentiles in such studies involves fitting data distributions, each of which is specific for one age group, and a subsequent smoothing of the pe...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/(sici)1097-0258(20000315)19:5<697::aid-sim

    authors: Sorribas A,March J,Voit EO

    更新日期:2000-03-15 00:00:00

  • Adjusting for confounding by neighborhood using generalized linear mixed models and complex survey data.

    abstract::When investigating health disparities, it can be of interest to explore whether adjustment for socioeconomic factors at the neighborhood level can account for, or even reverse, an unadjusted difference. Recently, we proposed new methods to adjust the effect of an individual-level covariate for confounding by unmeasure...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.5624

    authors: Brumback BA,Zheng HW,Dailey AB

    更新日期:2013-04-15 00:00:00

  • Estimating the stage-specific numbers of HIV infection using a Markov model and back-calculation.

    abstract::The back-calculation method has been used to estimate the number of HIV infections from AIDS incidence data in a particular population. We present an extension of back calculation that provides estimates of the numbers of HIV infectives in different stages of infection. We model the staging process with a time-depende...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780110612

    authors: Longini IM Jr,Byers RH,Hessol NA,Tan WY

    更新日期:1992-04-01 00:00:00

  • Traffic accident mapping in Bangkok metropolis: a case study.

    abstract::Results from an analysis of traffic accidents from a study of the police records of four police stations in the Bangkok metropolis are presented. The main emphasis in this study was put on the development of a measure for traffic accident density. The traffic flow was estimated at the various study locations by traine...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780142113

    authors: Ayuthya RS,Böhning D

    更新日期:1995-11-15 00:00:00

  • A random forest approach for competing risks based on pseudo-values.

    abstract::Random forest is a supervised learning method that combines many classification or regression trees for prediction. Here we describe an extension of the random forest method for building event risk prediction models in survival analysis with competing risks. In case of right-censored data, the event status at the pred...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.5775

    authors: Mogensen UB,Gerds TA

    更新日期:2013-08-15 00:00:00

  • Properties of R(2) statistics for logistic regression.

    abstract::Various R(2) statistics have been proposed for logistic regression to quantify the extent to which the binary response can be predicted by a given logistic regression model and covariates. We study the asymptotic properties of three popular variance-based R(2) statistics. We find that two variance-based R(2) statistic...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2300

    authors: Hu B,Palta M,Shao J

    更新日期:2006-04-30 00:00:00

  • Comparison of predictive values of two diagnostic tests from the same sample of subjects using weighted least squares.

    abstract::Screening and diagnostic tests are important in disease prevention or control. The predictive values of positive and negative (PPV and NPV) test results are two of four operational characteristics of a screening test. We review an existing method based on the generalized estimating equation (GEE) methodology for compa...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2332

    authors: Wang W,Davis CS,Soong SJ

    更新日期:2006-07-15 00:00:00

  • Four-fold table cell frequencies imputation in meta analysis.

    abstract::Meta analysis is a collection of quantitative methods devoted to combine summary information from related but independent studies. Because research reports usually present only data reductions and summary statistics rather than detailed data, the reviewer must often resort to rather crude methods for constructing summ...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2287

    authors: Di Pietrantonj C

    更新日期:2006-07-15 00:00:00

  • Comparisons of risk prediction methods using nested case-control data.

    abstract::Using both simulated and real datasets, we compared two approaches for estimating absolute risk from nested case-control (NCC) data and demonstrated the feasibility of using the NCC design for estimating absolute risk. In contrast to previously published results, we successfully demonstrated not only that data from a ...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.7143

    authors: Salim A,Delcoigne B,Villaflores K,Koh WP,Yuan JM,van Dam RM,Reilly M

    更新日期:2017-02-10 00:00:00