Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.

Abstract:

:Identification of key factors associated with the risk of developing cardiovascular disease and quantification of this risk using multivariable prediction algorithms are among the major advances made in preventive cardiology and cardiovascular epidemiology in the 20th century. The ongoing discovery of new risk markers by scientists presents opportunities and challenges for statisticians and clinicians to evaluate these biomarkers and to develop new risk formulations that incorporate them. One of the key questions is how best to assess and quantify the improvement in risk prediction offered by these new models. Demonstration of a statistically significant association of a new biomarker with cardiovascular risk is not enough. Some researchers have advanced that the improvement in the area under the receiver-operating-characteristic curve (AUC) should be the main criterion, whereas others argue that better measures of performance of prediction models are needed. In this paper, we address this question by introducing two new measures, one based on integrated sensitivity and specificity and the other on reclassification tables. These new measures offer incremental information over the AUC. We discuss the properties of these new measures and contrast them with the AUC. We also develop simple asymptotic tests of significance. We illustrate the use of these measures with an example from the Framingham Heart Study. We propose that scientists consider these types of measures in addition to the AUC when assessing the performance of newer biomarkers.

journal_name

Stat Med

journal_title

Statistics in medicine

authors

Pencina MJ,D'Agostino RB Sr,D'Agostino RB Jr,Vasan RS

doi

10.1002/sim.2929

subject

Has Abstract

pub_date

2008-01-30 00:00:00

pages

157-72; discussion 207-12

issue

2

eissn

0277-6715

issn

1097-0258

journal_volume

27

pub_type

杂志文章
  • An extension of the continual reassessment method using decision theory.

    abstract::The primary goal of a phase I trial is to find the maximally tolerated dose (MTD) of a treatment. The MTD is usually defined in terms of a tolerable probability, q(*), of toxicity. Our objective is to find the highest dose with toxicity risk that does not exceed q(*), a criterion that is often desired in designing pha...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.970

    authors: Leung DH,Wang YG

    更新日期:2002-01-15 00:00:00

  • Application of kriging models for a drug combination experiment on lung cancer.

    abstract::Combinatorial drugs have been widely applied in disease treatment, especially chemotherapy for cancer, due to its improved efficacy and reduced toxicity compared with individual drugs. The study of combinatorial drugs requires efficient experimental designs and proper follow-up statistical modeling techniques. Linear ...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.7971

    authors: Xiao Q,Wang L,Xu H

    更新日期:2019-01-30 00:00:00

  • Variance estimation of a survival function for interval-censored survival data.

    abstract::Interval-censored survival data often occur in medical studies, especially in clinical trials. In this case, many authors have considered estimation of a survival function. There is, however, relatively little discussion on estimating the variance of estimated survival functions. For right-censored data, a special cas...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.719

    authors: Sun J

    更新日期:2001-04-30 00:00:00

  • An extension of the continual reassessment methods using a preliminary up-and-down design in a dose finding study in cancer patients, in order to investigate a greater range of doses.

    abstract::In a phase I clinical trial in cancer patients, the drug involved had one known main adverse effect, which also occurs spontaneously in cancer patients with a fairly high frequency. Experiments in rats have shown marked effects of the drug on tumour growth in high doses, but also dose-dependent toxicity. Consequently,...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780140909

    authors: Møller S

    更新日期:1995-05-15 00:00:00

  • Semiparametric Bayesian variable selection for gene-environment interactions.

    abstract::Many complex diseases are known to be affected by the interactions between genetic variants and environmental exposures beyond the main genetic and environmental effects. Study of gene-environment (G×E) interactions is important for elucidating the disease etiology. Existing Bayesian methods for G×E interaction studie...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.8434

    authors: Ren J,Zhou F,Li X,Chen Q,Zhang H,Ma S,Jiang Y,Wu C

    更新日期:2020-02-28 00:00:00

  • Designs for phase I trials in ordered groups.

    abstract::We propose a new design for dose finding for cytotoxic agents in two ordered groups of patients. By ordered groups, we mean that prior to the study there is clinical information that would indicate that for a given dose one group would be more susceptible to toxicities than patients in the other group. The designs are...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.7133

    authors: Conaway MR,Wages NA

    更新日期:2017-01-30 00:00:00

  • Effects and non-effects of paired identical observations in comparing proportions with binary matched-pairs data.

    abstract::Binary matched-pairs data occur commonly in longitudinal studies, such as in cross-over experiments. Many analyses for comparing the matched probabilities of a particular outcome do not utilize pairs having the same outcome for each observation. An example is McNemar's test. Some methodologists find this to be counter...

    journal_title:Statistics in medicine

    pub_type: 杂志文章,评审

    doi:10.1002/sim.1589

    authors: Agresti A,Min Y

    更新日期:2004-01-15 00:00:00

  • Properties of R(2) statistics for logistic regression.

    abstract::Various R(2) statistics have been proposed for logistic regression to quantify the extent to which the binary response can be predicted by a given logistic regression model and covariates. We study the asymptotic properties of three popular variance-based R(2) statistics. We find that two variance-based R(2) statistic...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2300

    authors: Hu B,Palta M,Shao J

    更新日期:2006-04-30 00:00:00

  • Identifiability and estimation of causal mediation effects with missing data.

    abstract::Mediation analysis is a standard approach to understanding how and why an intervention works in social and medical sciences. However, the presence of missing data, especially missing not at random data, poses a great challenge for the applicability of this approach in practice. Current methods for handling such missin...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.7413

    authors: Li W,Zhou XH

    更新日期:2017-11-10 00:00:00

  • Survival probabilities with time-dependent treatment indicator: quantities and non-parametric estimators.

    abstract::The 'landmark' and 'Simon and Makuch' non-parametric estimators of the survival function are commonly used to contrast the survival experience of time-dependent treatment groups in applications such as stem cell transplant versus chemotherapy in leukemia. However, the theoretical survival functions corresponding to th...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.6765

    authors: Bernasconi DP,Rebora P,Iacobelli S,Valsecchi MG,Antolini L

    更新日期:2016-03-30 00:00:00

  • Robust fitting for neuroreceptor mapping.

    abstract::Among many other uses, positron emission tomography (PET) can be used in studies to estimate the density of a neuroreceptor at each location throughout the brain by measuring the concentration of a radiotracer over time and modeling its kinetics. There are a variety of kinetic models in common usage and these typicall...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.3510

    authors: Chang C,Ogden RT

    更新日期:2009-03-15 00:00:00

  • Multivariate joint frailty model for the analysis of nonlinear tumor kinetics and dynamic predictions of death.

    abstract::The Response Evaluation Criteria in Solid Tumors are used as standard guidelines for the clinical evaluation of cancer treatments. The assessment is based on the anatomical tumor burden: change in size of target lesions and evolution of nontarget lesions (NTL). Despite unquestionable advantages of this standard tool, ...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.7640

    authors: Król A,Tournigand C,Michiels S,Rondeau V

    更新日期:2018-06-15 00:00:00

  • Second-stage least squares versus penalized quasi-likelihood for fitting hierarchical models in epidemiologic analyses.

    abstract::Hierarchical regression analysis holds much promise for epidemiologic analysis, but has as yet seen limited application because of lack of easily used software and the relatively lengthy run times of preferred fitting methods (such as true maximum likelihood and Bayesian approaches). This paper compares three relative...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/(sici)1097-0258(19970315)16:5<515::aid-sim

    authors: Greenland S

    更新日期:1997-03-15 00:00:00

  • Bayesian bivariate meta-analysis of diagnostic test studies using integrated nested Laplace approximations.

    abstract::For bivariate meta-analysis of diagnostic studies, likelihood approaches are very popular. However, they often run into numerical problems with possible non-convergence. In addition, the construction of confidence intervals is controversial. Bayesian methods based on Markov chain Monte Carlo (MCMC) sampling could be u...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.3858

    authors: Paul M,Riebler A,Bachmann LM,Rue H,Held L

    更新日期:2010-05-30 00:00:00

  • Promoting interactions with basic scientists and clinicians: the NIA Alzheimer's Disease Data Coordinating Center.

    abstract::To benefit Alzheimer's disease research, a central data co-ordinating centre (CDCC) is planned that will systematically collect data from 27 Alzheimer's disease centres (ADCs) located nationwide. This CDCC will combine, analyse and disseminate epidemiologic, demographic, clinical and neuropathological data to research...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/(sici)1097-0258(20000615/30)19:11/12<1453:

    authors: Cronin-Stubbs D,DeKosky ST,Morris JC,Evans DA

    更新日期:2000-06-15 00:00:00

  • Multiple statistics for multiple events, with application to repeated infections in the growth factor studies.

    abstract::Clinical studies that involve the recording of two or more distinct and well-defined events on each subject give rise to multiple event data. Treatment comparisons are usually reported in univariate analyses of time to first event or number of events observed. However, this approach may not uncover the 'full story' of...

    journal_title:Statistics in medicine

    pub_type: 临床试验,杂志文章,多中心研究,随机对照试验

    doi:10.1002/(sici)1097-0258(19970430)16:8<941::aid-sim

    authors: Barai U,Teoh N

    更新日期:1997-04-30 00:00:00

  • Estimates of disease incidence in women based on antenatal or neonatal seroprevalence data: HIV in New York City.

    abstract::Piecewise constant incidence models were developed to estimate the force of infection in women from age- and time-specific antenatal or neonatal seroprevalence data. Differential inclusion of infected women in sero-surveys compared to uninfected women was taken into account, with respect to both changes in inclusion r...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780131809

    authors: Ades AE,Medley GF

    更新日期:1994-09-30 00:00:00

  • Using mark-recapture methodology to estimate the size of a population at risk for sexually transmitted diseases.

    abstract::To study the spread of sexually transmitted diseases (STDs) using social/sexual mixing models, one must have quantitative information about sexual mixing. An unavoidable complication in gathering such information by survey is that members of the surveyed population will almost certainly have sexual contacts outside th...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780111202

    authors: Rubin G,Umbach D,Shyu SF,Castillo-Chavez C

    更新日期:1992-09-15 00:00:00

  • Analysis of incomplete multivariate data using linear models with structured covariance matrices.

    abstract::Incomplete and unbalanced multivariate data often arise in longitudinal studies due to missing or unequally-timed repeated measurements and/or the presence of time-varying covariates. A general approach to analysing such data is through maximum likelihood analysis using a linear model for the expected responses, and s...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780070132

    authors: Schluchter MD

    更新日期:1988-01-01 00:00:00

  • Simple methods for checking for possible errors in reported odds ratios, relative risks and confidence intervals.

    abstract::Meta-analyses of data from epidemiological studies are often based on odds ratios (ORs) or relative risks (RRs) and their 95 per cent confidence intervals (CIs) as reported by the authors. Where possible ORs, RRs and CIs should be checked against the source data. Some simple methods are presented for checking the vali...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/(sici)1097-0258(19990815)18:15<1973::aid-s

    authors: Lee PN

    更新日期:1999-08-15 00:00:00

  • Performance assessment for radiologists interpreting screening mammography.

    abstract::When interpreting screening mammograms radiologists decide whether suspicious abnormalities exist that warrant the recall of the patient for further testing. Previous work has found significant differences in interpretation among radiologists; their false-positive and false-negative rates have been shown to vary widel...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2633

    authors: Woodard DB,Gelfand AE,Barlow WE,Elmore JG

    更新日期:2007-03-30 00:00:00

  • Reinforcement learning design for cancer clinical trials.

    abstract::We develop reinforcement learning trials for discovering individualized treatment regimens for life-threatening diseases such as cancer. A temporal-difference learning method called Q-learning is utilized that involves learning an optimal policy from a single training set of finite longitudinal patient trajectories. A...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.3720

    authors: Zhao Y,Kosorok MR,Zeng D

    更新日期:2009-11-20 00:00:00

  • Constructing time-specific reference ranges.

    abstract::Reference ranges which take time (such as age) into account are often required in medicine, but simple, systematic and efficient statistical methods for constructing them are lacking. A method is described which is based on low order polynomial curves (linear, quadratic or occasionally cubic), together with guidelines...

    journal_title:Statistics in medicine

    pub_type: 临床试验,杂志文章,多中心研究

    doi:10.1002/sim.4780100502

    authors: Royston P

    更新日期:1991-05-01 00:00:00

  • Spatially regularized estimation for the analysis of dynamic contrast-enhanced magnetic resonance imaging data.

    abstract::Competing compartment models of different complexities have been used for the quantitative analysis of dynamic contrast-enhanced magnetic resonance imaging data. We present a spatial elastic net approach that allows to estimate the number of compartments for each voxel such that the model complexity is not fixed a pri...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.5997

    authors: Sommer JC,Gertheiss J,Schmid VJ

    更新日期:2014-03-15 00:00:00

  • Estimating probit models with self-selected treatments.

    abstract::Outcomes research often requires estimating the impact of a binary treatment on a binary outcome in a non-randomized setting, such as the effect of taking a drug on mortality. The data often come from self-selected samples, leading to a spurious correlation between the treatment and outcome when standard binary depend...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2226

    authors: Bhattacharya J,Goldman D,McCaffrey D

    更新日期:2006-02-15 00:00:00

  • A standardization method to adjust for the effect of patient selection in phase II clinical trials.

    abstract::New combination regimens evaluated in phase II cancer clinical trials often show promising results compared to the standard therapy for a disease system. Selection of patients with a better prognosis can be a prominent factor for this optimism. For most disease systems, prognostic variables that are related to the out...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.706

    authors: Mazumdar M,Fazzari M,Panageas KS

    更新日期:2001-03-30 00:00:00

  • Statistical issues in the assessment of the evidence for an interaction between factors in epilepsy trials.

    abstract::We examine the common clinical belief that there is an interaction between epilepsy type and the two standard anti-epileptic drugs, valproate and carbamazepine, using data from several randomized clinical trials. Epilepsy type is not always easy to define, and three possible reclassifications are investigated to see w...

    journal_title:Statistics in medicine

    pub_type: 杂志文章,meta分析

    doi:10.1002/sim.1044

    authors: Williamson PR,Clough HE,Hutton JL,Marson AG,Chadwick DW

    更新日期:2002-09-30 00:00:00

  • Heterogeneity in the probability of HIV transmission per sexual contact: the case of male-to-female transmission in penile-vaginal intercourse.

    abstract::Recent studies have indicated variation in the infectivity beta of HIV among heterosexual couples. We represent this heterogeneity by modelling beta as a random variable. Using data on the number of contacts and seroconversion of couples, we fit the model by maximum-likelihood estimation with a beta distribution and a...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780080110

    authors: Wiley JA,Herschkorn SJ,Padian NS

    更新日期:1989-01-01 00:00:00

  • An evaluation of phase I clinical trial designs in the continuous dose-response setting.

    abstract::Both traditional phase I designs and the increasingly popular continual reassessment method (CRM) designs select an estimate of maximum tolerable dose (MTD) from among a set of prespecified dose levels. Although CRM designs use an implied dose-response model to select the next dose level, in general it is neither assu...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.903

    authors: Storer BE

    更新日期:2001-08-30 00:00:00

  • A cost-function approach to the design of reliability studies.

    abstract::We present a method to determine the number of subjects, k, and number of repeated measurements, n, that minimize the overall cost of conducting a reliability study, while providing acceptable power for tests of hypotheses concerning the reliability coefficient rho. Tables showing optimal choices of k and n under vari...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780060602

    authors: Eliasziw M,Donner A

    更新日期:1987-09-01 00:00:00