The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models.

Abstract:

:Assessing the calibration of methods for estimating the probability of the occurrence of a binary outcome is an important aspect of validating the performance of risk-prediction algorithms. Calibration commonly refers to the agreement between predicted and observed probabilities of the outcome. Graphical methods are an attractive approach to assess calibration, in which observed and predicted probabilities are compared using loess-based smoothing functions. We describe the Integrated Calibration Index (ICI) that is motivated by Harrell's Emax index, which is the maximum absolute difference between a smooth calibration curve and the diagonal line of perfect calibration. The ICI can be interpreted as weighted difference between observed and predicted probabilities, in which observations are weighted by the empirical density function of the predicted probabilities. As such, the ICI is a measure of calibration that explicitly incorporates the distribution of predicted probabilities. We also discuss two related measures of calibration, E50 and E90, which represent the median and 90th percentile of the absolute difference between observed and predicted probabilities. We illustrate the utility of the ICI, E50, and E90 by using them to compare the calibration of logistic regression with that of random forests and boosted regression trees for predicting mortality in patients hospitalized with a heart attack. The use of these numeric metrics permitted for a greater differentiation in calibration than was permissible by visual inspection of graphical calibration curves.

journal_name

Stat Med

journal_title

Statistics in medicine

authors

Austin PC,Steyerberg EW

doi

10.1002/sim.8281

subject

Has Abstract

pub_date

2019-09-20 00:00:00

pages

4051-4065

issue

21

eissn

0277-6715

issn

1097-0258

journal_volume

38

pub_type

杂志文章
  • Comparison of hypertabastic survival model with other unimodal hazard rate functions using a goodness-of-fit test.

    abstract::We studied the problem of testing a hypothesized distribution in survival regression models when the data is right censored and survival times are influenced by covariates. A modified chi-squared type test, known as Nikulin-Rao-Robson statistic, is applied for the comparison of accelerated failure time models. This st...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.7244

    authors: Tahir MR,Tran QX,Nikulin MS

    更新日期:2017-05-30 00:00:00

  • Bias in the evaluation of DNA-amplification tests for detecting Chlamydia trachomatis.

    abstract::The purpose of this paper is to show that the sensitivity and specificity estimates obtained by 'discrepant analysis' are biased. Discrepant analysis is a widely used technique that attempts to provide estimates of sensitivity and specificity in the presence of an imperfect gold standard. Many researchers have applied...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/(sici)1097-0258(19970630)16:12<1391::aid-s

    authors: Hadgu A

    更新日期:1997-06-30 00:00:00

  • A transition model for quality-of-life data with non-ignorable non-monotone missing data.

    abstract::In this paper, we consider a full likelihood method to analyze continuous longitudinal responses with non-ignorable non-monotone missing data. We consider a transition probability model for the missingness mechanism. A first-order Markov dependence structure is assumed for both the missingness mechanism and observed d...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.5359

    authors: Liao K,Freres DR,Troxel AB

    更新日期:2012-12-10 00:00:00

  • Partitioned GMM logistic regression models for longitudinal data.

    abstract::Correlation is inherent in longitudinal studies due to the repeated measurements on subjects, as well as due to time-dependent covariates in the study. In the National Longitudinal Study of Adolescent to Adult Health (Add Health), data were repeatedly collected on children in grades 7-12 across four waves. Thus, obser...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.8099

    authors: Irimata KM,Broatch J,Wilson JR

    更新日期:2019-05-30 00:00:00

  • Determining the value of additional surrogate exposure data for improving the estimate of an odds ratio.

    abstract::We consider the design of both cohort and case-control studies in which an initial ('stage 1') sample of complete data on an error-free disease indicator (D), a correct ('gold standard') dichotomous exposure measurement (X) and an error-prone exposure measurement (Z) are available. We calculate the amount of additiona...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780142307

    authors: Dahm PF,Gail MH,Rosenberg PS,Pee D

    更新日期:1995-12-15 00:00:00

  • On design considerations and randomization-based inference for community intervention trials.

    abstract::This paper discusses design considerations and the role of randomization-based inference in randomized community intervention trials. We stress that longitudinal follow-up of cohorts within communities often yields useful information on the effects of intervention on individuals, whereas cross-sectional surveys can us...

    journal_title:Statistics in medicine

    pub_type: 杂志文章,评审

    doi:10.1002/(SICI)1097-0258(19960615)15:11<1069::AID-S

    authors: Gail MH,Mark SD,Carroll RJ,Green SB,Pee D

    更新日期:1996-06-15 00:00:00

  • Sam Greenhouse's years at the Census Bureau and the UNRRA.

    abstract::Sam Greenhouse joined the Census Bureau as a clerk at an interesting time period for the agency. The first use of sampling in the decennial census occurred in 1940. There was a major expansion of the amount of data collected. The organization of the Census Bureau underwent radical changes, including the growth of the ...

    journal_title:Statistics in medicine

    pub_type: 传,历史文章,杂志文章

    doi:10.1002/sim.1627

    authors: Keller J,Clark CZ

    更新日期:2003-11-15 00:00:00

  • Local influence measure of zero-inflated generalized Poisson mixture regression models.

    abstract::In many practical applications, count data often exhibit greater or less variability than allowed by the equality of mean and variance, referred to as overdispersion/underdispersion, and there are several reasons that may lead to the overdispersion/underdispersion such as zero inflation and mixture. Moreover, if the c...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.5560

    authors: Chen XD,Fu YZ,Wang XR

    更新日期:2013-04-15 00:00:00

  • Sample size calculation for clinical trials with correlated count measurements based on the negative binomial distribution.

    abstract::Statistical inference based on correlated count measurements are frequently performed in biomedical studies. Most of existing sample size calculation methods for count outcomes are developed under the Poisson model. Deviation from the Poisson assumption (equality of mean and variance) has been widely documented in pra...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.8378

    authors: Li D,Zhang S,Cao J

    更新日期:2019-12-10 00:00:00

  • The analysis of contingency tables with ordinal data: an application to monitoring antibiotic resistance.

    abstract::Rationalization of antibiotic therapy in the management of infectious diseases is helped by a knowledge of the patterns of sensitivity and resistance of bacteria to antibiotics and their possible changes both in time and from one hospital unit to another. In this paper we present the results regarding the sensitivitie...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2447

    authors: Bonetto C,Giannerini S,Giovagnoli A

    更新日期:2006-10-30 00:00:00

  • Testing for publication bias in diagnostic meta-analysis: a simulation study.

    abstract::The present study investigates the performance of several statistical tests to detect publication bias in diagnostic meta-analysis by means of simulation. While bivariate models should be used to pool data from primary studies in diagnostic meta-analysis, univariate measures of diagnostic accuracy are preferable for t...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.6177

    authors: Bürkner PC,Doebler P

    更新日期:2014-08-15 00:00:00

  • A cluster model for space-time disease counts.

    abstract::Modelling disease clustering over space and time can be helpful in providing indications of possible exposures and planning corresponding public health practices. Though a considerable number of studies focus on modelling spatio-temporal patterns of disease, most of them do not directly model a spatio-temporal cluster...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2424

    authors: Yan P,Clayton MK

    更新日期:2006-03-15 00:00:00

  • Bayesian disease mapping using product partition models.

    abstract::Our objective is to develop a model to estimate the relative risk of disease in each area, Ai, i=1, ... , n, of a region and to identify areas of unusually high or low risk. We use a product partition model (PPM) in which we assume that the true relative risks can be partitioned into a number of components or sets of ...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.3253

    authors: Hegarty A,Barry D

    更新日期:2008-08-30 00:00:00

  • Automated time series forecasting for biosurveillance.

    abstract::For robust detection performance, traditional control chart monitoring for biosurveillance is based on input data free of trends, day-of-week effects, and other systematic behaviour. Time series forecasting methods may be used to remove this behaviour by subtracting forecasts from observations to form residuals for al...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2835

    authors: Burkom HS,Murphy SP,Shmueli G

    更新日期:2007-09-30 00:00:00

  • Dose-interpolation of immunoassay data: uncertainties associated with curve-fitting.

    abstract::Estimates of analyte concentrations, obtained by immunoassay, have error distributions which are generally underestimated. Better estimates, which take into account the distribution of the response metameter of the calibration curve and uncertainties associated with the location of the fitted curve, have been obtained...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780050208

    authors: Kay C,Nix AB,Kemp KW,Rowlands RJ,Richards G,Groom GV,Griffiths K,Wilson DW

    更新日期:1986-03-01 00:00:00

  • Drug treatment of mild hypertension to reduce the risk of CHD: is it worth-while?

    abstract::Although hypertension is regarded as a causal factor for coronary heart disease (CHD) a reduction in the risk of CHD as a result of lowering blood pressure in mild hypertension could not be demonstrated. This conclusion is based on an overview analysis of all published randomized trials in mild hypertension, including...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780071104

    authors: Holme I

    更新日期:1988-11-01 00:00:00

  • STRengthening analytical thinking for observational studies: the STRATOS initiative.

    abstract::The validity and practical utility of observational medical research depends critically on good study design, excellent data quality, appropriate statistical methods and accurate interpretation of results. Statistical methodology has seen substantial development in recent times. Unfortunately, many of these methodolog...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.6265

    authors: Sauerbrei W,Abrahamowicz M,Altman DG,le Cessie S,Carpenter J,STRATOS initiative.

    更新日期:2014-12-30 00:00:00

  • The k-in-a-row up-and-down design, revisited.

    abstract::The percentile-finding experimental design known variously as 'forced-choice fixed-staircase', 'geometric up-and-down' or 'k-in-a-row' (KR) was introduced by Wetherill four decades ago. To date, KR has been by far the most widely used up-and-down (U&D) design for estimating non-median percentiles; it is implemented mo...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.3590

    authors: Oron AP,Hoff PD

    更新日期:2009-06-15 00:00:00

  • A comparison of methods for determining HIV viral set point.

    abstract::During a course of human immunodeficiency virus (HIV-1) infection, the viral load usually increases sharply to a peak following infection and then drops rapidly to a steady state, where it remains until progression to AIDS. This steady state is often referred to as the viral set point. It is believed that the HIV vira...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.3038

    authors: Mei Y,Wang L,Holte SE

    更新日期:2008-01-15 00:00:00

  • Design and analysis of non-inferiority mortality trials in oncology.

    abstract::The recent revision of the Declaration of Helsinki and the existence of many new therapies that affect survival or serious morbidity, and that therefore cannot be denied patients, have generated increased interest in active-control trials, particularly those intended to show equivalence or non-inferiority to the activ...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.1400

    authors: Rothmann M,Li N,Chen G,Chi GY,Temple R,Tsou HH

    更新日期:2003-01-30 00:00:00

  • Signal detection in FDA AERS database using Dirichlet process.

    abstract::In the recent two decades, data mining methods for signal detection have been developed for drug safety surveillance, using large post-market safety data. Several of these methods assume that the number of reports for each drug-adverse event combination is a Poisson random variable with mean proportional to the unknow...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.6510

    authors: Hu N,Huang L,Tiwari RC

    更新日期:2015-08-30 00:00:00

  • Using marginal structural models to adjust for treatment drop-in when developing clinical prediction models.

    abstract::Clinical prediction models (CPMs) can inform decision making about treatment initiation, which requires predicted risks assuming no treatment is given. However, this is challenging since CPMs are usually derived using data sets where patients received treatment, often initiated postbaseline as "treatment drop-ins." Th...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.7913

    authors: Sperrin M,Martin GP,Pate A,Van Staa T,Peek N,Buchan I

    更新日期:2018-12-10 00:00:00

  • Discriminant analysis when all variables are ordered.

    abstract::Determination of the equation that relates an ordered dependent variable to ordered independent variables is sought. One solution, non-parametric discriminant analysis (NPD), involves obtaining the best monotonic step function by means of a computer search procedure. Although one can use alternative selection criteria...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780110804

    authors: Johnston B,Seshia SS

    更新日期:1992-06-15 00:00:00

  • A Bayesian hierarchical variable selection prior for pathway-based GWAS using summary statistics.

    abstract::While genome-wide association studies (GWASs) have been widely used to uncover associations between diseases and genetic variants, standard SNP-level GWASs often lack the power to identify SNPs that individually have a moderate effect size but jointly contribute to the disease. To overcome this problem, pathway-based ...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.8442

    authors: Yang Y,Basu S,Zhang L

    更新日期:2020-03-15 00:00:00

  • Non-parametric methods for recurrent event data with informative and non-informative censorings.

    abstract::Recurrent event data are commonly encountered in health-related longitudinal studies. In this paper time-to-events models for recurrent event data are studied with non-informative and informative censorings. In statistical literature, the risk set methods have been confirmed to serve as an appropriate and efficient ap...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.1029

    authors: Wang MC,Chiang CT

    更新日期:2002-02-15 00:00:00

  • Maximum likelihood estimation of the kappa coefficient from models of matched binary responses.

    abstract::We present an estimate of the kappa-coefficient of agreement between two methods of rating based on matched pairs of binary responses and show that the estimate depends on the common intraclass correlation coefficient between the pairs. Via Monte Carlo simulation, we investigate power of the test of significance on ka...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780140109

    authors: Shoukri MM,Martin SW,Mian IU

    更新日期:1995-01-15 00:00:00

  • Non-inferiority trials: the 'at least as good as' criterion with dichotomous data.

    abstract::The 'at least as good as' criterion, introduced by Laster and Johnson for a continuous response variate, is developed here for applications with dichotomous data. This approach is adaptive in nature, as the margin of non-inferiority is not taken as a fixed difference; it varies as a function of the positive control re...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.2476

    authors: Laster LL,Johnson MF,Kotler ML

    更新日期:2006-04-15 00:00:00

  • Estimating the stage-specific numbers of HIV infection using a Markov model and back-calculation.

    abstract::The back-calculation method has been used to estimate the number of HIV infections from AIDS incidence data in a particular population. We present an extension of back calculation that provides estimates of the numbers of HIV infectives in different stages of infection. We model the staging process with a time-depende...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4780110612

    authors: Longini IM Jr,Byers RH,Hessol NA,Tan WY

    更新日期:1992-04-01 00:00:00

  • Multivariate meta-analysis: a robust approach based on the theory of U-statistic.

    abstract::Meta-analysis is the methodology for combining findings from similar research studies asking the same question. When the question of interest involves multiple outcomes, multivariate meta-analysis is used to synthesize the outcomes simultaneously taking into account the correlation between the outcomes. Likelihood-bas...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/sim.4327

    authors: Ma Y,Mazumdar M

    更新日期:2011-10-30 00:00:00

  • On the statistical analysis of allelic-loss data.

    abstract::This paper concerns the statistical analysis of certain binary data arising in molecular studies of cancer. In allelic-loss experiments, tumour cell genomes are analysed at informative molecular marker loci to identify deleted chromosomal regions. The resulting binary data are used to infer properties of putative supp...

    journal_title:Statistics in medicine

    pub_type: 杂志文章

    doi:10.1002/(sici)1097-0258(19980715)17:13<1425::aid-s

    authors: Newton MA,Gould MN,Reznikoff CA,Haag JD

    更新日期:1998-07-15 00:00:00