Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions.

Abstract:

BACKGROUND:Optimal cutoff values for tests results involving continuous variables are often derived in a data-driven way. This approach, however, may lead to overly optimistic measures of diagnostic accuracy. We evaluated the magnitude of the bias in sensitivity and specificity associated with data-driven selection of cutoff values and examined potential solutions to reduce this bias. METHODS:Different sample sizes, distributions, and prevalences were used in a simulation study. We compared data-driven estimates of accuracy based on the Youden index with the true values and calculated the median bias. Three alternative approaches (assuming a specific distribution, leave-one-out, smoothed ROC curve) were examined for their ability to reduce this bias. RESULTS:The magnitude of bias caused by data-driven optimization of cutoff values was inversely related to sample size. If the true values for sensitivity and specificity are both 84%, the estimates in studies with a sample size of 40 will be approximately 90%. If the sample size increases to 200, the estimates will be 86%. The distribution of the test results had little impact on the amount of bias when sample size was held constant. More robust methods of optimizing cutoff values were less prone to bias, but the performance deteriorated if the underlying assumptions were not met. CONCLUSIONS:Data-driven selection of the optimal cutoff value can lead to overly optimistic estimates of sensitivity and specificity, especially in small studies. Alternative methods can reduce this bias, but finding robust estimates for cutoff values and accuracy requires considerable sample sizes.

journal_name

Clin Chem

journal_title

Clinical chemistry

authors

Leeflang MM,Moons KG,Reitsma JB,Zwinderman AH

doi

10.1373/clinchem.2007.096032

subject

Has Abstract

pub_date

2008-04-01 00:00:00

pages

729-37

issue

4

eissn

0009-9147

issn

1530-8561

pii

clinchem.2007.096032

journal_volume

54

pub_type

杂志文章
  • Mechanized amidolytic technique for determination of factor X and factor-X antigen, and its application to patients being treated with oral anticoagulants.

    abstract::We describe a mechanized chromogenic assay for factor X, the results of which correlate well with those for the one-stage clotting assays for factor X in which it is activated either via the extrinsic pathway by thromboplastin or directly by Russell's viper venom. We purified human factor X and raised monospecific ant...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: van Wijk EM,Kahlé LH,ten Cate JW

    更新日期:1980-06-01 00:00:00

  • Use of polymerase chain reaction to detect heterozygous familial hypercholesterolemia.

    abstract::We used a modification of the polymerase chain reaction (PCR), involving two pairs of oligonucleotide primers, to detect a mutation in the low-density lipoprotein (LDL) receptor gene, commonly occurring among patients with familial hypercholesterolemia (FH) in Finland. This mutation, called FH-Helsinki, involves a lar...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Keinänen M,Ojala JP,Helve E,Aalto-Setälä K,Kontula K,Kovanen PT

    更新日期:1990-06-01 00:00:00

  • Urinary polyamine excretion during the menstrual cycle.

    abstract::Urinary polyamines were measured by a fluorometric thin-layer chromatographic technique in samples collected daily throughout a single menstrual cycle by each of 13 healthy women. Polyamine excretion fluctuated considerably, but excretion of putrescine, spermidine, and spermine combined was consistently greatest durin...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Osterberg S,Rosén S,Heby O

    更新日期:1978-05-01 00:00:00

  • Immunoluminometric assay for measurement of the C-terminal endothelin-1 precursor fragment in human plasma.

    abstract:BACKGROUND:Endothelin-1 (ET-1), a potent vasoconstrictor, is difficult to measure because of its instability and its binding to receptors and plasma proteins. We report a rapid, robust way to indirectly quantify ET-1 release by measuring the C-terminal ET-1 precursor fragment (CT-proET-1) without an extraction step. M...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:10.1373/clinchem.2005.065581

    authors: Papassotiriou J,Morgenthaler NG,Struck J,Alonso C,Bergmann A

    更新日期:2006-06-01 00:00:00

  • Laboratory computing--process and information management supporting high-quality, cost-effective healthcare.

    abstract::One currently observes many healthcare institutions rushing to reengineer and install information systems with the expectation of achieving enhanced efficiency, competitiveness, and, it is hoped, higher patient satisfaction resulting from timely, high-quality care. Unfortunately, information system concepts, design, a...

    journal_title:Clinical chemistry

    pub_type: 杂志文章,评审

    doi:

    authors: Buffone GJ,Moreau DR

    更新日期:1995-09-01 00:00:00

  • Tartrate-resistant acid phosphatase isoform 5b as serum marker for osteoclastic activity.

    abstract:BACKGROUND:Tartrate-resistant acid phosphatase (AcP) 5b is a marker of osteoclastic activity and bone resorption. Immunoassays for serum TRAcP may lack sensitivity and specificity because of the presence of non-bone isoform 5a. The purpose of this study was to isolate the serum isoforms, quantify their disease-related ...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Janckila AJ,Takahashi K,Sun SZ,Yam LT

    更新日期:2001-01-01 00:00:00

  • Determination of 5-hydroxytryptophan in plasma by high-performance liquid chromatography and fluorometric detection after phthaldialdehyde reaction.

    abstract::We describe the sensitive and precise determination 5-hydroxytryptophan in plasma from patients treated with this compound. The method is based on solvent extraction of plasma, high-performance liquid chromatography and continuous fluorometric detection after reaction with phthaldialdehyde in strong acid. The combined...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Enbaek F,Magnussen I

    更新日期:1978-02-01 00:00:00

  • Production and certification of an enzyme reference material for gamma-glutamyltransferase (CRM 319). Part 2: Certification campaign.

    abstract::We describe the process of certification for a gamma-glutamyltransferase reference material (CRM no. 319). Fifteen laboratories participated to this interlaboratory evaluation. All steps of the measurements were controlled in an effort to locate potential sources of variations. In particular, the exclusion of some dat...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Schiele F,Muller J,Colinet E,Siest G

    更新日期:1987-11-01 00:00:00

  • Clinical performance of an in-line, ex vivo point-of-care monitor: a multicenter study.

    abstract:BACKGROUND:The management of critically ill infants and neonates includes frequent determination of arterial blood gas, electrolyte, and hematocrit values. An objective of attached point-of-care patient monitoring is to provide clinically relevant data without the adverse consequences associated with serial phlebotomy....

    journal_title:Clinical chemistry

    pub_type: 杂志文章,多中心研究

    doi:

    authors: Billman GF,Hughes AB,Dudell GG,Waldman E,Adcock LM,Hall DM,Orsini EN Jr,Koska AJ,Van Marter LJ,Finer NN,Kulhavy JC,Feld RD,Widness JA

    更新日期:2002-11-01 00:00:00

  • Immunochemical approach to the question of inactive subunits in cases of lactate dehydrogenase B subunit deficiency.

    abstract::If there were enzymatically inactive B subunits in a solution containing lactate dehydrogenase (LD) B4 prepared from the hemolysates of heterozygous individuals with LD-B subunit deficiency, the inactive subunits would compete with normal subunits for antibodies to the B4 subunit. Using the activity of normal LD-B4 as...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Maekawa M,Sudo K,Kanno T

    更新日期:1986-01-01 00:00:00

  • Plasma apolipoproteins A-I and B in survivors of myocardial infarction and in a control group.

    abstract::The values of apolipoproteins (apo) A-I and B were determined in a population sample of hospital outpatients with a standardized method to verify if the cutpoints calculated in a cross-sectional study in the US are usable with other populations. We also tested the apolipoproteins' ability to discriminate between healt...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Graziani MS,Zanolla L,Righetti G,Marchetti C,Mocarelli P,Marcovina SM

    更新日期:1998-01-01 00:00:00

  • Clinical evaluation of the Elecsys beta-CrossLaps serum assay, a new assay for degradation products of type I collagen C-tlopeptides.

    abstract:BACKGROUND:The Elecsys beta-CrossLaps serum assay measures type I collagen degradation fragments (beta-CTx) that contain the beta-isomerized octapeptide EKAHD-beta-GGR. We investigated the analytical performance of the assay and changes in beta-CrossLaps in patients with metabolic bone diseases. METHODS:The electroche...

    journal_title:Clinical chemistry

    pub_type: 临床试验,杂志文章

    doi:

    authors: Okabe R,Nakatsuka K,Inaba M,Miki T,Naka H,Masaki H,Moriguchi A,Nishizawa Y

    更新日期:2001-08-01 00:00:00

  • Toward Worldwide Hepcidin Assay Harmonization: Identification of a Commutable Secondary Reference Material.

    abstract:BACKGROUND:Absolute plasma hepcidin concentrations measured by various procedures differ substantially, complicating interpretation of results and rendering reference intervals method dependent. We investigated the degree of equivalence achievable by harmonization and the identification of a commutable secondary refere...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:10.1373/clinchem.2016.256768

    authors: van der Vorm LN,Hendriks JC,Laarakkers CM,Klaver S,Armitage AE,Bamberg A,Geurts-Moespot AJ,Girelli D,Herkert M,Itkonen O,Konrad RJ,Tomosugi N,Westerman M,Bansal SS,Campostrini N,Drakesmith H,Fillet M,Olbina G,Pasricha

    更新日期:2016-07-01 00:00:00

  • Stratus automated creatine kinase-MB assay evaluated: identification and elimination of falsely increased results associated with a high-molecular-mass form of alkaline phosphatase.

    abstract::We compared the performance of an automated assay of creatine kinase MB isoenzyme (CK-MB) mass (Stratus) with that of a CK-MB enzymatic assay routinely used at our institutions. Both of these assays use the same CK-MB-specific monoclonal antibody to immunocapture CK-MB, thus providing a direct means of comparing a mas...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Butch AW,Goodnow TT,Brown WS,McClellan A,Kessler G,Scott MG

    更新日期:1989-10-01 00:00:00

  • High-dose alpha-tocopherol therapy does not affect HDL subfractions in patients with coronary artery disease on statin therapy.

    abstract:BACKGROUND:Subfractions of HDL, particularly large HDL (HDL2), are inversely correlated with the severity of coronary artery disease (CAD). alpha-Tocopherol (AT) is the main lipid-soluble antioxidant in plasma. Results of a previous small study (n = 44) suggested that either a combination of an antioxidant cocktail [80...

    journal_title:Clinical chemistry

    pub_type: 杂志文章,随机对照试验

    doi:10.1373/clinchem.2006.078865

    authors: Singh U,Otvos J,Dasgupta A,de Lemos JA,Devaraj S,Jialal I

    更新日期:2007-03-01 00:00:00

  • Liquid-phase reactions started by rehydrating lyophilized reagents in a centrifugal analyzer.

    abstract::In the disposable rotor of the SAM microcentrifugal analyzer, various lyophilized reagents are predistributed in 24 33-microL cuvets, for determination of multiple analytes in one specimen (e.g., for patient profiles). We evaluated a prototype of this system, which can be used at 25, 30, or 37 degrees C; absorbance re...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Truchaud A,Glikmanas G,Gourmelin Y,Hersant J,Trepo D,Foray G,Collombel C

    更新日期:1985-09-01 00:00:00

  • Use of a quality-control plasma sample to decrease interassay variation in radioimmunoassays of apolipoprotein A-I.

    abstract::The apolipoprotein A-I (apo A-I) radioimmunoassay established in our laboratory involves use of purified apo A-I as the primary standard for quantifying apo A-I in plasma and a pooled plasma (quality-control sample) as a secondary standard to decrease interassay variation. The measured values for apo A-I increased ove...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Au YP,Hallaway BJ,Kottke BA

    更新日期:1986-07-01 00:00:00

  • Review of actual proficiency-testing performance under CLIA '67 (March 14, 1990) rules: perspective from the first year's data.

    abstract::Under the Clinical Laboratory Improvement Act of 1967 the Health Care Financing Administration's proficiency-testing requirement applies to approximately 12,000 hospital, reference, and large-clinic laboratories in the United States. The Wisconsin State Laboratory of Hygiene is approved by the Health Care Financing Ad...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Lanphear BJ,Burmeister BJ,Ehrmeyer SS,Laessig RH,Hassemer DJ

    更新日期:1992-07-01 00:00:00

  • Comparison of countercurrent immunoelectrophoretic assay with commercial radioimmunoassay kits for measuring prostatic acid phosphatase.

    abstract::We evaluated and compared five commercial radioimmunoassay kits with a standard counter-immunoelectrophoretic assay for the measurement of prostatic acid phosphatase in serum. Four of the five radioimmunoassays performed as described by the supplier with respect to sensitivity, stability, precision, linearity, analyti...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Wright GL Jr,Schellhammer PF,Brassil DN,Sieg SM,Leffell MS

    更新日期:1981-10-01 00:00:00

  • Meaning of serum free-thyroxin values in nonthyroidal illnesses: seven methods compared.

    abstract::Serum free thyroxin (FT4) was determined in 40 patients with various nonthyroidal illnesses. We studied seven methods: (1) a free thyroxin index calculated from total T4 and triiodothyronine resin uptake; (2) a free T4 index determined by enzyme inhibitor assays (Abbott's "Tetrazyme" and "Thyrozyme"); (3) a free T4 in...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Chan DW,Waud JM,Taylor E,Stem J,Drew H,Oropeza M,Sucupira MS

    更新日期:1983-12-01 00:00:00

  • Automated measurement of alpha-amylase isoenzymes with 6(3)-deoxymaltotriose as selective amylase inhibitor.

    abstract::We developed an automated method for measurement of alpha-amylase isoenzymes in serum by a single kinetic assay (SKA) and a double kinetic assay (DKA) with 2-chloro-4-nitrophenyl-6(5)-azido-6(5)-deoxy-beta-maltopentaoside as a substrate and 6(3)-deoxymaltotriose (DOG3) as a novel selective amylase inhibitor. DOG3 show...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Uchida R,Tokutake S,Motoyama Y,Hosoi K,Yamaji N

    更新日期:1995-04-01 00:00:00

  • Regulation of serum paraoxonase activity by genetic, nutritional, and lifestyle factors in the general population.

    abstract:BACKGROUND:Paraoxonase may protect lipoproteins and cell membranes from peroxidation, and alterations in the activity of this enzyme have been associated with some chronic diseases. Serum paraoxonase appears to be mainly under genetic control, but some studies suggest that environmental factors may also modulate its ac...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:10.1373/49.9.1491

    authors: Ferré N,Camps J,Fernández-Ballart J,Arija V,Murphy MM,Ceruelo S,Biarnés E,Vilella E,Tous M,Joven J

    更新日期:2003-09-01 00:00:00

  • Detection of Colorectal Cancer in Circulating Cell-Free DNA by Methylated CpG Tandem Amplification and Sequencing.

    abstract:BACKGROUND:Aberrant DNA hypermethylation of CpG islands occurs frequently throughout the genome in human colorectal cancer (CRC). A genome-wide DNA hypermethylation analysis technique using circulating cell-free DNA (cfDNA) is attractive for the noninvasive early detection of CRC and discrimination between CRC and othe...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:10.1373/clinchem.2019.301804

    authors: Li J,Zhou X,Liu X,Ren J,Wang J,Wang W,Zheng Y,Shi X,Sun T,Li Z,Kang A,Tang F,Wen L,Fu W

    更新日期:2019-07-01 00:00:00

  • Deamidation of glutamine by increased plasma gamma-glutamyltransferase is a source of rapid ammonia formation in blood and plasma specimens.

    abstract::Owing to increased enzymatic hydrolysis of glutamine, additional ammonia is formed in blood and plasma specimens with increased gamma-glutamyltransferase activity (gamma-GT, EC 2.3.2.2.). The close correlation between gamma-GT and glutaminase (EC 3.5.1.2) activities (r = 0.97) in plasma, the inhibition with 6-diazo-5-...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: da Fonseca-Wollheim F

    更新日期:1990-08-01 00:00:00

  • Comparison of LDL cholesterol concentrations by Friedewald calculation and direct measurement in relation to cardiovascular events in 27,331 women.

    abstract:BACKGROUND:National Cholesterol Education Program (NCEP) guidelines recommend development of direct assays for LDL cholesterol (LDL-C) measurement, but it is unclear how these assays compare with Friedewald calculation in predicting cardiovascular disease (CVD). METHODS:In a study of 27 331 healthy women with triglyce...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:10.1373/clinchem.2008.117929

    authors: Mora S,Rifai N,Buring JE,Ridker PM

    更新日期:2009-05-01 00:00:00

  • Detection of mitochondrial DNA mutations by temporal temperature gradient gel electrophoresis.

    abstract:BACKGROUND:A unique requirement for the molecular diagnosis of mitochondrial DNA (mtDNA) disorders is the ability to detect heteroplasmic mtDNA mutations and to distinguish them from homoplasmic sequence variations before further testing (e.g., sequencing) is performed. We evaluated the potential utility of temporal te...

    journal_title:Clinical chemistry

    pub_type: 临床试验,杂志文章

    doi:

    authors: Chen TJ,Boles RG,Wong LJ

    更新日期:1999-08-01 00:00:00

  • Use of magnetic circular dichroism spectroscopy for biologic monitoring of occupational exposures to toxicants.

    abstract::Zinc protoporphyrin is the predominent fluorescent porphyrin accumulating in erythrocytes as a result of chronic lead absorption or iron-deficiency anemia. Although diagnostic concentrations or thresholds for it in erythrocytes have been inferred, normal adult averages have not been established. We quantitatively asse...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Zygowicz ER,Hollebone BR,Perkins HM

    更新日期:1980-09-01 00:00:00

  • Evaluation of four assay methods for determination of tobramycin in human serum.

    abstract::Four assay procedures for tobramycin in serum--enzyme immunoassay (I), substrate-labeled fluorescent immunoassay (II), radioimmunoassay (III), and bioassay (IV)--were compared and evaluated by replicate and analytical recovery studies. I and II were about 50% more precise than III and IV. II was substantially more nea...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Acton WJ,Van Duyn OM,Allen LV,Flournoy DJ

    更新日期:1982-01-01 00:00:00

  • Measurement of urinary oxalate and citrate by capillary electrophoresis and indirect ultraviolet absorbance.

    abstract::We describe a method for measuring urinary oxalate and citrate with capillary electrophoresis (CE) and indirect ultraviolet absorbance detection. Sample preparation is minimal, requiring an acidification, brief centrifugation, and dilution. The method is rapid, with oxalate and citrate having mean migration times of 4...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Holmes RP

    更新日期:1995-09-01 00:00:00

  • Performance of an enzyme-linked immunosorbent assay system for antibodies to hepatitis C virus with two new antigens (c11/c7).

    abstract::We developed an enzyme-linked immunosorbent assay (ELISA) system for antibodies to the hepatitis C virus (HCV), using two new recombinant antigens (c11 and c7) derived from the HCV genome. The performance of this ELISA system (Imucheck HCV Ab) was examined. The CV values for both intra-assay precision and reproducibil...

    journal_title:Clinical chemistry

    pub_type: 杂志文章

    doi:

    authors: Saito M,Hasegawa A,Kashiwakuma T,Kohara M,Sugi M,Miki K,Yamamoto T,Mori H,Ohta Y,Tanaka E

    更新日期:1992-12-01 00:00:00