当前位置： SCI文献检索 > PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA期刊下所有文献 > Prevalence of neural collapse during the terminal phase of deep learning training.

Prevalence of neural collapse during the terminal phase of deep learning training.

Abstract：

:Modern practice for training classification deepnets involves a terminal phase of training (TPT), which begins at the epoch where training error first vanishes. During TPT, the training error stays effectively zero, while training loss is pushed toward zero. Direct measurements of TPT, for three prototypical deepnet architectures and across seven canonical classification datasets, expose a pervasive inductive bias we call neural collapse (NC), involving four deeply interconnected phenomena. (NC1) Cross-example within-class variability of last-layer training activations collapses to zero, as the individual activations themselves collapse to their class means. (NC2) The class means collapse to the vertices of a simplex equiangular tight frame (ETF). (NC3) Up to rescaling, the last-layer classifiers collapse to the class means or in other words, to the simplex ETF (i.e., to a self-dual configuration). (NC4) For a given activation, the classifier's decision collapses to simply choosing whichever class has the closest train class mean (i.e., the nearest class center [NCC] decision rule). The symmetric and very simple geometry induced by the TPT confers important benefits, including better generalization performance, better robustness, and better interpretability.

journal_name

Proc Natl Acad Sci U S A

journal_title

Proceedings of the National Academy of Sciences of the United States of America

authors

Papyan V,Han XY,Donoho DL

doi

10.1073/pnas.2015509117

subject

Has Abstract

pub_date

2020-10-06 00:00:00

pages

24652-24663

issue

40

eissn

0027-8424

issn

1091-6490

pii

2015509117

journal_volume

117

pub_type

杂志文章

相关文献

PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA文献大全
  • Ferulate-5-hydroxylase from Arabidopsis thaliana defines a new family of cytochrome P450-dependent monooxygenases.

    abstract：:The fah1 mutant of Arabidopsis is defective in the accumulation of sinapic acid-derived metabolites, including the guaiacyl-syringyl lignin typical of angiosperms. Earlier results indicated that the FAH1 locus encodes ferulate-5-hydroxylase (F5H), a cytochrome P450-dependent monooxygenase (P450) of the general phenylp...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.93.14.6869

    authors： Meyer K,Cusumano JC,Somerville C,Chapple CC

    更新日期：1996-07-09 00:00:00

  • Molecular comparison of alpha 1- and alpha 2-adrenergic receptors suggests that these proteins are structurally related "isoreceptors".

    abstract：:The structures of human platelet alpha 2-adrenergic receptors and rat liver alpha 1-adrenergic receptors were compared by utilizing isoelectric focusing, NaDodSO4/PAGE, and monoclonal antibody crossreactivity. Digitonin-solubilized alpha 1- and alpha 2-adrenergic receptors have an identical isoelectric point of 4.6. U...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.82.14.4842

    authors： Shreeve SM,Fraser CM,Venter JC

    更新日期：1985-07-01 00:00:00

  • Genomic analysis of human and mouse TCL1 loci reveals a complex of tightly clustered genes.

    abstract：:TCL1 and TCL1b genes on human chromosome 14q23.1 are activated in T cell leukemias by translocations and inversions at 14q32.1, juxtaposing them to regulatory elements of T cell receptor genes. In this report we present the cloning, mapping, and expression analysis of the human and murine TCL1/Tcl1 locus. In addition ...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.96.25.14418

    authors： Hallas C,Pekarsky Y,Itoyama T,Varnum J,Bichi R,Rothstein JL,Croce CM

    更新日期：1999-12-07 00:00:00

  • Ab initio molecular dynamics and quasichemical study of H+(aq).

    abstract：:The excess proton in water, H(+)(aq), plays a fundamental role in aqueous solution chemistry. Its solution thermodynamic properties are essential to molecular descriptions of that chemistry and for validation of dynamical calculations. Within the quasichemical theory of solutions those thermodynamic properties are con...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.0408071102

    authors： Asthagiri D,Pratt LR,Kress JD

    更新日期：2005-05-10 00:00:00

  • Microbial iron transport via a siderophore shuttle: a membrane ion transport paradigm.

    abstract：:A mechanism of ion transport across membranes is reported. Microbial transport of Fe(3+) generally delivers iron, a growth-limiting nutrient, to cells via highly specific siderophore-mediated transport systems. In contrast, iron transport in the fresh water bacterium Aeromonas hydrophila is found to occur by means of ...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.200318797

    authors： Stintzi A,Barnes C,Xu J,Raymond KN

    更新日期：2000-09-26 00:00:00

  • Delays in Pseudomonas aeruginosa quorum-controlled gene expression are conditional.

    abstract：:Acyl-homoserine lactone (acyl-HSL) signaling is thought to mediate quorum sensing in many species of Proteobacteria. The opportunistic human pathogen Pseudomonas aeruginosa uses acyl-HSLs to regulate hundreds of genes, including many that code for extracellular virulence factors. The idea that the P. aeruginosa acyl-H...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.0503728102

    authors： Yarwood JM,Volper EM,Greenberg EP

    更新日期：2005-06-21 00:00:00

  • Apoptosis of nur77/N10-transgenic thymocytes involves the Fas/Fas ligand pathway.

    abstract：:The orphan nuclear receptor Nur77/N10 has recently been demonstrated to be involved in apoptosis of T cell hybridomas. We report here that chronic expression of Nur77/N10 in thymocytes of transgenic mice results in a dramatic reduction of CD4+CD8+ double-positive as well as CD4+CD8- and CD4-CD8+ single-positive cell p...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.93.11.5533

    authors： Weih F,Ryseck RP,Chen L,Bravo R

    更新日期：1996-05-28 00:00:00

  • Sprouting from chicken embryo dorsal root ganglia induced by nerve growth factor is specifically inhibited by affinity-purified antiganglioside antibodies.

    abstract：:The involvement of gangliosides in processes related to nerve regeneration and sprouting has been demonstrated recently. The type of interaction by which gangliosides may influence neuronal sprouting was investigated in the present work. Affinity-purified rabbit anti-GM1 antibodies were found to block the sprouting fr...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.79.19.6080

    authors： Schwartz M,Spirman N

    更新日期：1982-10-01 00:00:00

  • Mesoscale texture of cement hydrates.

    abstract：:Strength and other mechanical properties of cement and concrete rely upon the formation of calcium-silicate-hydrates (C-S-H) during cement hydration. Controlling structure and properties of the C-S-H phase is a challenge, due to the complexity of this hydration product and of the mechanisms that drive its precipitatio...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.1520487113

    authors： Ioannidou K,Krakowiak KJ,Bauchy M,Hoover CG,Masoero E,Yip S,Ulm FJ,Levitz P,Pellenq RJ,Del Gado E

    更新日期：2016-02-23 00:00:00

  • Structural gene for beta-nerve growth factor not defective in familial dysautonomia.

    abstract：:The developmental loss of neurons in sympathetic, sensory, and some parasympathetic ganglia in familial dysautonomia suggests an inherited defect in the action of beta-nerve growth factor (beta-NGF). The role of this growth factor in dysautonomia has been difficult to resolve as there is no known source of authentic h...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.81.13.4213

    authors： Breakefield XO,Orloff G,Castiglione C,Coussens L,Axelrod FB,Ullrich A

    更新日期：1984-07-01 00:00:00

  • Time-evolving genetic networks reveal a NAC troika that negatively regulates leaf senescence in Arabidopsis.

    abstract：:Senescence is controlled by time-evolving networks that describe the temporal transition of interactions among senescence regulators. Here, we present time-evolving networks for NAM/ATAF/CUC (NAC) transcription factors in Arabidopsis during leaf aging. The most evident characteristic of these time-dependent networks w...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.1721523115

    authors： Kim HJ,Park JH,Kim J,Kim JJ,Hong S,Kim J,Kim JH,Woo HR,Hyeon C,Lim PO,Nam HG,Hwang D

    更新日期：2018-05-22 00:00:00

  • Robustness, scalability, and integration of a wound-response gene expression signature in predicting breast cancer survival.

    abstract：:Based on the hypothesis that features of the molecular program of normal wound healing might play an important role in cancer metastasis, we previously identified consistent features in the transcriptional response of normal fibroblasts to serum, and used this "wound-response signature" to reveal links between wound h...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.0409462102

    authors： Chang HY,Nuyten DS,Sneddon JB,Hastie T,Tibshirani R,Sørlie T,Dai H,He YD,van't Veer LJ,Bartelink H,van de Rijn M,Brown PO,van de Vijver MJ

    更新日期：2005-03-08 00:00:00

  • Estrogen promotes the survival and pulmonary metastasis of tuberin-null cells.

    abstract：:Lymphangioleiomyomatosis (LAM) is an often fatal disease primarily affecting young women in which tuberin (TSC2)-null cells metastasize to the lungs. The mechanisms underlying the striking female predominance of LAM are unknown. We report here that 17-beta-estradiol (E(2)) causes a 3- to 5-fold increase in pulmonary m...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.0810790106

    authors： Yu JJ,Robb VA,Morrison TA,Ariazi EA,Karbowniczek M,Astrinidis A,Wang C,Hernandez-Cuebas L,Seeholzer LF,Nicolas E,Hensley H,Jordan VC,Walker CL,Henske EP

    更新日期：2009-02-24 00:00:00

  • Role of DNA mismatch repair and p53 in signaling induction of apoptosis by alkylating agents.

    abstract：:All cells are unavoidably exposed to chemicals that can alkylate DNA to form genotoxic damage. Among the various DNA lesions formed, O(6)-alkylguanine lesions can be highly cytotoxic, and we recently demonstrated that O(6)-methylguanine (O(6)MeG) and O(6)-chloroethylguanine (O(6)CEG) specifically initiate apoptosis in...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.96.19.10764

    authors： Hickman MJ,Samson LD

    更新日期：1999-09-14 00:00:00

  • Subcortical differentiation of stop consonants relates to reading and speech-in-noise perception.

    abstract：:Children with reading impairments have deficits in phonological awareness, phonemic categorization, speech-in-noise perception, and psychophysical tasks such as frequency and temporal discrimination. Many of these children also exhibit abnormal encoding of speech stimuli in the auditory brainstem, even though response...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.0901123106

    authors： Hornickel J,Skoe E,Nicol T,Zecker S,Kraus N

    更新日期：2009-08-04 00:00:00

  • Identification of a human CD8+ regulatory T cell subset that mediates suppression through the chemokine CC chemokine ligand 4.

    abstract：:Regulatory T cells (Treg) comprise multiple subsets and are important in controlling immunity and inflammation. However, the induction and mode of action of the various distinct Treg subsets remain ill defined, particularly in humans. Here, we describe a human CD8+ lymphocyte activation gene-3 (LAG-3)+CD25+FoxP3+ Treg...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.0702257104

    authors： Joosten SA,van Meijgaarden KE,Savage ND,de Boer T,Triebel F,van der Wal A,de Heer E,Klein MR,Geluk A,Ottenhoff TH

    更新日期：2007-05-08 00:00:00

  • Heterogeneous activation thresholds to cytokines in genetically distinct endothelial cells: evidence for diverse transcriptional responses.

    abstract：:It is well accepted that the induction of endothelial cell (EC) adhesion molecules is a critical component in acute inflammatory responses as well as allogeneic interactions in vascularized allografts and, possibly, atherogenesis. The "inflammatory triad" of interleukin 1 (IL-1), tumor necrosis factor, and lipopolysac...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.91.9.3994

    authors： Bender JR,Sadeghi MM,Watson C,Pfau S,Pardi R

    更新日期：1994-04-26 00:00:00

  • BCR/ABL inhibition by an escort/phosphatase fusion protein.

    abstract：:Cellular transformation by the BCR/ABL oncogene depends on the ABL-encoded tyrosine kinase activity. To block BCR/ABL function, we created a unique tyrosine phosphatase by fusing the catalytic domain of SHP1 (SHP1c) to the ABL binding domain (ABD) of RIN1, an established binding partner and substrate for c-ABL and BCR...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.210253497

    authors： Lim YM,Wong S,Lau G,Witte ON,Colicelli J

    更新日期：2000-10-24 00:00:00

  • Dynamic periplasmic chaperone reservoir facilitates biogenesis of outer membrane proteins.

    abstract：:Outer membrane protein (OMP) biogenesis is critical to bacterial physiology because the cellular envelope is vital to bacterial pathogenesis and antibiotic resistance. The process of OMP biogenesis has been studied in vivo, and each of its components has been studied in isolation in vitro. This work integrates paramet...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.1601002113

    authors： Costello SM,Plummer AM,Fleming PJ,Fleming KG

    更新日期：2016-08-16 00:00:00

  • NF-kappa B prevents beta cell death and autoimmune diabetes in NOD mice.

    abstract：:Whereas NF-kappaB has potent antiapoptotic function in most cell types, it was reported that in pancreatic beta cells it serves a proapoptotic function and may contribute to the pathogenesis of autoimmune type 1 diabetes. To investigate the role of beta cell NF-kappaB in autoimmune diabetes, we produced transgenic mic...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.0610690104

    authors： Kim S,Millet I,Kim HS,Kim JY,Han MS,Lee MK,Kim KW,Sherwin RS,Karin M,Lee MS

    更新日期：2007-02-06 00:00:00

  • Lipid bilayer mediates ion-channel cooperativity in a model of hair-cell mechanotransduction.

    abstract：:Mechanoelectrical transduction in the inner ear is a biophysical process underlying the senses of hearing and balance. The key players involved in this process are mechanosensitive ion channels. They are located in the stereocilia of hair cells and opened by the tension in specialized molecular springs, the tip links,...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.1713135114

    authors： Gianoli F,Risler T,Kozlov AS

    更新日期：2017-12-19 00:00:00

  • Neural basis of contagious itch and why some people are more prone to it.

    abstract：:Watching someone scratch himself can induce feelings of itchiness in the perceiver. This provides a unique opportunity to characterize the neural basis of subjective experiences of itch, independent of changes in peripheral inputs. In this study, we first established that the social contagion of itch is essentially a ...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.1216160109

    authors： Holle H,Warne K,Seth AK,Critchley HD,Ward J

    更新日期：2012-11-27 00:00:00

  • Trans effect of the E1 region of adenoviruses on the expression of a prokaryotic gene in mammalian cells: resistance to 5' -CCGG- 3' methylation.

    abstract：:The plasmid construct pSVO-CAT has been used to test adenovirus promoter activities in the unmethylated or methylated state. We have now observed that the E2A late promoter of adenovirus type 2 (Ad2) DNA also activated the chloramphenicol acetyltransferase (CAT) gene upon transfection of the pAd2E2A-CAT construct into...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.83.6.1598

    authors： Langner KD,Weyer U,Doerfler W

    更新日期：1986-03-01 00:00:00

  • A multiplasmid approach to preparing large libraries of polyketides.

    abstract：:A three-plasmid system for heterologous expression of 6-deoxyerythronolide B synthase (DEBS) has been developed to facilitate combinatorial biosynthesis of polyketides made by type I modular polyketide synthases (PKSs). The eryA PKS genes encoding the three DEBS subunits were individually cloned into three compatible ...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.96.21.11740

    authors： Xue Q,Ashley G,Hutchinson CR,Santi DV

    更新日期：1999-10-12 00:00:00

  • Arabidopsis JAGGED links floral organ patterning to tissue growth by repressing Kip-related cell cycle inhibitors.

    abstract：:Plant morphogenesis requires coordinated cytoplasmic growth, oriented cell wall extension, and cell cycle progression, but it is debated which of these processes are primary drivers for tissue growth and directly targeted by developmental genes. Here, we used ChIP high-throughput sequencing combined with transcriptome...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.1320457111

    authors： Schiessl K,Muiño JM,Sablowski R

    更新日期：2014-02-18 00:00:00

  • Amphiregulin: a bifunctional growth-modulating glycoprotein produced by the phorbol 12-myristate 13-acetate-treated human breast adenocarcinoma cell line MCF-7.

    abstract：:A glycoprotein, termed amphiregulin (AR), inhibits growth of several human carcinoma cells in culture and stimulates proliferation of human fibroblasts and certain other tumor cells. It has been purified to apparent homogeneity from serum-free conditioned medium of MCF-7 human breast carcinoma cells that had been trea...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.85.17.6528

    authors： Shoyab M,McDonald VL,Bradley JG,Todaro GJ

    更新日期：1988-09-01 00:00:00

  • Hair-specific expression of chloramphenicol acetyltransferase in transgenic mice under the control of an ultra-high-sulfur keratin promoter.

    abstract：:We have generated a transgenic mouse line by microinjection of a chimeric DNA fragment (KER-CAT) containing a hair-specific, murine ultra-high-sulfur keratin promoter (KER) fused to the coding region of the bacterial chloramphenicol acetyltransferase (CAT) gene. A 671-base pair (bp) stretch of the 5' promoter region w...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.87.17.6848

    authors： McNab AR,Andrus P,Wagner TE,Buhl AE,Waldon DJ,Kawabe TT,Rea TJ,Groppi V,Vogeli G

    更新日期：1990-09-01 00:00:00

  • Two classes of N-methyl-D-aspartate recognition sites: differential distribution and differential regulation by glycine.

    abstract：:The N-methyl-D-aspartate (NMDA) receptor, a subtype of excitatory amino acid receptor, mediates synaptic responses in many regions of the central nervous system. This receptor plays a critical role in the mechanisms of both synaptic plasticity and excitotoxicity. Although these receptors were generally thought to be a...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.85.24.9836

    authors： Monaghan DT,Olverman HJ,Nguyen L,Watkins JC,Cotman CW

    更新日期：1988-12-01 00:00:00

  • Unique characteristics of Ca2+ homeostasis of the trans-Golgi compartment.

    abstract：:Taking advantage of a fluorescent Ca(2+) indicator selectively targeted to the trans-Golgi lumen, we here demonstrate that its Ca(2+) homeostatic mechanisms are distinct from those of the other Golgi subcompartments: (i) Ca(2+) uptake depends exclusively on the activity of the secretory pathway Ca(2+) ATPase1 (SPCA1),...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.1004702107

    authors： Lissandron V,Podini P,Pizzo P,Pozzan T

    更新日期：2010-05-18 00:00:00

  • The weierstrass-laguerre transform.

    abstract：:An elegant expression is obtained for the product of the inverse Weierstrass-Laguerre transforms of two functions in terms of their convolution. It is also shown how the main result can be extended to hold for the product of the inverse Weierstrass-Laguerre transforms of several functions. ...

    journal_title：Proceedings of the National Academy of Sciences of the United States of America

    pub_type： 杂志文章

    doi：10.1073/pnas.68.3.554

    authors： Srivastava HM

    更新日期：1971-03-01 00:00:00