Hardware-Efficient On-line Learning through Pipelined Truncated-Error Backpropagation in Binary-State Networks.

Abstract:

:Artificial neural networks (ANNs) trained using backpropagation are powerful learning architectures that have achieved state-of-the-art performance in various benchmarks. Significant effort has been devoted to developing custom silicon devices to accelerate inference in ANNs. Accelerating the training phase, however, has attracted relatively little attention. In this paper, we describe a hardware-efficient on-line learning technique for feedforward multi-layer ANNs that is based on pipelined backpropagation. Learning is performed in parallel with inference in the forward pass, removing the need for an explicit backward pass and requiring no extra weight lookup. By using binary state variables in the feedforward network and ternary errors in truncated-error backpropagation, the need for any multiplications in the forward and backward passes is removed, and memory requirements for the pipelining are drastically reduced. Further reduction in addition operations owing to the sparsity in the forward neural and backpropagating error signal paths contributes to highly efficient hardware implementation. For proof-of-concept validation, we demonstrate on-line learning of MNIST handwritten digit classification on a Spartan 6 FPGA interfacing with an external 1Gb DDR2 DRAM, that shows small degradation in test error performance compared to an equivalently sized binary ANN trained off-line using standard back-propagation and exact errors. Our results highlight an attractive synergy between pipelined backpropagation and binary-state networks in substantially reducing computation and memory requirements, making pipelined on-line learning practical in deep networks.

journal_name

Front Neurosci

authors

Mostafa H,Pedroni B,Sheik S,Cauwenberghs G

doi

10.3389/fnins.2017.00496

subject

Has Abstract

pub_date

2017-09-06 00:00:00

pages

496

eissn

1662-4548

issn

1662-453X

journal_volume

11

pub_type

杂志文章
  • Manipulation of the Duration of the Initial Self-Control Task Within the Sequential-Task Paradigm: Effect on Exercise Performance.

    abstract::Self-control exertion on an initial task has been associated with impaired performance on subsequent physical tasks also requiring self-control; an effect suggested to be mediated by changes in perceptions of pain and motivation. However, the effects of spending longer on the initial self-control task are unknown. Thi...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2020.571312

    authors: Boat R,Hunte R,Welsh E,Dunn A,Treadwell E,Cooper SB

    更新日期:2020-10-08 00:00:00

  • Hypoxia in CNS Pathologies: Emerging Role of miRNA-Based Neurotherapeutics and Yoga Based Alternative Therapies.

    abstract::Cellular respiration is a vital process for the existence of life. Any condition that results in deprivation of oxygen (also termed as hypoxia) may eventually lead to deleterious effects on the functioning of tissues. Brain being the highest consumer of oxygen is prone to increased risk of hypoxia-induced neurological...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2017.00386

    authors: Minhas G,Mathur D,Ragavendrasamy B,Sharma NK,Paanu V,Anand A

    更新日期:2017-07-11 00:00:00

  • Role of central vagal 5-HT3 receptors in gastrointestinal physiology and pathophysiology.

    abstract::Vagal neurocircuits are vitally important in the co-ordination and modulation of GI reflexes and homeostatic functions. 5-hydroxytryptamine (5-HT; serotonin) is critically important in the regulation of several of these autonomic gastrointestinal (GI) functions including motility, secretion and visceral sensitivity. W...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2015.00413

    authors: Browning KN

    更新日期:2015-10-29 00:00:00

  • White Matter Tract Segmentation as Multiple Linear Assignment Problems.

    abstract::Diffusion magnetic resonance imaging (dMRI) allows to reconstruct the main pathways of axons within the white matter of the brain as a set of polylines, called streamlines. The set of streamlines of the whole brain is called the tractogram. Organizing tractograms into anatomically meaningful structures, called tracts,...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2017.00754

    authors: Sharmin N,Olivetti E,Avesani P

    更新日期:2018-02-06 00:00:00

  • Asiatic Acid Prevents Retinal Ganglion Cell Apoptosis in a Rat Model of Glaucoma.

    abstract::Asiatic acid (AA), a pentacyclic triterpene derived from the tropical medicinal plant Centella asiatica, has been widely used as an antioxidant and anti-inflammatory agent. Evidence regarding the neuroprotective properties of AA is emerging. However, the protective effects of AA and its mechanism in glaucoma are poorl...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2018.00489

    authors: Huang W,Gao F,Hu F,Huang J,Wang M,Xu P,Zhang R,Chen J,Sun X,Zhang S,Wu J

    更新日期:2018-07-20 00:00:00

  • Online tracking of the contents of conscious perception using real-time fMRI.

    abstract::Perception is an active process that interprets and structures the stimulus input based on assumptions about its possible causes. We use real-time functional magnetic resonance imaging (rtfMRI) to investigate a particularly powerful demonstration of dynamic object integration in which the same physical stimulus interm...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2014.00116

    authors: Reichert C,Fendrich R,Bernarding J,Tempelmann C,Hinrichs H,Rieger JW

    更新日期:2014-05-23 00:00:00

  • Tamoxifen Activation of Cre-Recombinase Has No Persisting Effects on Adult Neurogenesis or Learning and Anxiety.

    abstract::Adult neurogenesis is a tightly regulated process continuously taking place in the central nervous system of most mammalian species. In neuroscience research, transgenic animals bearing the tamoxifen-inducible CreERT2-Lox system are widely used. In this study, we made use of a Nestin-CreERT2/R26R-YFP transgenic mouse ...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2017.00027

    authors: Rotheneichner P,Romanelli P,Bieler L,Pagitsch S,Zaunmair P,Kreutzer C,König R,Marschallinger J,Aigner L,Couillard-Després S

    更新日期:2017-02-01 00:00:00

  • Full Field Masking Causes Reversals in Perceived Event Order.

    abstract::We generally experience a stable visual world in spite of regular disruptions caused by our own movements (saccades, blinks) or by the visual input itself (flashes, occlusions). In trying to understand the mechanisms responsible for this stability, saccades have been particularly well-studied, and a number of peri-sac...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2020.00217

    authors: Chota S,McLelland D,Lavergne L,Zimmermann E,Cavanagh P,VanRullen R

    更新日期:2020-03-17 00:00:00

  • Impaired Motor Skill Acquisition Using Mirror Visual Feedback Improved by Transcranial Direct Current Stimulation (tDCS) in Patients With Parkinson's Disease.

    abstract::Recent non-invasive brain stimulation techniques in combination with motor training can enhance neuroplasticity and learning. It is reasonable to assume that such neuroplasticity-based interventions constitute a useful rehabilitative tool for patients with Parkinson's Disease (PD). Regarding motor skill training, many...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2019.00602

    authors: Horiba M,Ueki Y,Nojima I,Shimizu Y,Sahashi K,Itamoto S,Suzuki A,Yamada G,Matsukawa N,Wada I

    更新日期:2019-06-19 00:00:00

  • Evolution of Orexin Neuropeptide System: Structure and Function.

    abstract::Orexins are hypothalamic neuropeptides that were initially identified in the rat brain as endogenous ligands for an (previously) orphan G-protein-coupled receptor (GPCR). They are multitasking peptides involved in many physiological functions, including regulation of feeding behavior, wakefulness and autonomic/neuroen...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2020.00691

    authors: Soya S,Sakurai T

    更新日期:2020-07-10 00:00:00

  • Neuronal migration and protein kinases.

    abstract::The formation of the six-layered structure of the mammalian cortex via the inside-out pattern of neuronal migration is fundamental to neocortical functions. Extracellular cues such as Reelin induce intracellular signaling cascades through the protein phosphorylation. Migrating neurons also have intrinsic machineries t...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2014.00458

    authors: Ohshima T

    更新日期:2015-01-13 00:00:00

  • Masking the Integration of Complementary Shape Cues.

    abstract::Retinal and cortical mechanisms provide for persistence of visual information across intervals of many hundreds of milliseconds, which supports the integration of partial shape cues. The present experiments displayed unknown shapes in a match recognition task, wherein a target shape was quickly followed by a compariso...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2019.00178

    authors: Geoly A,Greene E

    更新日期:2019-03-13 00:00:00

  • Induction of NTPDase1/CD39 by Reactive Microglia and Macrophages Is Associated With the Functional State During EAE.

    abstract::Purinergic signaling is critically involved in neuroinflammation associated with multiple sclerosis (MS) and its major inflammatory animal model, experimental autoimmune encephalomyelitis (EAE). Herein, we explored the expression of ectonucleoside triphosphate diphosphohydrolase1 (NTPDase1/CD39) in the spinal cord, at...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2019.00410

    authors: Jakovljevic M,Lavrnja I,Bozic I,Milosevic A,Bjelobaba I,Savic D,Sévigny J,Pekovic S,Nedeljkovic N,Laketa D

    更新日期:2019-04-26 00:00:00

  • Sleep in Isolated, Confined, and Extreme (ICE): A Review on the Different Factors Affecting Human Sleep in ICE.

    abstract::The recently renewed focus on the human exploration of outer space has boosted the interest toward a variety of questions regarding health of astronauts and cosmonauts. Among the others, sleep has traditionally been considered a central issue. To extend the research chances, human sleep alterations have been investiga...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2020.00851

    authors: Zivi P,De Gennaro L,Ferlazzo F

    更新日期:2020-08-11 00:00:00

  • Awake Rat Brain Functional Magnetic Resonance Imaging Using Standard Radio Frequency Coils and a 3D Printed Restraint Kit.

    abstract::Functional magnetic resonance imaging (fMRI) is a powerful noninvasive tool for studying spontaneous resting state functional connectivity (RSFC) in laboratory animals. Brain function can be significantly affected by generally used anesthetics, however, rendering the need for awake imaging. Only a few different awake ...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2018.00548

    authors: Stenroos P,Paasonen J,Salo RA,Jokivarsi K,Shatillo A,Tanila H,Gröhn O

    更新日期:2018-08-20 00:00:00

  • Non-parametric statistical thresholding for sparse magnetoencephalography source reconstructions.

    abstract::Uncovering brain activity from magnetoencephalography (MEG) data requires solving an ill-posed inverse problem, greatly confounded by noise, interference, and correlated sources. Sparse reconstruction algorithms, such as Champagne, show great promise in that they provide focal brain activations robust to these confoun...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2012.00186

    authors: Owen JP,Sekihara K,Nagarajan SS

    更新日期:2012-12-26 00:00:00

  • Toward the Understanding of Topographical and Spectral Signatures of Infant Movement Artifacts in Naturalistic EEG.

    abstract::Electroencephalography (EEG) is perhaps the most widely used brain-imaging technique for pediatric populations. However, EEG signals are prone to distortion by motion. Compared to adults, infants' motion is both more frequent and less stereotypical yet motion effects on the infant EEG signal are largely undocumented. ...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2020.00352

    authors: Georgieva S,Lester S,Noreika V,Yilmaz MN,Wass S,Leong V

    更新日期:2020-04-28 00:00:00

  • Coding of reward probability and risk by single neurons in animals.

    abstract::Probability and risk are important factors for value-based decision making and optimal foraging. In order to survive in an unpredictable world, organisms must be able to assess the probability and risk attached to future events and use this information to generate adaptive behavior. Recent studies in non-human primate...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2011.00121

    authors: Burke CJ,Tobler PN

    更新日期:2011-10-11 00:00:00

  • Frequency-Following Responses to Complex Tones at Different Frequencies Reflect Different Source Configurations.

    abstract::The neural generators of the frequency-following response (FFR), a neural response widely used to study the human auditory system, remain unclear. There is evidence that the balance between cortical and subcortical contributions to the FFR varies with stimulus frequency. In this study, we tried to clarify whether this...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2019.00130

    authors: Zhang X,Gong Q

    更新日期:2019-02-26 00:00:00

  • The Energy Metabolism Dysfunction in Psychiatric Disorders Postmortem Brains: Focus on Proteomic Evidence.

    abstract::Psychiatric disorders represent a great medical and social challenge and people suffering from these conditions face many impairments regarding personal and professional life. In addition, a mental disorder will manifest itself in approximately one quarter of the world's population at some period of their life. Dysfun...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2017.00493

    authors: Zuccoli GS,Saia-Cereda VM,Nascimento JM,Martins-de-Souza D

    更新日期:2017-09-07 00:00:00

  • New Vision for Visual Prostheses.

    abstract::Developments of new strategies to restore vision and improving on current strategies by harnessing new advancements in material and electrical sciences, and biological and genetic-based technologies are of upmost health priorities around the world. Federal and private entities are spending billions of dollars on visua...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2020.00036

    authors: Farnum A,Pelled G

    更新日期:2020-02-18 00:00:00

  • Corticotropin-releasing factor-related peptides, serotonergic systems, and emotional behavior.

    abstract::Corticotropin-releasing factor (CRF) is a 41-amino acid neuropeptide that is involved in stress-related physiology and behavior, including control of the hypothalamic-pituitary-adrenal (HPA) axis. Members of the CRF family of neuropeptides, including urocortin 1 (UCN 1), UCN 2, and UCN 3, bind to the G protein-coupled...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2013.00169

    authors: Fox JH,Lowry CA

    更新日期:2013-09-20 00:00:00

  • Bisphenol A and phthalate endocrine disruption of parental and social behaviors.

    abstract::Perinatal exposure to endocrine disrupting chemicals (EDCs) can induce promiscuous neurobehavioral disturbances. Bisphenol A and phthalates are two widely prevalent and persistent EDCs reported to lead to such effects. Parental and social behaviors are especially vulnerable to endocrine disruption, as these traits are...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2015.00057

    authors: Rosenfeld CS

    更新日期:2015-03-03 00:00:00

  • Sparse Estimation of Resting-State Effective Connectivity From fMRI Cross-Spectra.

    abstract::In functional magnetic resonance imaging (fMRI), functional connectivity is conventionally characterized by correlations between fMRI time series, which are intrinsically undirected measures of connectivity. Yet, some information about the directionality of network connections can nevertheless be extracted from the ma...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2018.00287

    authors: Lennartz C,Schiefer J,Rotter S,Hennig J,LeVan P

    更新日期:2018-05-08 00:00:00

  • Topography of the complete corticopontine projection: from experiments to principal Maps.

    abstract::The mammalian brain is characterized by orderly spatial distribution of its cellular components, commonly referred to as topographical organization. The topography of cortical and subcortical maps is thought to represent functional or computational properties. In the present investigation, we have studied map transfor...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/neuro.01.1.1.016.2007

    authors: Leergaard TB,Bjaalie JG

    更新日期:2007-10-15 00:00:00

  • Populations Norms for "SLURP"-An iPad App for Quantification of Visuomotor Coordination Testing.

    abstract::Currently the integrity of brain function that drives behavior is predominantly measured in terms of pure motor function, yet most human behavior is visually driven. A means of easily quantifying such visually-driven brain function for comparison against population norms is lacking. Analysis of eye-hand coordination (...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2019.00711

    authors: Junghans BM,Khuu SK

    更新日期:2019-07-10 00:00:00

  • The Melanin-Concentrating Hormone (MCH) System: A Tale of Two Peptides.

    abstract::The melanin-concentrating hormone (MCH) system is a robust integrator of exogenous and endogenous information, modulating arousal and energy balance in mammals. Its predominant function in teleosts, however, is to concentrate melanin in the scales, contributing to the adaptive color change observed in several teleost ...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2019.01280

    authors: Diniz GB,Bittencourt JC

    更新日期:2019-11-26 00:00:00

  • The Functional Tactile Object Recognition Test: A Unidimensional Measure With Excellent Internal Consistency for Haptic Sensing of Real Objects After Stroke.

    abstract:Introduction:Our hands, with their exquisite sensors, work in concert with our sensing brain to extract sensory attributes of objects as we engage in daily activities. One in two people with stroke experience impaired body sensation, with negative impact on hand use and return to previous valued activities. Valid, quan...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2020.542590

    authors: Carey LM,Mak-Yuen YYK,Matyas TA

    更新日期:2020-09-23 00:00:00

  • An Axiology of Information Security for Futuristic Neuroprostheses: Upholding Human Values in the Context of Technological Posthumanization.

    abstract::Previous works exploring the challenges of ensuring information security for neuroprosthetic devices and their users have typically built on the traditional InfoSec concept of the "CIA Triad" of confidentiality, integrity, and availability. However, we argue that the CIA Triad provides an increasingly inadequate found...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章

    doi:10.3389/fnins.2017.00605

    authors: Gladden ME

    更新日期:2017-11-07 00:00:00

  • Fluxes of lactate into, from, and among gap junction-coupled astrocytes and their interaction with noradrenaline.

    abstract::Lactate is a versatile metabolite with important roles in modulation of brain glucose utilization rate (CMRglc), diagnosis of brain-injured patients, redox- and receptor-mediated signaling, memory, and alteration of gene transcription. Neurons and astrocytes release and accumulate lactate using equilibrative monocarbo...

    journal_title:Frontiers in neuroscience

    pub_type: 杂志文章,评审

    doi:10.3389/fnins.2014.00261

    authors: Hertz L,Gibbs ME,Dienel GA

    更新日期:2014-09-09 00:00:00