Skip to main content
Kent Academic Repository

Low-variance Forward Gradients using Direct Feedback Alignment and momentum

Bacho, Florian, Chu, Dominique (2024) Low-variance Forward Gradients using Direct Feedback Alignment and momentum. Neural Networks, 169 . pp. 572-583. ISSN 0893-6080. (doi:10.1016/j.neunet.2023.10.051) (KAR id:104083)

Abstract

Supervised learning in deep neural networks is commonly performed using error backpropagation. However, the sequential propagation of errors during the backward pass limits its scalability and applicability to low-powered neuromorphic hardware. Therefore, there is growing interest in finding local alternatives to backpropagation. Recently proposed methods based on forward-mode automatic differentiation suffer from high variance in large deep neural networks, which affects convergence. In this paper, we propose the Forward Direct Feedback Alignment algorithm that combines Activity-Perturbed Forward Gradients with Direct Feedback Alignment and momentum. We provide both theoretical proofs and empirical evidence that our proposed method achieves lower variance than forward gradient techniques. In this way, our approach enables faster convergence and better performance when compared to other local alternatives to backpropagation and opens a new perspective for the development of online learning algorithms compatible with neuromorphic systems.

Item Type: Article
DOI/Identification number: 10.1016/j.neunet.2023.10.051
Uncontrolled keywords: Backpropagation, Forward Gradient, Gradient estimates, Direct Feedback Alignment, Low variance
Subjects: Q Science > QA Mathematics (inc Computing science)
Divisions: Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing
SWORD Depositor: JISC Publications Router
Depositing User: JISC Publications Router
Date Deposited: 15 Mar 2024 15:14 UTC
Last Modified: 18 Mar 2024 11:59 UTC
Resource URI: https://kar.kent.ac.uk/id/eprint/104083 (The current URI for this page, for reference purposes)

University of Kent Author Information

Bacho, Florian.

Creator's ORCID:
CReDIT Contributor Roles:

Chu, Dominique.

Creator's ORCID: https://orcid.org/0000-0002-3706-2905
CReDIT Contributor Roles:
  • Depositors only (login required):

Total unique views for this document in KAR since July 2020. For more details click on the image.