Chu, Dominique, Bacho, Florian (2024) Random feedback alignment algorithms to train neural networks: why do they align? Machine Learning: Science and Technology, 5 (2). Article Number 025023. ISSN 2632-2153. (doi:10.1088/2632-2153/ad3ee5) (KAR id:105821)
PDF
Publisher pdf
Language: English
This work is licensed under a Creative Commons Attribution 4.0 International License.
|
|
Download this file (PDF/3MB) |
Preview |
Request a format suitable for use with assistive technology e.g. a screenreader | |
Official URL: https://doi.org/10.1088/2632-2153/ad3ee5 |
Abstract
Feedback alignment algorithms are an alternative to backpropagation to train neural networks, whereby some of the partial derivatives that are required to compute the gradient are replaced by random terms. This essentially transforms the update rule into a random walk in weight space. Surprisingly, learning still works with those algorithms, including training of deep neural networks. The performance of FA is generally attributed to an alignment of the update of the random walker with the true gradient—the eponymous gradient alignment—which drives an approximate gradient descent. The mechanism that leads to this alignment remains unclear, however. In this paper, we use mathematical reasoning and simulations to investigate gradient alignment. We observe that the feedback alignment update rule has fixed points, which correspond to extrema of the loss function. We show that gradient alignment is a stability criterion for those fixed points. It is only a necessary criterion for algorithm performance. Experimentally, we demonstrate that high levels of gradient alignment can lead to poor algorithm performance and that the alignment is not always driving the gradient descent.
Item Type: | Article |
---|---|
DOI/Identification number: | 10.1088/2632-2153/ad3ee5 |
Uncontrolled keywords: | feedback alignment; random walk; neural networks |
Subjects: | Q Science > QA Mathematics (inc Computing science) |
Divisions: | Divisions > Division of Computing, Engineering and Mathematical Sciences > School of Computing |
SWORD Depositor: | JISC Publications Router |
Depositing User: | JISC Publications Router |
Date Deposited: | 03 May 2024 10:18 UTC |
Last Modified: | 07 May 2024 13:37 UTC |
Resource URI: | https://kar.kent.ac.uk/id/eprint/105821 (The current URI for this page, for reference purposes) |
- Link to SensusAccess
- Export to:
- RefWorks
- EPrints3 XML
- BibTeX
- CSV
- Depositors only (login required):