NeurIPS 2020

Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures

Meta Review

Strong empirical evaluation for Direct Feedback Alignment (DFA) on a broad range of tasks. The contributions are well summarized by R2's comments, "The work is motivated by arguing that DFA was so far only used on small datasets, and was shown to not perform well on computer vision tasks, in part because of the usage of CNNs in these settings. This survey challenges these views by conducting an extensive set of experiments using DFA to train s.o.t.a. models on s.o.t.a. benchmarks. The benchmarks include view synthesis, language modeling, recommender systems, and graph embedding. They compare the performance of these models to ones trained using a normal BP approach. The authors show that DFA can be competitive to classical BP in many scenarios, and also show how further improvements could be implemented. They also highlight potential benefits (e.g. parallelization) of training models with DFA vs BP." I agree with this. Authors have also addressed most of the reviewer's concerns in their rebuttal.