NeurIPS 2019
Sun Dec 8th through Sat the 14th, 2019 at Vancouver Convention Center
Paper ID:165
Title:Deep ReLU Networks Have Surprisingly Few Activation Patterns

This paper provides evidence that the number of possible activations is quite small, in stylized settings (it seems the paper is based mainly on remark 1). The paper is interesting and appears to be the first most concrete evidence of this, though to some extent this is implied by ntk analyses; I ask that the authors provide a concrete technical comparison to ntk and related papers/consequences. I also ask for a more thorough comparison to VC dimension results, which also proceed via region (and activation) counting.