The paper proposes a simple and novel shallow two stage RNN (SRNN) architecture which enjoys computational advantages over other RNNs-type models. The authors provide a theoretical justification under weak assumptions that are verified on real-world benchmarks. The reviewers think that the contribution is interesting and could have high impact. The also think that paper is well written and motivated, contains an interesting theoretical analysis and satisfactory empirical analysis. Concerns around clarity and experimentation were successfully addressed during the rebuttal.