This paper proposes techniques to deal with the softmax bottleneck problem. Pros • Experimental results show strong performances in language modeling and machine translation. • The paper is clearly written. • The proposed techniques are technically sound and novel. Cons • Writing of the paper can be further enhanced by making it self-contained. • More experiments can be conducted. The paper represents solid work. There are clarity issues pointed out by the reviewers. The authors have promised to provide more details and experiments in their final version. They also promise to release the codes.