The reviewers initially had concerns, especially related to feature representations. lack of details about the human evaluation, and the justification for the end to end approach. However, the reviewers agreed that the author response was well written and addressed any major concerns about the paper. There was still a sentiment that integration of the ideas could be stronger, and that more complex, realistic environments would improve the paper, but that it was strong enough to be accepted as-is (though the authors are encouraged to take the advice of the reviewers for the camera ready).