After discussing, the reviewers agree that this paper makes a good contribution to the field. The main concerns are about improving clarity and presenting more intuition for the results, both of which should be done in the revised version. The reviewers would also like to see source code released so that proper comparisons can be done by future researchers. Additionally, if you haven't already I encourage you to take a look at Belousov and Peter (2018) on f-divergence constrained policy improvement, and clarify its relationship to your work.