This paper proposes a synchronous training scheme for reinforcement learning which address issues with existing synchronous methods (low throughput) and existing asynchronous methods (unstable, non-reproducible, etc.). The reviewers viewed this more of an engineering paper, but the design, execution, and experiments are solid, so we are recommending acceptance. I saw that the paper mentions that code will be released, but I want to emphasize the importance of this, as a large part of the value here is in enabling others to build on and use the proposed method.