The paper presents a general-purpose control algorithm combining planning and RL to solve tasks with sparse rewards or with long horizon. This algorithm is novel and interesting. The three reviewers agree that the contributions presented here should be published at the conference. The rebuttal helped solving most clarification issues. The reviewers also suggest various ways to further improve the manuscript. These include: - A more detailed discussion on the types of tasks the method could efficiently solve. - A discussion on how the replay buffer could be designed and optimized. - A more precise description of the algorithm and of the experiments.