The paper has been actively discussed after the rebuttal that the reviewers found useful and actionable (e.g., reply about the scalability with respect to larger population sizes). The paper is recommended for acceptance. All reviewers have acknowledged that the paper is making a step towards model-based online hyperparameter optimization and it is likely to inform future research. One suggestion of the reviewers is to possibly move the supervised learning evaluation to the appendix.