The reviewers agreed that this paper addresses an important notion that should be disseminated widely in the ML community working on causal learning. While some reviewers were concerned that sample size issues may lie at the root of some of the findings of the paper, most found that the papers' contribution is more foundational: is asks what types of questions and metrics should even be used when evaluating causal inference methods. Beyond the wide survey of existing practice, the proposal for interventional measures and the novel type of benchmark dataset proposed would be interesting and useful to the community.