After discussion, the reviewers and I agree that this paper is an interesting and novel extension of DDCs to nonlinear filtering and smoothing problems. The reviewers raised a number of areas for improvement -- it is important that these be addressed prior to publication. To highlight two, there were key places where clarity should be improved (esp. surrounding Figure 1 and the core inference/learning algorithms), and the framing relative to the long line of work on neurally plausible filtering and smoothing mechanisms should be made clear. Ideally, previous works would be included in the experimental comparisons.