
Submitted by Assigned_Reviewer_5
Q1: Comments to author(s). First provide a summary of the paper, and then address the following criteria: Quality, clarity, originality and significance. (For detailed reviewing guidelines, see http://nips.cc/PaperInformation/ReviewerInstructions)
Summary:
The paper introduces an iterative extension of NADE (Neural autoregressive distribution estimator), a generative model that uses a neural network with a variable number of inputs to model each conditional in an autoregressive factorization of a joint distribution. The paper builds up on top of an orderagnostic version of NADE where all dimensions not present in the input are modelled independently by the network at each autoregressive step.
The main idea introduced in the paper is using a prediction of the missing inputs at each iteration, starting with the marginal probability distribution over the training data, and the factorial (each dimension is predicted independently conditioned on the input) approximation obtained from NADE in the following iterations.
The authors hypothesise that prediction in several steps is easier than in one step. A claim they support with experimental results.
NADEk can be seen as a deep NADE with tied parameters across layers, it would be interesting to see a comparison with NADEmask that has the same effective number of layers and parameters (example a NADEmask with 5 layers and h=100 vs NADEk k=5 h=600).
In the conclusions, the authors suggest adjusting the confidence of intermediate predictions based on d. Shouldn’t NADE already able to do that by outputting values closer to 0.5 when there are few dimensions present in the input? Isn’t it in part the purpose of using masks? For NADEk, did the authors try using masks at every iteration or just during the first one?
In 3.2 the authors report which regularization methods they tried, but not which worked best, was it the L2 values reported in Table 3?
The complexity of evaluating densities and sampling should be reported. Will the increase in computation be linear with k when using several hidden layers O(DH^2) > O(kDH^2), but worse when using one hidden layer O(DH) > O(kDH^2)? This con of the technique should be explicitly reported.
Quality: The paper is technically sound and the usefulness of the ideas presented is supported by thorough experimental results that obtain stateoftheart modelling performance on two standard datasets.
Clarity: The paper is well written.
Originality: This paper combines ideas (NADE and iterative inference steps) in a novel and pragmatic way. The paper includes a good “related work” section where they ideas they build on are explained and referenced .
Significance: The paper presents a tractable distribution estimator that obtains stateoftheart results. The ideas presented will be of interest to the NIPS community, especially for the unsupervised learning community.
Q2: Please summarize your review in 12 sentences
The paper presents an iterative extension of the orderagnostic NADE that obtains significantly better results at a linear increase in computation time. It is an interesting paper with a good explanation of a combination of ideas that work in synergy. The experimental results presented are convincing and show this model obtains stateoftheart modelling performance.
Submitted by Assigned_Reviewer_33
Q1: Comments to author(s). First provide a summary of the paper, and then address the following criteria: Quality, clarity, originality and significance. (For detailed reviewing guidelines, see http://nips.cc/PaperInformation/ReviewerInstructions)
The paper proposes a simple extension to the inference scheme of the neural autoregressive density estimator (NADE) model.
The key idea is the following: In a NADE model, the conditional probability p(miss  observed) can be viewed as a feedforward neural network with a single hidden layer, and tied weights going in and out of the hidden layer. The authors propose to define this conditional probability using a multilayer feedforward neural net with nk layers (n weights that are used k times). Everything else, including using an unbiased stochastic estimator of the negative loglikelihood that uses all possible orderings, is borrowed from Uria et.al.
The paper is wellwritten. But my main concern is that there is very little novelty in this work.
The authors also conduct experiments on 2 somewhat toyish datasets, MNIST and Caltech101 silhouettes. The main strength of this paper is that the authors were able to show that on both datasets, their NADEk model outperforms NADE model that uses 1step of inference.
Q2: Please summarize your review in 12 sentences
In general, this is a wellwritten paper. However, given that the authors propose a rather simple extension to the existing model, I would have liked to see a much stronger experimental results on more realistic datasets. Submitted by Assigned_Reviewer_42
Q1: Comments to author(s). First provide a summary of the paper, and then address the following criteria: Quality, clarity, originality and significance. (For detailed reviewing guidelines, see http://nips.cc/PaperInformation/ReviewerInstructions)
The paper describes a new variant of (order agnostic, deep) NADE called NADEk that performs k successive reconstructions/inference steps when imputing the unobserved components of the input vector. To train NADEk, one must sample a subset of the input variables to impute while observing the remaining input variables. The paper includes experiments showing improved log probs over other versions of NADE on binarized MNIST digit generation and Caltech101 silhouette generation. There are also experiments exploring different values of k and the variance of the imputations with respect to the choice of variable order.
The NADEk algorithm presented in the paper seems sensible and is for the most part explained clearly. The experiments included are reasonable ones. The NADEk algorithm is an incremental improvement of deep, orderagnostic NADE and a relatively straightforward one. However, it is more expensive at both training time and test time (since test k is best set to equal training k). Any algorithm based on unrolling inference steps and backpropagating through them can have a k step version of itself created. The question is really how interesting a multistep variant is and whether it is worth the computational expense. If CD1 had been introduced without reference to the infinite steps of maximum likelihood and separately from CDk, would a paper on extending CD1 to CDk have been very novel?
One possible experiment not included in the paper that would have been interesting to see is whether slowly increasing k during training works well. In RBM training, to get very good generative performance quickly, it is helpful to switch from CD1 to CDk by stepping up k during training. At the start of training, the model is so inaccurate that CD1 can make rapid progress, but as training continues, more CD steps can help. Would this idea benefit NADEk?
Somewhat more practical applications of high quality density models would have been welcome to motivate the paper more strongly, perhaps in compression or denoising. Learning a density over binarized MNIST pixels is not a very compelling task. Since the NADE family of algorithms provide tractable, normalized, highquality density models, an application where those features are essential would have been more interesting.
The paper was mostly clear. There are a few sentencelevel issues, such as line 199 "allow to compute" and footnote 2 "for helping better converge." I had to slow down a lot while reading when I came upon equation 10, it would be clearer if there was a bit more english text introducing it. In general, the paper is well organized. Q2: Please summarize your review in 12 sentences
The paper presents an incremental extension to orderagnostic NADE. The experiments convincingly demonstrates its advantages in terms of final logprob. Q1:Author rebuttal: Please respond to any concerns raised in the reviews. There are no constraints on how you want to argue your case, except for the fact that your text should be limited to a maximum of 6000 characters. Note however, that reviewers and area chairs are busy and may not read long vague rebuttals. It is in your own interest to be concise and to the point. Dear reviewers,
We would like to thank you for your thorough and insightful comments. Please, see below for our answers to your comments.
= About the novelty and contribution (R33&R42) =
We agree that one may argue that our work has very little novelty against (Uria et al., 2014) but also against (Goodfellow et al., 2013). We acknowledge that we borrowed what we could from both of them and changed only what we had to, keeping things as simple as possible. However, we would like to emphasize that the proposed NADEk is not a simple adhoc combination of two methods, rather, it is well motivated from the principles (such as variational inference and denoising autoencoders) we discuss in the paper.
Specifically about CD1 vs. CDk raised by the Reviewer 42, we would like to point out that an idea very close to CD1 was already described in 1988 (see Fig. 2 of (Hinton&McClelland, 1988)). However, it was only CDk that led to the deep learning revolution.
Hinton, Geoffrey E., and James L. McClelland. "Learning representations by recirculation." NIPS, 1988.
= R33 =
"The authors propose to define this conditional probability using a multilayer feedforward neural net with nk layers. Everything else .. is borrowed from Uria et.al."
 Please, see our answer above.
= R42 =
"If CD1 had been introduced without reference to the infinite steps of maximum likelihood and separately from CDk, would a paper on extending CD1 to CDk have been very novel?"
 Please, see our answer at the beginning of this response letter.
"whether slowly increasing k during training works well."
 Thanks for your suggestion. We agree that slowly increasing k during training should work well. Our pretraining scheme Eq. (1011) seemed to work well enough, too, and it was easier to implement in practice for technical reasons (not having to change the model structure during training).
"There are a few sentencelevel issues"
 Thanks for pointing them out. We will fix them in the final version.
"equation 10"
 We will describe it more clearly in the final version.
= R5 =
"NADEk can be seen as a deep NADE with tied parameters across layers, it would be interesting to see a comparison with NADEmask that has the same effective number of layers and parameters (example a NADEmask with 5 layers and h=100 vs NADEk k=5 h=600)."
 Uria et al. (2014) reported that using 3 or 4 hidden layers was worse than using only 2 hidden layers. We do not expect the NADEmask 5 hidden layers to work well.
"the authors suggest adjusting the confidence of intermediate predictions based on d"
 Yes, the mask input could do that. We tried giving the mask only to the first step (out of k) but it made the learning progress slower. Flipping the maskrepresentation bits (1 for missing or observed) changed the performance, which indirectly suggests that it was probably an optimization issue. We think not using the mask as an input makes the model more elegant, requires fewer parameters, and makes the whole inference iteration of NADEk closer to, e.g., variational meanfield fixedpoint iteration.
"was it the L2 values reported in Table 3?"
 Yes, the best regularizations are the one reported in Table 3. We apologize for not mentioning that in the text, and we will make it clearer in the final version.
"The complexity of evaluating densities and sampling should be reported."
 The complexities should be O(D^2 H)>O(k D^2 H) for one hidden layer and O(D^2 H + D H^2) > O(k D^2 H + k D H^2) for two hidden layers. There is nothing specifically worse for the one hidden layer case.
 