Inspired by a PAC-Bayes risk bound for Deep Neural Nets (DNNs) with a Gaussian posterior having a covariance matrix determined by the correlation between the weight vectors within the same layer, the authors propose a weight correlation descent algorithm for regularizing DNNs. The extensive numerical experiments provide a clear evidence of the advantage of reducing the correlation between the weight vectors within the same layer. We think that this regularizer, easy to implement, can provide an alternative (or be complementary) to other currently-used regularizers such as weight decay and drop-out.