Second Order Approximations for Probability Models

Part of Advances in Neural Information Processing Systems 13 (NIPS 2000)

Bibtex Metadata Paper

Authors

Hilbert Kappen, Wim Wiegerinck

Abstract

In this paper, we derive a second order mean field theory for directed graphical probability models. By using an information theoretic argu(cid:173) ment it is shown how this can be done in the absense of a partition function. This method is a direct generalisation of the well-known TAP approximation for Boltzmann Machines. In a numerical example, it is shown that the method greatly improves the first order mean field ap(cid:173) proximation. For a restricted class of graphical models, so-called single overlap graphs, the second order method has comparable complexity to the first order method. For sigmoid belief networks, the method is shown to be particularly fast and effective.