{"title": "Variational Bayesian Decision-making for Continuous Utilities", "book": "Advances in Neural Information Processing Systems", "page_first": 6395, "page_last": 6405, "abstract": "Bayesian decision theory outlines a rigorous framework for making optimal decisions based on maximizing expected utility over a model posterior. However, practitioners often do not have access to the full posterior and resort to approximate inference strategies. In such cases, taking the eventual decision-making task into account while performing the inference allows for calibrating the posterior approximation to maximize the utility. We present an automatic pipeline that co-opts continuous utilities into variational inference algorithms to account for decision-making. We provide practical strategies for approximating and maximizing the gain, and empirically demonstrate consistent improvement when calibrating approximations for specific utilities.", "full_text": "Variational Bayesian Decision-making\n\nfor Continuous Utilities\n\nTomasz Ku\u00b4smierczyk\n\nJoseph Sakaya Arto Klami\n\nHelsinki Institute for Information Technology HIIT\n\nDepartment of Computer Science, University of Helsinki\n\n{tomasz.kusmierczyk,joseph.sakaya,arto.klami}@helsinki.fi\n\nAbstract\n\nBayesian decision theory outlines a rigorous framework for making optimal de-\ncisions based on maximizing expected utility over a model posterior. However,\npractitioners often do not have access to the full posterior and resort to approxi-\nmate inference strategies. In such cases, taking the eventual decision-making task\ninto account while performing the inference allows for calibrating the posterior\napproximation to maximize the utility. We present an automatic pipeline that\nco-opts continuous utilities into variational inference algorithms to account for\ndecision-making. We provide practical strategies for approximating and maximiz-\ning the gain, and empirically demonstrate consistent improvement when calibrating\napproximations for speci\ufb01c utilities.\n\n1\n\nIntroduction\n\nA considerable proportion of research on Bayesian machine learning concerns itself with the task\nof inference, developing techniques for an ef\ufb01cient and accurate approximation of the posterior\ndistribution p(\u03b8|D) of the model parameters \u03b8 conditional on observed data D. However, in most\ncases, this is not the end goal in itself. Instead, we eventually want to solve a decision problem of\nsome kind and merely use the posterior as a summary of the information provided by the data and the\nmodeling assumptions. For example, we may want to decide to automatically shut down a process to\navoid costs associated with its potential failure, and do not care about the exact posterior as long as\nwe can make good decisions that still account for our uncertainty of the parameters.\nFocusing on inference is justi\ufb01ed by Bayesian decision theory [2] formalizing the notion that the\nposterior is suf\ufb01cient for making optimal decisions. This is achieved by selecting decisions that\nmaximize the expected utility, computed by integrating over the posterior. The theory, however, only\napplies when integrating over the true posterior which can be computed only for simple models. With\napproximate posteriors it is no longer optimal to separate inference from decision-making. Standard\napproximation algorithms try to represent the full posterior accurately, yet lack guarantees for high\naccuracy for parameter regions that are critical for decision-making. This holds for both distributional\ntechniques, such as variational approximation [4] and expectation propagation [10, 18], as well as\nMarkov chain Monte Carlo (MCMC) \u2013 even though the latter are asymptotically exact, a \ufb01nite set of\nsamples is still an approximation and it is often dif\ufb01cult to sample from the correct distribution.\nLoss-calibrated inference refers to techniques that adapt the inference process to better capture the\nposterior regions relevant to the decision-making task. First proposed by Lacoste-Julien et al. [16] in\nthe context of variational approximation, the principle has been used also for calibrating MCMC [1],\nand recently for Bayesian neural networks [7]. The core idea of loss calibration is to maximize the\nexpected utility computed over the approximating distribution, instead of maximizing the approxi-\nmation accuracy, while still retaining a reasonable posterior approximation. Figure 1 demonstrates\nhow the calibration process shifts an otherwise sub-optimal approximation to improve the decisions,\n\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.\n\n\fFigure 1: Loss-calibration (red) modi\ufb01es the posterior approximation (left) so that Bayes optimal\ndecisions for the predictive distribution (right) are better in terms of a user-de\ufb01ned loss, here squared\nerror, while still characterizing the posterior almost as well as the standard variational approximation\n(blue). See Section 6.2 for detailed description of the experiment.\n\nbut still represents the uncertainty over the parameter space. That is, we are merely \ufb01ne-tuning \u2013\ncalibrating \u2013 the approximation instead of solely optimizing for the decision.\nPrevious work on calibrating variational approximations only deals with classi\ufb01cation problems [7,\n16] making discrete decision amongst \ufb01nitely many classes. This allows algorithms based on explicit\nenumeration and summation of alternative decisions, which are inapplicable for continuous spaces.\nLack of tools for continuous utilities has thus far ruled out, for example, calibration for regression\nproblems. We provide these tools. We analyse the degree of calibration under linear transformations\nof utility, and describe how ef\ufb01cient calibration can be carried out also when the user characterises the\nrelative quality of the decisions with losses instead of utilities. To cope with the challenges imposed\nby moving from discrete to continuous output spaces, we replace the enumeration over possible\nchoices by nested Monte Carlo integration combined with double reparameterization technique, and\nprovide algorithms for learning optimal decisions for a \ufb02exible choice of utilities. We demonstrate the\ntechnique in predictive machine learning tasks on the eight schools model [9, 29] and probabilistic\nmatrix factorization on media consumption data.\n\n2 Background\n\n2.1 Bayesian decision theory\n\nBayesian decision theory [2, 24] is the axiomatic formalization of decision-making under uncertainty.\nGiven a posterior distribution p(\u03b8|D) of a parametric model conditioned on data D, we desire to\nmake optimal decisions h. The value of individual decisions depends on the utility \u02dcu(\u03b8, h) \u2265 0 that\nis a function of both: (1) the state of the world \u03b8 and (2) the decision h. The optimal decisions hp\nmaximize the gain (=the expected utility)\n\n(cid:90)\n\nGu(h) =\n\np(\u03b8|D)\u02dcu(\u03b8, h)d\u03b8.\n\nAn equivalent formulation is obtained by evaluating individual decisions by a loss function \u02dc(cid:96)(\u03b8, h)\n\nand solving for optimal decisions by minimizing the risk Rl(h) =(cid:82) p(\u03b8|D)\u02dc(cid:96)(\u03b8, h)d\u03b8.\nu(y, h), which together with the model induces the utility \u02dcu(\u03b8, h) =(cid:82) p(y|\u03b8,D)u(y, h)dy, where\n\nEven though some decision problems operate directly on model parameters \u03b8, it is more typical to\nmake decisions regarding predictions y \u223c p(y|D). For such problems the utility is expressed as\nwe use the notation p(y|\u03b8,D) to indicate the prediction may depend on some covariates in D. This\ncomplicates computation because evaluating the gain requires nested integration over p(\u03b8|D) and\np(y|\u03b8,D). In the remainder of the paper we focus on this more challenging family of decisions, and\nalways use \u02dcu(\u03b8, h) to denote the expected utility induced by the predictive utility u(y, h).\n\n2\n\nqW[i=577,k=1]qW[i=577,k=2]qW[i=577,k=3]qW[i=577,k=4]qZ[k=1,j=Muse]qZ[k=2,j=Muse]qZ[k=3,j=Muse]qZ[k=4,j=Muse]ApproximatePosteriorDensitiesMarginalize out latent variables 2468Outputy0.00.10.20.3PosteriorPredictiveDensityLCVIpredictionhLCVIVIpredictionhVItrueuserno:577artist:MuseLCVIVI\f2.2 Variational inference\nVariational inference approximates the posterior p(\u03b8|D) with a proxy distribution q\u03bb(\u03b8) parameterized\nby \u03bb, typically by maximizing a lower bound LVI(\u03bb) for the marginal log-likelihood\n\n(cid:90)\n\nlog p(D) = log\n\nq\u03bb(\u03b8)\n\np(D, \u03b8)\nq\u03bb(\u03b8)\n\nd\u03b8 \u2265\n\nq\u03bb(\u03b8) log\n\np(D, \u03b8)\nq\u03bb(\u03b8)\n\nd\u03b8 =: LVI(\u03bb).\n\n(cid:90)\n\nTraditional methods use coordinate ascent updates, mean-\ufb01eld approximations, and conjugate priors\nfor computational tractability [4]. Recently, several gradient-based optimization algorithms [5, 22, 28]\nhave made variational inference feasible for non-conjugate models and richer approximation families,\nusing gradient-based optimization of Monte Carlo estimates of the bound.\nThe most ef\ufb01cient techniques use reparameterization to compute the gradients \u2207\u03bbLVI(\u03bb) =\n\u2207\u03bbEq\u03bb(\u03b8)[log p(D, \u03b8) \u2212 log q\u03bb(\u03b8)], by rewriting the distribution q\u03bb(\u03b8) using a differentiable trans-\nformation \u03b8 = f (\u0001, \u03bb) of an underlying, parameter-free standard distribution q0(\u0001) [28]. We can then\nuse Monte Carlo integration over q0(\u0001) for evaluating the expectations, yet the value depends on \u03b8 and\nhence we can propagate gradients through f (\u00b7) for learning. The reparameterization can be carried\nout either explicitly [20, 25] or implicitly [8]; the latter strategy makes reparameterization possible for\nalmost any distribution. Our derivations and experiments are on simple parametric approximations,\nbut we note that the loss calibration elements can be combined with wide range of recent advances\nin variational inference, such as generalized VI [14], boosting VI [11, 17], more ef\ufb01cient structural\napproximations [12], and normalizing \ufb02ows for \ufb02exible approximations [23].\n\n2.3 Loss-calibrated variational inference\n\nThe idea of calibrating a variational approximation was proposed by Lacoste-Julien et al. [16], based\non lower bounding the logarithmic gain using Jensen\u2019s inequality as\np(\u03b8|D)\u02dcu(\u03b8, h) d\u03b8 \u2265 \u2212KL(q, p) +\n\nlog Gu(h) = log\n\nq\u03bb(\u03b8) log \u02dcu(\u03b8, h)d\u03b8\n\nq\u03bb(\u03b8)\n\n.\n\n(cid:90) q\u03bb(\u03b8)\n\n(cid:123)(cid:122)\n\n(cid:125)\n\nU(\u03bb,h) - utility-dependent term\n\n(cid:90)\n(cid:124)\n\nThe bound consists of two terms. The \ufb01rst term, negative Kullback-Leibler divergence between\nthe approximation and the posterior, can be further replaced by a lower bound for the marginal\nlikelihood [4] analogous to standard variational approximation to provide the \ufb01nal bound L(\u03bb, h) :=\nELBO(\u03bb) + U(\u03bb, h) \u2264 log Gu(h). The second term accounts for decision making. It is independent\nof the observed y and only depends on the current approximation q\u03bb(\u03b8), favoring approximations\nthat optimize the utility. For ef\ufb01cient optimization the bound for multiple predictions can be written\nas sum over individual data instances as\n\nL(\u03bb,{h}) =\n\n(ELBOi(\u03bb) + U(\u03bb, hi)) \u2264 log Gu({h}),\n\n(1)\n\n(cid:88)\n\ni\u2208[D]\n\nwhere ELBOi accounts for an individual data point yi for which the hypothesis is hi. This holds for\npredictive models that assume i.i.d. predictions and additive log-gains, which captures most practical\nscenarios and allows for making decisions hi separately for individual data points. For clarity, we\ndrop the subscript i in hi during derivations.\nFor optimizing the bound, Lacoste-Julien et al. [16] derived an EM algorithm that alternates between\nlearning optimal \u03bb and selecting optimal decisions:\nU(\u03bb,{h}).\nE-step: \u03bb := arg max\nHowever, they used closed-form analytic updates for \u03bb, for which incorporating the utility-dependent\nterm is dif\ufb01cult, and only demonstrated the principle in classi\ufb01cation problems with discrete h.\nCobb et al. [7] derived a loss-calibrated variational lower bound for Bayesian neural networks with\ndiscrete decisions h. Unlike Lacoste-Julien et al. [16], however, their updates for \u03bb are gradient-based\nand applicable to generic utility-dependent terms as long as the decisions are discrete.\n\n{h} L(\u03bb,{h}) = arg max\n\nM-step: {h} := arg max\n\nL(\u03bb,{h}),\n\n{h}\n\n\u03bb\n\n3 Loss calibration for continuous utilities\n\nHandling continuous decisions requires both more careful treatment of utilities and losses, described\nbelow, and new algorithms for optimizing the bound, provided in Section 4.\n\n3\n\n\f3.1 The calibration effect\nOptimal decisions are invariant to linear transformations of the utility, so that arg maxh Gu(h) =\narg maxh Gu(cid:48)(h) for u(cid:48)(y, h) = \u03b1 \u00b7 u(y, h) + \u03b2 for \u03b1 > 0 [2]. However, this does not hold for the\nloss-calibration procedure. Instead, translating the utilities with \u03b2 in\ufb02uences the degree of calibration.\nTo see this we assume (for notational simplicity) inf y,h u(y, h) = 0 and write U corresponding to\n(cid:20)\nu(cid:48)(y, h) as\nEq\n\n(cid:19)(cid:21)\n\n= Eq\n\n+ log \u03b2,\n\n(cid:18)\n\n(cid:18)\n\n\u03b2 + \u03b1\n\n(cid:90)\n\nlog\n\n1 +\n\n(cid:90)\n\n(cid:20)\n\nlog\n\np(y|\u03b8,D)u(y, h)dy\n\np(y|\u03b8,D)u(y, h)dy\n\n(cid:124)\n\n\u03b1\n\u03b2\n\n(cid:123)(cid:122)\n\nexpectation term\n\n(cid:19)(cid:21)\n(cid:125)\n\nwhere the equality holds for any \u03b2 > 0 (negative values would lead to negative utilities). Since\nlog \u03b2 is constant w.r.t variational parameters \u03bb, only the expectation term is relevant for optimization.\nHowever, its impact (=magnitude) relative to ELBO depends on the ratio \u03b1\n\u03b2 . In particular, as \u03b2 \u2192 \u221e\n(for any \ufb01xed \u03b1) the expectation term converges to 0 removing the calibration effect completely. In\ncontrast, pushing \u03b2 \u2192 0 maximizes the effect of calibration by maximizing the magnitude of U.\nHence, maximal calibration is achieved when \u03b2 = 0, i.e., when inf y,h u(cid:48)(y, h) = 0. Finally, for\n\u03b2 = 0 the scaling constant \u03b1 can be taken out of the expectation and hence has no effect on optimal\nsolution.\nIn summary, the calibration effect is maximized by using utilities with zero in\ufb01mum, and for such\nutilities the procedure is scale invariant. The procedure is valid also when this does not hold, but the\ncalibration effect diminishes and depends on the scaling in an unknown manner.\n\n3.2 Utilities and losses\n\nAs stated in Section 2.1, decision problems can be formulated in terms of maximizing gain de\ufb01ned\nby a utility u(y, h) \u2265 0, or in terms of minimizing risk de\ufb01ned by a loss (cid:96)(y, h) \u2265 0. The calibration\nprocedure above is provided for the gain, since both the bound for the marginal likelihood and the\nbound for the gain need to be in the same direction in Eq. (1). To calibrate for user-de\ufb01ned loss\n(which tends to be more common in practical applications), we need to convert the loss into a utility.\nUnfortunately, the way this is carried out in\ufb02uences the \ufb01nal risk evaluated using the original loss.\nOnly linear transformations retain the optimal decisions, and the simplest one providing non-negative\nutilities is u(y, h) = M \u2212 (cid:96)(y, h) where M \u2265 supy,h (cid:96)(y, h) [2], with equality providing optimal\ncalibration as explained above. However, we cannot evaluate M = supy,h (cid:96)(y, h) for continuous\nunbounded losses. Furthermore, this value may be excessively large due to outliers, so that M (cid:29)\n(cid:96)(y, h) for almost all instances, effectively removing the calibration even if we knew how to \ufb01nd\nthe optimal value. As a remedy, we propose two practical strategies to calibrate for continuous\nunbounded losses, both based on the intuitive idea of bringing the utilities close to zero for the losses\nwe expect to see in practice.\nRobust maximum For u(y, h) = M \u2212 (cid:96)(y, h), any value of (cid:96)(y, h) > M may lead to \u02dcu(\u03b8, h) < 0\nand hence to negative input to log in U. However, this problem disappears if we linearize the\nlogarithm in U around M similar to [16]. Using Taylor\u2019s expansion\n\u02dc(cid:96)(\u03b8, h)\n\n(cid:32) \u02dc(cid:96)(\u03b8, h)\n\n2\n\nlog \u02dcu(\u03b8, h) = log(M \u2212 \u02dc(cid:96)(\u03b8, h)) = log M \u2212\n\nM\n\n+ O\n\nM 2\n\n(cid:104)\n\n(cid:105)\n\n\u02dc(cid:96)(\u03b8,h)\nand dropping the error term,\nM and the utility-dependent term\nEq[log \u02dcu(\u03b8, h)] can be re-expressed as Eq\n= log M \u2212 1\n. We can\nignore log M as it is constant with respect to the decisions h and variational parameters \u03bb. Now that\n\u02dcu(\u03b8, h) no longer appears inside a log, we can use also M that is not a strict upper bound, but instead\na robust estimator for maximum that excludes the tail of the loss distribution. We propose\n\nlog \u02dcu(\u03b8, h) \u2248 log M \u2212\n\u02dc(cid:96)(\u03b8,h)\n\nlog M \u2212\n\nEq\n\nM\n\nM\n\n(2)\nwhere Mq is the qth quantile (e.g. 90%) of the expected loss distribution. To obtain Mq we \ufb01rst run\nstandard VI for suf\ufb01ciently many iterations (until the losses converge), and then compute the loss for\n\nu(y, h) = Mq \u2212 (cid:96)(y, h),\n\n4\n\n,\n\n(cid:33)\n(cid:104)\u02dc(cid:96)(\u03b8, h)\n(cid:105)\n\n\fevery training instance. We then sort the resulting losses and set Mq to match the desired quantile of\nthis empirical distribution of losses. In many cases, Mq is considerably smaller than supy,h l(y, h),\nwhich increases the calibration effect.\n\nNon-linear loss transformation The other alternative is to use transformations that guarantee\nnon-negative utilities by mapping losses into positive values. A practical example is\n\n(3)\nwhere the rate parameter \u03b3 can be related to the quantiles of the loss distribution as \u03b3 = M\u22121\n, by\nsolving for a value for which linearization of the utility at (cid:96)(y, h) = 0 would be zero for (cid:96)(y, h) = Mq.\n\nu(y, h) = e\n\nq\n\n\u2212\u03b3(cid:96)(y,h),\n\n4 Algorithms for calibrating variational inference\n\nFor practical computation we need to optimize Eq. (1) w.r.t. both \u03bb and {h} in a model-independent\nmanner, which can be carried out using techniques described next.\n\n4.1 Monte Carlo approximation of U\n\nThe \ufb01rst practical challenge concerns evaluation and optimization of U =(cid:82) q\u03bb(\u03b8) log \u02dcu(\u03b8, h)d\u03b8 =\n(cid:82) q\u03bb(\u03b8) log(cid:82) p(y|\u03b8,D)u(y, h)dyd\u03b8. Since we already reparameterized \u03b8 for optimization of ELBO,\n\nwe can as well approximate the outer expectation as\n\nU(\u03bb, h) \u2248\n\n1\nS\u03b8\n\nlog \u02dcu(\u03b8, h) =\n\n1\nS\u03b8\n\nlog \u02dcu(f (\u0001, \u03bb), h),\n\n(4)\n\n(cid:88)\n\n\u03b8\u223cq\u03bb(\u03b8)\n\n(cid:88)\n\n\u0001\u223cq0\n\nwhere q0 is the zero-parameter distribution, f transforms samples from q0(\u0001) into samples from q\u03bb(\u03b8),\nand S\u03b8 is a number of samples for Monte Carlo integration. For discrete outputs the inner expectation\ncomputing \u02dcu(\u03b8, h) becomes a sum over possible values Y, which makes U straightforward to optimize\nboth w.r.t. \u03bb (via gradient ascent) and h (by enumeration). For continuous y, however, the integral\nremains as the main challenge in developing ef\ufb01cient algorithms.\nWe address this challenge by a double reparametrization scheme. Besides reparameterizing the\napproximation q\u03bb(\u03b8), we reparameterize also the predictive likelihood p(y|\u03b8,D), what was made\npossible for most densities by implicit reparameterization gradients [8]. This enables approximating\nthe inner integral with MC samples as\n\n\u02dcu(\u03b8, h) \u2248\n\n1\nSy\n\nu(g(\u03b4, \u03b8,D), h),\n\n(5)\n\n(cid:88)\n\n\u03b4\u223cp0\n\nwhile preserving differentiability w.r.t. both \u03bb and h. Here \u03b4 denotes samples from parameter-free\ndistribution p0 used to simulate samples y \u223c p(y|f (\u0001, \u03bb),D) via the transformation g(\u00b7). Similar\nderivation for approximation of the utility-dependent term exists for discrete decisions [7]. This\nhowever, does not require the double reparameterization scheme proposed here.\nFor evaluating U we use a naive estimator that simply plugs (5) in place of \u02dcu(f (.), h) into (4):\n\n\uf8eb\uf8ed 1\n\n(cid:88)\n\n(cid:88)\n\n\u0001\u223cq0\n\nU(\u03bb, h) \u2248\n\n1\nS\u03b8\n\nlog\n\nSy\n\n\u03b4\u223cp0\n\nu(g(\u03b4, f (\u0001, \u03bb),D), h)\n\n\uf8f6\uf8f8 .\n\n(cid:82) q\u03bb(\u03b8)(cid:82) p(y|\u03b8,D) log u(y, h)dyd\u03b8, but such\nEven though this estimator is slightly biased, it works well in practice. The bias could be reduced, for\nexample, by employing the Taylor expansion of Ep[log u] in a manner similar to [27], or removed\nby bounding U with Jensen\u2019s inequality as U(\u03bb, h) \u2265\nestimators display numerical instability for utilities close to zero and are not useful in practice.\nFinally, we note that for linearized utility-dependent term, for problems de\ufb01ned in terms of losses,\n(cid:88)\n(cid:88)\nwe can directly use the simple unbiased estimator\n\n1\n\nU(\u03bb, h) \u2248 \u2212\n\nM S\u03b8Sy\n\n\u0001\u223cq0\n\n\u03b4\u223cp0\n\n(cid:96)(g(\u03b4, f (\u0001, \u03bb),D), h).\n\nWe note that, however useful in practice, the linearization of U may violate the bound in Eq. (1).\n\n5\n\n(6)\n\n(7)\n\n\fTable 1: Losses and their closed-form Bayes estimators that minimize their posterior expected value.\n\nLOSS\n\nSquared\nLinEx\nAbsolute\n\nTilted\n\n(cid:40)\n\nEXPRESSION\n(h \u2212 y)2\n\nec(h\u2212y) \u2212 c(h \u2212 y) \u2212 1\nq \u00b7 |h \u2212 y|\ny \u2265 h\n(1 \u2212 q) \u00b7 |h \u2212 y|\ny < h\n\n|h \u2212 y|\n\nBAYES ESTIMATOR\n\nc log(cid:82) e\u2212cyp(y)dy\n\nEp[y]\n\n\u2212 1\n\nmedianp[y]\n\nq-percentilep[y]\n\n4.2 Optimization\n\nGradient-based optimization w.r.t. \u03bb is easy with automatic differentiation, and hence we focus here\non the optimization w.r.t. h, \ufb01rst in M-step of EM algorithm and then jointly with \u03bb.\n\nClosed-form optimal decision If the utility is expressed in terms of a loss and we use the linearized\nestimator (7), the optimal h corresponds to the Bayes estimator for the loss and can often be computed\nin closed form as a statistic of the posterior predictive distribution p(y|D). Some examples are listed\nin Table 1. However, we typically do not have the predictive distribution in closed form, and hence,\nthe statistics are estimated by sampling from the predictive distribution.\n\nusing an objective aggregated over data (mini-)batch D: arg max{h}(cid:80)\n\nNumerical optimization When no closed-form solution for optimal decision is available, we need\nto numerically optimize a Monte Carlo estimate of U. One could consider parallel execution of\nmultiple one-dimensional solvers, but a more practical approach is to jointly optimize for all {h}\nU(\u03bb, hi) allowing for\nuse of standard optimization routines with ef\ufb01cient implementations. Gradient-based optimization\nw.r.t. {h} is made easier by the observation that the expectation \u02dcu(\u03b8, h) (also when approximated by\nsampling) tends to be relatively smooth even when the underlying utility u(y, h) is not [26].\n\ni\u2208[D]\n\nJoint optimization of decisions and approximation parameters For numerical optimization, we\nalternatively propose to think of {h} as additional parameters and jointly optimize for (1) w.r.t. both\n\u03bb and {h} using \u2207L = [\u2207\u03bbELBO, 0, . . . , 0]T + [\u2207\u03bbU, \u2202U\n\u2202h|D| ]T . This has the advantage of\nnot requiring full numerical optimization of {h} for every step, but comes with additional memory\noverhead of storing {h} for the whole data set even when using mini-batches for optimization.\n\n, ..., \u2202U\n\n\u2202h1\n\n5 Automatic VI for decision-making\n\nBuilding on the above ideas, we provide a practical procedure for calibrating variational approxima-\ntions for continuous utilities. A problem speci\ufb01cation consist of (a) a differentiable model p(y, \u03b8),\ntypically speci\ufb01ed using a probabilistic programming language, along with training data D, (b) a\nutility (or loss) expressing the quality of decisions, and (c) an approximating family q\u03bb(\u03b8).\nFor problems de\ufb01ned in terms of u(y, h), the utilities should be scaled so that inf \u03b8,y,h u(y, h) = 0, to\nachieve maximal calibration. For problems de\ufb01ned in terms of loss (cid:96)(y, h), we need to \ufb01rst transform\nthe loss into a utility. For unbounded losses this should be done with a transformation such that\nutilities corresponding to losses below a suitable upper quantile (typically between 50% and 95%) of\nthe expected loss distribution (or the empirical loss distribution of uncalibrated VI) remain positive.\nGiven the quantile, we can either linearize the utility and use (2) or use the exponential transformation\n(3), neither of which has no tuning parameters besides the quantile.\nTo maximize the bound (1) we use joint gradient-based optimization over \u03bb and {h}, reparameteriza-\ntion for \u03b8 and y, and (6) or (7) for evaluating U with suf\ufb01ciently many samples (e.g., S\u03b8Sy \u2248 300).\nFor problems speci\ufb01ed in terms of losses with known Bayes estimators (Table 1), one can also use\nEM algorithm where optimal decisions are determined by statistics of posterior predictive samples.\n\n6\n\n\fFigure 2: Eight-schools model calibrated for tilted loss (q = 0.2). LCVI consistently reduces the risk\n(left) while shifting the posterior approximation (middle) and compressing marginal densities (right).\n\n6 Experiments\n\nWe \ufb01rst demonstrate how the calibration affects the posterior approximation on a simple hierarchical\nmodel, and then highlight how calibration changes the decisions of a continue-valued probabilistic\nmatrix factorization model in an intuitive but hard-to-predict manner. Finally, we demonstrate the\neffect of utility transformations and technical properties of the optimization process. The code for\nreproducing all experiments (with additional \ufb01gures) is available online1.\nTo highlight the calibration effect, we compare loss-calibrated VI (LCVI) against standard reparame-\nterization VI. The evaluations were carried out for decision problems characterized by loss functions\nde\ufb01ned for model outputs, i.e., l(y, h) measured by empirical risk reduction on test data:\n\nI = ERVI \u2212 ERLCVI\n\nERVI\n\n,\n\nERALG =\n\n1\n\n|Dtest|\n\ni\u2208[Dtest]\n\n(cid:88)\n\n(cid:96)(yi, hALG\n\ni\n\n).\n\ni\n\nHere ERALG denotes empirical risk and hALG\nis the decision obtained for the ith point using\nALG \u2208 {VI, LCVI}, optimal w.r.t. loss (cid:96). For ERVI we always use the \ufb01nal risk for converged\nVI, and hence the value of I corresponds to practical improvement in quality of the \ufb01nal decision\nproblem. On convergence plots, we report its mean \u00b1 standard deviation estimated for 10 different\nrandom initializations, and on boxplots the boxes indicate 25th and 75th percentiles.\nWhenever not stated differently, we used joint optimization of {h} and \u03bb with Adam [13] (learning\nrate set to 0.01) ran until convergence (20k epochs for hierarchical model and 3k epochs for matrix\nfactorization with minibatches of 100 rows). For the \ufb01rst two experiments we set the quantile Mq at\n90%, to illustrate robust performance without tuning of hyper-parameters.\n\n6.1\n\nIllustration on hierarchical model\n\nThe eight schools model [9, 29] is a simple Bayesian hierarchical model often used to demonstrate\nmean-\ufb01eld approximations failing to \ufb01t the true posterior. Individual data points are noisy observations\n{(yj, \u03c3j)} of effects \u03b8j with shared prior parameterized by \u00b5 and \u03c4\n\nyj \u223c N (\u03b8j, \u03c32\nj ),\n\n\u03b8j \u223c N (\u00b5, \u03c4 2), \u00b5 \u223c N (0, 5),\n\n\u03c4 \u223c half-Cauchy(0, 5).\n\ni=1 q\u03bb\u03b8i\n\nq\u03bb\u00b5(\u00b5)q\u03bb\u03c4 (\u03c4 )(cid:81)8\n\nWe couple the model with the tilted loss (Table 1) with q = 0.2 to indicate a preference to\nnot overestimate treatments effects, and use the mean-\ufb01eld approximation q\u03bb(\u00b5, \u03c4, \u03b81 . . . \u03b88) =\n(\u03b8i), where each term is a normal distribution parameterized with mean and\nstandard deviation. We used linearized U with M matching the 90th percentile. Due to small size of\nthe data, empirical risks were calculated on the training data (Dtest = D).\nFigure 2 illustrates the calibration process by comparing LCVI with standard mean-\ufb01eld VI and\nHamiltonian Monte Carlo using Stan [5], which characterizes the true posterior well and hence\nprovides the ideal baseline. LCVI converges smoothly and provides stable (but small) 1% reduction\nin risk, validating the procedure. The middle sub-\ufb01gure illustrates the effect of calibration on the\nposterior. Standard VI fails to capture the dependencies between \u03c4 and \u03b8 and misses the true posterior\nmode. Even though LCVI also uses mean-\ufb01eld approximation, it here shifts the approximation\ntowards the region with more probability mass in true posterior. The right sub-\ufb01gure shows that\nbesides shifting the approximation the calibration process slightly reduces the marginal variances.\n\n1https://github.com/tkusmierczyk/lcvi\n\n7\n\n05000100001500020000Epoch\u22122\u221210123EmpiricalRiskReduction(I)[%]linearizedwithM900246810\u03c4\u221210\u2212505101520\u03b81PosteriorVILCVI\u00b5\u03c4\u03b81\u03b82\u03b83\u03b84\u03b85\u03b86\u03b87\u03b88ModelParamater\u2212505101520Distribution(mean\u00b1std.dev.)PosteriorVILCVI\fFigure 3: Matrix factorization with tilted and squared losses on the Last.fm data set. Loss-calibration\nclearly reduces the risk (left), while changing the decisions in a non-trivial manner (middle and right).\n\nFigure 4: Matrix factorization of Last.fm data: (Left:) Joint optimization of \u03bb and {h} outperforms\nEM when we need to use numerical optimization of {h}, but EM with Bayes estimators may be\noptimal when applicable. (Middle:) The parameter M controlling the transformation of losses into\nutilities relates naturally to the quantiles of the loss distribution; optimal calibration is here obtained\nwith 70% quantile for both transformations. (Right:) Comparison of different estimators to U(\u03bb, h)\nin a decision problem expressed in terms of utility.\n\n6.2 Matrix factorization and music consumption\n\nWe demonstrate LCVI in a prototypical matrix factorization task, modeling the Last.fm data set [3],\na count matrix C of how many times each user has listened to songs by each artist. We transform\nthe data with Y = log(1 + C) and restrict the analysis to the top 100 artists. We randomly split the\nmatrix entries into even-sized training and evaluation sets, and provide utilities for predictions.\nThe effect of loss calibration is best shown on straightforward models with no additional elements to\ncomplicate the analysis, and hence we use a simpli\ufb01ed probabilistic matrix factorization [19]\n\nY \u223c N (ZW, \u03c3y) Wik \u223c N (0, \u03c3W ) Zkj \u223c N (0, \u03c3z).\n\n(8)\n\nq(\u03b8) =(cid:81)\n\nk\n\n(cid:104)(cid:81)\n\n(wik)(cid:81)\n\n(cid:105)\n\nHere, Y is a 1000 \u00d7 100-dimensional data matrix, and Z and W are matrices of latent variables\nwith latent dimension K = 20. We set all \u03c3 terms to 10 and use the mean-\ufb01eld approximation\n\ni q\u03bbwik\n\nj q\u03bbzkj\n\n(zkj)\n\n, where each term is a normal distribution.\n\nFigure 3 (left) demonstrates the effect of calibrating for squared and titled loss (q = 0.2/0.5/0.8)\ntransformed to utilities by (3) with M90 quantile, showing that LCVI achieves risk reduction of\n1\u2212 4% for all choices. This holds already for the symmetric utilities (squared and tilted with q = 0.5)\nthat do not express any preference in favor of over- or underestimation, but simply specify the rate\nof decline of utilities. The middle sub-\ufb01gure shows the effect from the perspective of posterior\npredictive distributions for sample user-artist pairs. The 90%-intervals for all the cases overlap to a\nhigh extent, underlining the fact that LCVI calibrates the results of standard VI and usually does not\nresult in drastic changes, and that the decisions behave naturally as a function of q; penalizing more\nfor underestimation for larger values. The right sub-\ufb01gure further explores the effect by plotting the\nchange in optimal decision, presented as the the density of user-speci\ufb01c mean differences between\nLCVI and VI. The individual decisions can change to either direction, indicating that the calibration\nchanges the whole posterior and does not merely shift the decisions.\n\n8\n\nq=0.2q=0.5q=0.8squared012345EmpiricalRiskReduction(I)[%]tiltedloss49user791VItestpttrainingptq=.2/.5/.8squaredRadioheadEvanescenceArcadeFire27user290Posterior90%-CIandDecisions\u22120.10\u22120.050.000.050.10Per-userAverageDecisionShift0.02.55.07.510.012.5DensityVI=LCVIq=0.2q=0.5q=0.8squared05000100001500020000Time[s]0246EmpiricalRiskReduction(I)[%]jointoptimizationBayesestimatornum(10samples)num(100samples)10\u22121100101102M\u2212505EmpiricalRiskReduction(I)[%]3050709099percentileslin(Eq.2)exp(Eq.3)naive205080100\u22122.50.02.55.07.510.0EmpiricalRiskReduction(I)[%]linearized(percentile)\f6.3 Algorithm performance\n\nOptimization algorithm Figure 4 (left) compares the alternative optimization algorithms on the\nmatrix factorization problem with squared loss and M90 in (3). Here, the EM algorithm with Bayes\nestimator and joint optimization of \u03bb and {h} have comparable accuracy and computational cost.\nHowever, if we need to resort to numeric optimization of {h}, EM should not be used: It either\nbecomes clearly too slow (for S\u03b8Sy = 100) or remains inaccurate (for S\u03b8Sy = 10). Finally, we note\nthat standard VI is here roughly 10 times faster than LCVI; the calibration comes with increased but\nmanageable computational cost.\n\nCalibrating with losses Section 3.2 explains how losses l(y, h) need to be transfomed into utilties\nbefore calibration, and suggests two practical transformations expressed as functions M that can be\nrelated to quantiles of the empirical loss distribution of standard VI. Figure 4 (middle) plots the risk\nreduction for various choices of M to show that: (a) For large M we lose the calibration effect as\nexpected, (b) The optimal calibration is obtained with M < sup (cid:96)(y, h), and wide range of quantiles\nbetween M50 and M95 provide good calibration, and (c) both the linearized variant and exponential\ntransformation provide similar results. To sum up, the calibration process depends on the parameter\nM, but quantiles of the empirical distribution of losses provides a good basis for setting the value.\n\nEffect of linearization Finally, we demonstrate that linearization may have a detrimental effect\ncompared to directly calibrating for a utility, even for optimal M. We use u(y, h) = e\u2212(h\u2212y)2 and\ncompare (6) against (7) (for (cid:96)(y, h) = 1 \u2212 u(y, h)). Since U is clearly non-linear close to zero, we\nsee that direct calibration for utility clearly outperforms the linearized estimator for all M.\n\n7 Discussion\n\nTo facilitate use of Bayesian models in practical data modeling tasks, we need tools that better solve the\nreal goal of the user. While Bayesian decision theory formally separates the inference process from the\neventual decision-making, the unfortunate reality of needing to operate with approximate techniques\nnecessitates tools that integrate the two stages. This is of particular importance for distributional\napproximations that are typically less accurate than well carried out sampling inference [29], but\nhave advantage in terms of speed and may be easier to integrate into existing data pipelines.\nLoss-calibration [16] is a strong basis for achieving this, although it remains largely unexplored.\nThe previous work has been devoted to discrete decisions, and we expanded the scope by providing\npractical tools for continuous decisions that are considerably more challenging. We demonstrated\nconsistent improvement in expected utility with no complex tuning parameters, which would translate\nto improved value in real data analysis scenarios. We also demonstrated that for maximal improvement\nthe original decision problem should be expressed in terms of utilities, not losses, in order to avoid\ndetrimental approximations required for coping with decisions based on unbounded losses.\nOur work improves the decisions by altering the posterior approximation within a chosen distribution\nfamily, and is complementary to directly improving the approximation by richer approximation\nfamilies [11, 17, 23].\nIt also relates to the more general research on alternative objectives for\nvariational inference. The research is largely focused on improving the tightness of the bounds\n(e.g. [6]), but this is not necessarily optimal for all models and tasks [21]. In this context, we provided\na practical objective that improves the accuracy in terms of a speci\ufb01c decision task by making\nthe variational bound worse. Finally, recently an alternative approach for improving decisions for\napproximate posteriors was proposed, based on modifying the decision-making process itself instead\nof modifying the approximation [15]. The relative quality of these alternative strategies aiming at the\ncommon goal of improving decisions under approximate inference is worthy of further research.\n\nAcknowledgements\n\nThe work was supported by Academy of Finland (1266969, 1313125), as well as the Finnish Center\nfor Arti\ufb01cial Intelligence (FCAI), a Flagship of the Academy of Finland. We also thank the Finnish\nGrid and Cloud Infrastructure (urn:nbn:\ufb01:research-infras-2016072533) for computational resources.\n\n9\n\n\fReferences\n[1] Ehsan Abbasnejad, Justin Domke, and Scott Sanner. Loss-calibrated Monte Carlo Action\nSelection. In Proceedings of the Twenty-Ninth AAAI Conference on Arti\ufb01cial Intelligence, 2015.\n\n[2] James O Berger. Statistical Decision Theory and Bayesian Analysis; 2nd edition. Springer\n\nSeries in Statistics. Springer, New York, 1985.\n\n[3] Thierry Bertin-Mahieux, Daniel P.W. Ellis, Brian Whitman, and Paul Lamere. The Million Song\nDataset. In Proceedings of the 12th International Conference on Music Information Retrieval,\n2011.\n\n[4] David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational Inference: A Review for\n\nStatisticians. Journal of the American Statistical Association, 112(518), 2017.\n\n[5] Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael\nBetancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A Probabilistic\nProgramming Language. Journal of Statistical Software, 76(1), 2017.\n\n[6] Liqun Chen, Chenyang Tao, Ruiyi Zhang, Ricardo Henao, and Lawrence Carin Duke. Varia-\ntional inference and model selection with generalized evidence bounds. In Proceedings of the\n35th International Conference on Machine Learning, 2018.\n\n[7] Adam D Cobb, Stephen J Roberts, and Yarin Gal. Loss-Calibrated Approximate Inference in\n\nBayesian Neural Networks. In Theory of Deep Learning workshop, ICML, 2018.\n\n[8] Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit Reparameterization Gradients.\n\nAdvances in Neural Information Processing Systems 31, 2018.\n\n[9] Andrew Gelman, Hal S Stern, John B Carlin, David B Dunson, Aki Vehtari, and Donald B\n\nRubin. Bayesian Data Analysis. Chapman and Hall/CRC, 2013.\n\n[10] Andrew Gelman, Aki Vehtari, Pasi Jyl\u00e4nki, Tuomas Sivula, Dustin Tran, Swupnil Sahai, Paul\nBlomstedt, John P Cunningham, David Schiminovich, and Christian Robert. Expectation\npropagation as a way of life: A framework for Bayesian inference on partitioned data. arXiv\npreprint arXiv:1412.4869, 2017.\n\n[11] Fangjian Guo, Xiangyu Wang, Kai Fan, Tamara Broderick, and David B Dunson. Boosting\n\nVariational Inference. arXiv preprint arXiv:1611.05559, 2017.\n\n[12] Matthew Hoffman and David Blei. Stochastic Structured Variational Inference. In Proceedings\n\nof the 18th International Conference on Arti\ufb01cial Intelligence and Statistics, 2015.\n\n[13] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings\n\nof the 3rd International Conference on Learning Representations, 2015.\n\n[14] Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. Generalized Variational Inference.\n\narXiv preprint arXiv:1904.02063, 2019.\n\n[15] Tomasz Ku\u00b4smierczyk, Joseph Sakaya, and Arto Klami. Correcting Predictions for Approximate\n\nBayesian Inference. arXiv preprint arXiv:1909.04919, 2019.\n\n[16] Simon Lacoste-Julien, Ferenc Husz\u00e1r, and Zoubin Ghahramani. Approximate inference for\nthe loss-calibrated Bayesian. In Proceedings of the 14th International Conference on Arti\ufb01cial\nIntelligence and Statistics, 2011.\n\n[17] Francesco Locatello, Gideon Dresdner, Rajiv Khanna, Isabel Valera, and Gunnar Raetsch.\nBoosting Black Box Variational Inference. Advances in Neural Information Processing Systems\n31, 2018.\n\n[18] Thomas P. Minka. Expectation Propagation for Approximate Bayesian Inference. In Proceedings\n\nof the 17th Conference on Uncertainty in Arti\ufb01cial Intelligence, 2001.\n\n[19] Andriy Mnih and Ruslan Salakhutdinov. Probabilistic Matrix Factorization. In Advances in\n\nNeural Information Processing Systems 20, 2008.\n\n10\n\n\f[20] Christian Naesseth, Francisco Ruiz, Scott Linderman, and David Blei. Reparameterization\nGradients through Acceptance-Rejection Sampling Algorithms. In Proceedings of the 20th\nInternational Conference on Arti\ufb01cial Intelligence and Statistics, 2017.\n\n[21] Tom Rainforth, Adam R. Kosiorek, Tuan Anh Le, Chris J. Maddison, Maximilian Igl, Frank\nWood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. Proceedings of\nthe 35th International Conference on Machine Learning, 2018.\n\n[22] Rajesh Ranganath, Sean Gerrish, and David Blei. Black Box Variational Inference. In Proceed-\n\nings of the 17th International Conference on Arti\ufb01cial Intelligence and Statistics, 2014.\n\n[23] Danilo Rezende and Shakir Mohamed. Variational Inference with Normalizing Flow.\n\nProceedings of the 32nd International Conference on Machine Learning, 2015.\n\nIn\n\n[24] Christian Robert. The Bayesian Choice: From Decision-Theoretic Foundations to Computa-\n\ntional Implementation. Springer Texts in Statistics. Springer New York, 2007.\n\n[25] Francisco J.R. Ruiz, Michalis Titsias, and David Blei. The Generalized Reparameterization\n\nGradient. In Advances in Neural Information Processing Systems 29, 2016.\n\n[26] Tom Schaul and Yann LeCun. Adaptive learning rates and parallelization for stochastic,\nsparse, non-smooth gradients. In Proceedings of the 1st International Conference on Learning\nRepresentations, 2013.\n\n[27] Yee W Teh, David Newman, and Max Welling. A collapsed variational Bayesian inference\nalgorithm for latent Dirichlet allocation. In Advances in Neural Information Processing Systems\n19, 2007.\n\n[28] Michalis Titsias and Miguel L\u00e1zaro-Gredilla. Doubly Stochastic Variational Bayes for non-\nConjugate Inference. In Proceedings of 31st International Conference on Machine Learning,\n2014.\n\n[29] Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Yes, but did it work?: Evalu-\nating variational inference. In Proceedings of the 35th International Conference on Machine\nLearning, 2018.\n\n11\n\n\f", "award": [], "sourceid": 3446, "authors": [{"given_name": "Tomasz", "family_name": "Ku\u015bmierczyk", "institution": "University of Helsinki"}, {"given_name": "Joseph", "family_name": "Sakaya", "institution": "University of Helsinki"}, {"given_name": "Arto", "family_name": "Klami", "institution": "University of Helsinki"}]}