{"title": "Blossom Tree Graphical Models", "book": "Advances in Neural Information Processing Systems", "page_first": 1458, "page_last": 1465, "abstract": "We combine the ideas behind trees and Gaussian graphical models to form a new nonparametric family of graphical models. Our approach is to attach nonparanormal blossoms\", with arbitrary graphs, to a collection of nonparametric trees. The tree edges are chosen to connect variables that most violate joint Gaussianity. The non-tree edges are partitioned into disjoint groups, and assigned to tree nodes using a nonparametric partial correlation statistic. A nonparanormal blossom is then \"grown\" for each group using established methods based on the graphical lasso. The result is a factorization with respect to the union of the tree branches and blossoms, defining a high-dimensional joint density that can be efficiently estimated and evaluated on test points. Theoretical properties and experiments with simulated and real data demonstrate the effectiveness of blossom trees.\"", "full_text": "BlossomTreeGraphicalModelsZheLiuDepartmentofStatisticsUniversityofChicagoJohnLaffertyDepartmentofStatisticsDepartmentofComputerScienceUniversityofChicagoAbstractWecombinetheideasbehindtreesandGaussiangraphicalmodelstoformanewnonparametricfamilyofgraphicalmodels.Ourapproachistoattachnonpara-normal\u201cblossoms\u201d,witharbitrarygraphs,toacollectionofnonparametrictrees.ThetreeedgesarechosentoconnectvariablesthatmostviolatejointGaussianity.Thenon-treeedgesarepartitionedintodisjointgroups,andassignedtotreenodesusinganonparametricpartialcorrelationstatistic.Anonparanormalblossomisthen\u201cgrown\u201dforeachgroupusingestablishedmethodsbasedonthegraphicallasso.Theresultisafactorizationwithrespecttotheunionofthetreebranchesandblossoms,de\ufb01ningahigh-dimensionaljointdensitythatcanbeef\ufb01cientlyes-timatedandevaluatedontestpoints.Theoreticalpropertiesandexperimentswithsimulatedandrealdatademonstratetheeffectivenessofblossomtrees.1IntroductionLetp\u2217(x)beaprobabilitydensityonRdcorrespondingtoarandomvectorX=(X1,...,Xd).TheundirectedgraphG=(V,E)associatedwithp\u2217hasd=|V|verticescorrespondingtoX1,...,Xd,andmissingedges(i,j)6\u2208EwheneverXiandXjareconditionallyindependentgiventheothervariables.Theundirectedgraphisausefulwayofexploringandmodelingthedistribution.Inthispaperweareconcernedwithbuildinggraphicalmodelsforcontinuousvariables,underweakerassumptionsthanthoseimposedbyexistingmethods.Ifp\u2217(x)>0isstrictlypositive,theHammersley-Cliffordtheoremimpliesthatthedensityhastheformp\u2217(x)\u221dYC\u2208C\u03c8C(xC)=exp XC\u2208CfC(xC)!.(1.1)Inthisexpression,Cdenotesthesetofcliquesinthegraph,and\u03c8C(xC)=exp(fC(xC))>0denotesarbitrarypotentialfunctions.Thisrepresentsaverylargeandrichsetofnonparametricgraphicalmodels.Thefundamentaldif\ufb01cultyisthatitisingeneralintractabletocomputethenormalizingconstant.Acompromisemustbemadetoachievecomputationallytractableinference,typicallyinvolvingstrongassumptionsonthefunctionsfC,onthegraphG={C},orboth.ThedefaultmodelforgraphicalmodelingofcontinuousdataisthemultivariateGaussian.WhentheGaussianhascovariancematrix\u03a3,thegraphisencodedinthesparsitypatternoftheprecisionmatrix\u2126=\u03a3\u22121.Speci\ufb01cally,edge(i,j)ismissingifandonlyif\u2126ij=0.Recentworkhasfocusedonsparseestimatesoftheprecisionmatrix[8,10].Inparticular,anef\ufb01cientalgorithmforcomputingtheestimatorusingagraphicalversionofthelassoisdevelopedin[3].Thenonpara-normal[5],aformofGaussiancopula,weakenstheGaussianassumptionbyimposingGaussianityonthetransformedrandomvectorf(X)=(f1(X1),...,fd(Xd)),whereeachfjisamonotonicfunction.Thisallowsarbitrarysinglevariablemarginalprobabilitydistributionsinthemodel[5].1\fBoththeGaussiangraphicalmodelandthenonparanormalmaintaintractableinferencewithoutplacinglimitationsontheindependencegraph.Buttheyarelimitedintheirabilityto\ufb02exiblymodelthebivariateandhigherordermarginals.Atanotherextreme,forest-structuredgraphicalmodelspermitarbitrarybivariatemarginals,butmaintaintractabilitybyrestrictingtoacyclicgraphs.Annonparametricapproachbasedonforestsandtreesisdevelopedin[7]asanonparametricmethodforestimatingthedensityinhigh-dimensionalsettings.However,theabilitytomodelcomplexindependencegraphsiscompromised.InthispaperwebringtogethertheGaussian,nonparanormal,andforestgraphicalmodels,usingwhatwecallblossomtreegraphicalmodels.Informally,ablossomtreeconsistsofaforestoftrees,andacollectionofsubgraphs\u2013theblossoms\u2014possiblycontainingmanycycles.Thevertexsetsoftheblossomsaredisjoint,andeachblossomcontainsatmostonenodeofatree.Weestimatenonparanormalgraphicalmodelsovertheblossoms,andnonparametricbivariatedensitiesoverthebranches(edges)ofthetrees.Usingthepropertiesofthenonparanormal,thesecomponentscanbecombined,orfactored,togiveavalidjointdensityforX=(X1,...,Xd).ThedetailsofourconstructionaregiveninSection2.Wedevelopanestimationprocedureforblossomtreegraphi-calmodels,includinganalgorithmforselectingtreebranches,partitiontheremainingverticesintopotentialblossoms,andthenestimatingthegraphicalstructuresoftheblossoms.Sinceanobjec-tiveistorelaxtheGaussianassumption,ourcriterionforselectingtreebranchesisdeviationfromGaussianity.Towardthisend,weusethenegentropy,showingthatithasstrongstatisticalpropertiesinhighdimensions.Inordertopartitionthenodesintoblossoms,weemployanonparametricpar-tialcorrelationstatistic.Weuseadata-splittingschemetoselecttheoptimalblossomtreestructurebasedonheld-outrisk.Inthefollowingsection,wepresentthedetailsofourmethod,includingde\ufb01nitionsofblossomtreegraphs,theassociatedfamilyofgraphicalmodels,andourestimationmethods.InSections3and4,wepresentexperimentswithsimulatedandrealdata.Finally,weconcludeinSection5.Statisticalproperties,detailedproofs,andfurtherexperimentalresultsarecollectedinasupplement.2BlossomTreeGraphsandEstimationMethodsTounifytheGaussian,nonparanormalandforestgraphicalmodelswemakethefollowingde\ufb01nition.De\ufb01nition2.1.AblossomtreeonanodesetV={1,2,...,d}isagraphG=(V,E),togetherwithadecompositionoftheedgesetEasE=F\u222a{\u222aB\u2208BB}satisfyingthefollowingproperties:1.Fisacyclic;2.V(B)\u2229V(B0)=\u2205,forB,B0\u2208BwithB6=B0,whereV(B)denotesthevertexsetofB.3.|V(B)\u2229V(F)|\u22641foreachB\u2208B;4.V(F)\u222aSBV(B)=V.ThesubgraphsB\u2208Barecalledblossoms.Theuniquenode\u03c1(B)\u2208V(B)\u2229V(F),whichmaybeempty,iscalledthepediceloftheblossom.ThesetofpedicelsisdenotedP(F)\u2282V(F).Property1saysthatthesetofedgesFformsaunionoftrees\u2014aforest.Property2saysthatdistinctblossomssharenoverticesoredgesincommon.Property3saysthateachblossomisconnectedtoatmostonetreenode.Property4saysthateverynodeinthegraphiseitherinatreeorablossom.Notethattheblossomsarenotrequiredtobeconnected,butmusthaveatmostonevertexincommonwiththeforest\u2014thisisthepedicelnode.2\f(a)blossomtree(b)violation(c)blossomtree(d)violationFigure1:Fourgraphs,twoblossomtrees.Thetreeedgesarecoloredblue,theblossomedgesarecoloredblack,andpedicelsareorange.Graphs(a)and(c)correspondtoblossomtrees.Graphs(b)and(d)violatetherestrictionthateachblossomhasonlyasinglepedicel,orattachmenttoatree.Supposethatp(x)=p(x1,...,xd)isthedensityofadistributionthathasanindependencegraphgivenbyablossomtreeF\u222a{\u222aBB}.Thenfromtheblossomtreepropertieswehavethatp(x)=p(XV(F))YB\u2208Bp(XV(B)|XV(F))(2.1)=p(XV(F))YB\u2208Bp(XV(B)|X\u03c1(B))(2.2)=p(XV(F))YB\u2208Bp(XV(B))p(X\u03c1(B))(2.3)=Y(s,t)\u2208Fp(Xs,Xt)p(Xs)p(Xt)Ys\u2208V(F)p(Xs)YB\u2208Bp(XV(B))p(X\u03c1(B))(2.4)=Y(s,t)\u2208Fp(Xs,Xt)p(Xs)p(Xt)Ys\u2208V(F)\\P(F)p(Xs)YB\u2208Bp(XV(B)).(2.5)The\ufb01rstequalityfollowsfromdisjointnessoftheblossoms.Thesecondequalityfollowsfromtheexistenceofasinglepedicelnodeattachingtheblossomtoatree.Thefourthequalityfollowsfromthestandardfactorizationofforests,andthelastequalityfollowsfromthefactthateachnon-emptypedicelforablossomisunique.Wecallthesetofdistributionsthatfactorinthiswaythefamilyofblossomtreegraphicalmodels.Akeypropertyofthenonparanormal[5]isthatthesinglenodemarginalprobabilitiesp(Xs)arearbitrary.Thispropertyallowsustoformgraphicalmodelswhereeachblossomdistributionsatis\ufb01esXV(B)\u223cNPN(\u00b5B,\u03a3B,fB),whileenforcingthatthesinglenodemarginalofthepedicel\u03c1(B)agreeswiththemarginalsofthisnodede\ufb01nedbytheforest.Thisallowsustode\ufb01neandestimatedistributionsthatareconsistentwiththefactorization(2.5).LetX(1),...,X(n)beni.i.d.Rd-valueddatavectorssampledfromp\u2217(x)whereX(l)=(X(l)1,...,X(l)d).Ourgoalistoderiveamethodforhigh-dimensionalundirectedgraphestima-tionanddensityestimation,usingafamilyofsemiparametricestimatorsbasedontheblossomtreestructure.LetFBdenotetheblossomtreestructureF\u222a{\u222aBB}.Ourestimationprocedureisthefollowing.First,randomlypartitionthedataX(1),...,X(n)intotwosetsD1andD2ofsamplesizen1andn2.Thenapplythefollowingsteps.1.UsingD1,estimatethebivariatedensitiesp\u2217(xi,xj)usingkerneldensityestimation.Also,estimatethecovariance\u03a3ijforeachpairofvariables.ApplyKruskal\u2019salgorithmontheestimatedpairwisenegentropymatrixtoconstructafamilyofforests{bF(k)}withk=0,...,d\u22121edges;2.UsingD1,foreachforestbF(k)obtainedinStep1,buildtheblossomtree-structuredgraphbF(k)bB.TheforeststructurebF(k)ismodeledbynonparametrickerneldensityestimators,whileeachblossombB(k)iismodeledbythegraphicallassoornonparanormal.Afamilyof3\fgraphsisobtainedbycomputingregularizationpathsfortheblossoms,usingthegraphicallasso.3.UsingD2,choosebF(bk)bBfromthisfamilyofblossomtreemodelsthatmaximizestheheld-outlog-likelihood.Thedetailsofeachsteparepresentedbelow.2.1Step1:ConstructAFamilyofForestsIninformationtheoryandstatistics,negentropyisusedasameasureofdistancetonormality.ThenegentropyiszeroforGaussiandensitiesandisalwaysnonnegative.Thenegentropybetweenvari-ablesXiandXjisde\ufb01nedasJ(Xi;Xj)=H(\u03c6(xi,xj))\u2212H(p\u2217(xi,xj)),(2.6)whereH(\u00b7)denotesthedifferentialentropyofadensity,and\u03c6(xi,xj)isanGaussiandensitywiththesamemeanandcovariancematrixasp\u2217(xi,xj).Kruskal\u2019salgorithm[4]isagreedyalgorithmto\ufb01ndamaximumweightspanningtreeofaweightedgraph.Ateachstepitincludesanedgeconnectingthepairofnodeswiththemaximumweightamongallunvisitedpairs,ifdoingsodoesnotformacycle.Thealgorithmalsoresultsinthebestk-edgeweightedforestafterk