{"title": "Dip-means: an incremental clustering method for estimating the number of clusters", "book": "Advances in Neural Information Processing Systems", "page_first": 2393, "page_last": 2401, "abstract": "Learning the number of clusters is a key problem in data clustering. We present dip-means, a novel robust incremental method to learn the number of data clusters that may be used as a wrapper around any iterative clustering algorithm of the k-means family. In contrast to many popular methods which make assumptions about the underlying cluster distributions, dip-means only assumes a fundamental cluster property: each cluster to admit a unimodal distribution. The proposed algorithm considers each cluster member as a ''viewer'' and applies a univariate statistic hypothesis test for unimodality (dip-test) on the distribution of the distances between the viewer and the cluster members. Two important advantages are: i) the unimodality test is applied on univariate distance vectors, ii) it can be directly applied with kernel-based methods, since only the pairwise distances are involved in the computations. Experimental results on artificial and real datasets indicate the effectiveness of our method and its superiority over analogous approaches.", "full_text": "Dip-means:anincrementalclusteringmethodforestimatingthenumberofclustersArgyrisKalogeratosDepartmentofComputerScienceUniversityofIoanninaIoannina,Greece45110akaloger@cs.uoi.grAristidisLikasDepartmentofComputerScienceUniversityofIoanninaIoannina,Greece45110arly@cs.uoi.grAbstractLearningthenumberofclustersisakeyproblemindataclustering.Wepresentdip-means,anovelrobustincrementalmethodtolearnthenumberofdataclustersthatcanbeusedasawrapperaroundanyiterativeclusteringalgorithmofk-meansfamily.Incontrasttomanypopularmethodswhichmakeassumptionsabouttheunderlyingclusterdistributions,dip-meansonlyassumesafundamentalclusterproperty:eachclustertoadmitaunimodaldistribution.Theproposedalgorithmconsiderseachclustermemberasanindividual\u2018viewer\u2019andappliesaunivariatestatistichypothesistestforunimodality(dip-test)onthedistributionofdistancesbetweentheviewerandtheclustermembers.Importantadvantagesare:i)theunimodalitytestisappliedonunivariatedistancevectors,ii)itcanbedirectlyappliedwithkernel-basedmethods,sinceonlythepairwisedistancesareinvolvedinthecomputations.Experimentalresultsonarti\ufb01cialandrealdatasetsindicatethee\ufb00ectivenessofourmethodanditssuperiorityoveranalogousapproaches.1IntroductionDataclusteringisadataanalysismethodologywhichaimstoautomaticallyrevealtheunderlyingstructureofdata.Itproducesapartitionofagivendatasetintokgroupsofsimilarobjectsandasataskiswidelyapplicableinarti\ufb01cialintelligence,datamining,statisticsandotherinformationprocessing\ufb01elds.AlthoughitisanNP-hardproblem,variousalgorithmscan\ufb01ndreasonableclus-teringsinpolynomialtime.Mostclusteringmethodsconsiderthenumberofclusterskasarequiredinput,andthentheyapplyanoptimizationproceduretoadjusttheparametersoftheassumedclustermodel.Asaconsequence,inexploratoryanalysis,wherethedatacharacteristicsarenotknowninadvance,anappropriatekvaluemustbechosen.Thisisaratherdi\ufb03cultproblem,butatthesametimeveryfundamentalinordertoapplydataclusteringinpractice.Severalalgorithmshavebeenproposedtodetermineaproperkvalue,mostofwhichwraparoundaniterativemodel-basedclusteringframework,suchasthek-meansorthemoregeneralExpectation-Maximization(EM).Inatop-down(incremental)strategytheystartwithoneclusterandproceedtosplittingaslongasacertaincriterionissatis\ufb01ed.Ateachphase,theyevaluatetheclusteringproducedwitha\ufb01xedkandtheydecidewhethertoincreasethenumberofclustersasfollows:Repeatuntilnochangesoccurinthemodelstructure1.Improvemodelparametersbyrunningaconventionalclusteringalgorithmfora\ufb01xedkvalue.2.Improvemodelstructure,usuallythroughclustersplitting.Oneofthe\ufb01rstattemptsinextendingk-meansinthisdirectionwasx-means[1]whichusesaregu-larizationpenaltybasedonmodel\u2019scomplexity.Tothisend,BayesianInformationCriterion(BIC)[2]wasused,andamongmanymodelstheonewithhighestBICisselected.Thiscriterionworks1\fwellonlyincaseswherethereareplentyofdataandwell-separatedsphericalclusters.Alternativeselectioncriteriahavealsobeenexaminedinliterature[3].G-means[4]isanotherextensiontok-meansthatusesastatisticaltestforthehypothesisthateachclusterhasbeengeneratedfromGaussiandistribution.Sincestatisticaltestsbecomeweakerinhighdimensions,thealgorithm\ufb01rstprojectsthedatapointsofaclusteronanaxisofhighvarianceandthenappliesAnderson-Darlingstatisticwitha\ufb01xedsigni\ufb01cancelevel\u03b1.Clustersthatarenotac-ceptedaresplitrepeatedlyuntiltheentireassumedmixtureofGaussiansisdiscovered.Projectedg-means(pg-means)[5]againassumesthatthedatasethasbeengeneratedfromaGaussianmixture,butitteststheoverallmodelatonceandnoteachclusterseparately.Pg-meansbasesontheEMalgorithm.Usingaseriesofrandomlinearprojections,itconstructsaone-dimensionalprojectionofthedatasetandthelearnedmodelandthenteststhemodel\ufb01tnessintheprojectedspacewithKolmogorov-Smirnov(KS)test.TheadvantageofthismethodistheabilitytodiscoverGaussianclustersofvariousscalesanddi\ufb00erentcovariances,thatmayoverlap.Bayesiank-means[6]intro-ducesMaximization-Expectation(ME)tolearnamixturemodelbymaximizingoverhiddenvari-ables(datapointassignmentstoclusters)andcomputingexpectationoverrandommodelparameters(centersandcovariances).IfthedatacomefromamixtureofGaussiancomponents,thismethodcanbeusedto\ufb01ndthecorrectnumberofclustersandiscompetitivetotheaforementionedapproaches.Otheralternativeshavealsobeenproposed,suchasgapstatistic[7],self-tuningspectralclustering[8],dataspectroscopicclustering[9],andstability-basedmodelvalidation[10]-[12],howevertheyarenotcloselyrelatedtotheproposedmethod.Ourworkisprimarilymotivatedbythenongeneralityoftheapproachesin[4]and[5],astheymakeGausssianityassumptionsabouttheunderlyingdatadistribution.Asaconsequence,theytendtoover\ufb01tforclustersthatareuniformlydistributed,orhaveanon-Gaussianunimodaldistribution.Additionallimitationsarethattheyaredesignedtohandlenumericalvectorsonlyandrequirethedataintheoriginaldataspace.Thecontributionofourworkistwo-fold.Firstly,weproposeasta-tisticaltestforunimodality,calleddip-dist,tobeappliedintoadatasubsetinordertodetermineifitcontainsasingleormultipleclusterstructures.Thus,wemakeamoregeneralassumptionaboutwhatisanacceptablecluster.Moreover,thetestinvolvespairwisedistancesorsimilaritiesandnottheoriginaldatavectors.Secondly,weproposethedip-meansincrementalclusteringmethodwhichisawrapperaroundk-means.Weexperimentallyshowthatdip-meansisabletocopewithdatasetscontainingclustersofarbitrarydensitydistributions.Moreover,itcanbeeasilyextendedinkernelspacebyusingthekernelk-means[13]andmodifyingappropriatelytheclustersplittingprocedure.2Dip-distcriterionforclusterstructureevaluationInclusteranalysis,thedetectionofmultipleclusterstructuresinadatasetrequiresassumptionsaboutwhattheclustersweseeklooklike.Theassumptionsaboutthepresenceofcertaindatachar-acteristicsalongwiththetestsemployedforveri\ufb01cation,considerablyin\ufb02uencetheperformanceofvariousmethods.Itishighlydesirablefortheassumptionstobegeneralinordernottorestricttheapplicabilityofthemethodtocertaintypesofclustersonly(e.g.Gaussian).Moreover,itisofgreatvalueforamethodtobeabletoverifytheassumedclusterhypothesiswithwelldesignedstatisticalhypothesisteststhataretheoreticallysound,incontrasttovariousalternativeadhoccriteria.Weproposethenoveldip-distcriterionforevaluatingtheclusterstructureofadatasetthatisbasedontestingtheempiricaldensitydistributionofthedataforunimodality.Theunimodalityassumptionimpliesthattheempiricaldensityofanacceptableclustershouldhaveasinglemode;aregionwherethedensitybecomesmaximum,whilenon-increasingdensityisobservedwhenmovingawayfromthemode.Therearenootherunderlyingassumptionsabouttheshapeofaclusterandthedistributionthatgeneratedtheempiricallyobservedunimodalproperty.Underthisassumption,itispossibletoidentifyclustersgeneratedbyvariousunimodaldistributions,suchasGaussian,Student-t,etc.TheUniformdistributioncanalsobeidenti\ufb01ed,sinceitisanextremesinglemodecasewherethemodecoversalltheregionwithnon-zerodensity.Aconvenientissueisthatunimodalitycanbeveri\ufb01edusingpowerfulstatisticalhypothesistests(especiallyforone-dimensionaldata),suchasSilverman\u2019smethodwhichuses\ufb01xed-widthkerneldensityestimates[14]orthewidelyusedHartigan\u2019sdipstatistic[15].Asthedimensionalityofthedataincreases,thetestsrequireasu\ufb03cientnumberofdatapointsinordertobereliable.Thus,althoughthedatamaybeofarbitrarydimensionality,itisimportanttoapplyunimodalitytestson2\fone-dimensionaldatavalues.Furthermore,itwouldbedesirable,ifthetestcouldalsobeappliedincaseswherethedistance(orsimilarity)matrixisgivenandnottheoriginaldatapoints.Tomeettheaboverequirementsweproposethedip-distcriterionfordeterminingunimodalityinasetofdatapointsusingonlytheirpairwisedistances(orsimilarities).Morespeci\ufb01cally,ifwecon-sideranarbitrarydatapointasaviewerandformavectorwhosecomponentsarethedistancesoftheviewerfromallthedatapoints,thenthedistributionofthevaluesinthisdistancevectorcouldrevealinformationabouttheclusterstructure.Inpresenceofasinglecluster,thedistributionofdistancesisexpectedtobeunimodal.Inthecaseoftwodistinctclusters,thedistributionofdistancesshouldexhibittwodistinctmodes,witheachmodecontainingthedistancestothedatapointsofeachcluster.Consequently,aunimodalitytestonthedistributionofthevaluesofthedistancevectorwouldprovideindicationabouttheunimodalityoftheclusterstructure.However,thereisadependenceoftheresultsontheselectedviewer.Intuitively,viewersattheboundariesofthesetareexpectedtoformdistancevectorswhosedensitymodesaremoredistinctincaseofmorethanoneclusters.Totackletheviewerselectionproblem,weconsiderallthedatapointsofthesetasindividualviewersandperformtheunimodalitytestonthedistancevectorofeachviewer.Ifthereexistviewersthatrejectunimodality(calledsplitviewers),weconcludethattheexaminedclusterincludesmultipleclusterstructures.FortestingunimodalityweuseHartigans\u2019diptest[15].AfunctionF(t)isunimodalwithmodetheregionsm={(tL,tU):tL\u2264tU}ifitisconvexinsL=(\u2212\u221e,tL],constantin[tL,tU],andconcaveinsU=[tU,\u221e).Thisimpliesthenon-increasingprobabilitydensitybehaviorwhenmovingawayfromthemode.ForboundedinputfunctionsF,G,let\u03c1(F,G)=maxt|F(t)\u2212G(t)|,andletUbetheclassofallunimodaldistributions.ThenthedipstatisticofadistributionfunctionFisgivenby:dip(F)=minG\u2208U\u03c1(F,G).(1)Inotherwords,thedipstatisticcomputestheminimumamongthemaximumdeviationsobservedbetweenthecdfFandthecdfsfromtheclassofunimodaldistributions.Anicepropertyofdipisthat,ifFnisasampledistributionofnobservationsfromF,thenlimn\u2192\u221edip(Fn)=dip(F).In[15]itisarguedthattheclassofuniformdistributionsUisthemostappropriateforthenullhypothesis,sinceitsdipvaluesarestochasticallylargerthanotherunimodaldistributions,suchasthosehavingexponentiallydecreasingtails.Givenavectorofobservationsf={fi:fi\u2208R}ni=1,thenthealgorithmforperformingthediptest[15]isappliedontherespectiveempiricalcdfFn(t)=1nPnI(fi\u2264t).Itexaminesthen(n-1)/2possiblemodalintervals[tL,tU]betweenthesortednindividualobservations.ForallthesecombinationsitcomputesinO(n)timetherespectivegreatestconvexminorantandtheleastconcavemajorantcurvesin(mintFn,tL)and(tU,maxtFn),respectively.Fortunately,foragivenFn,thecomplexityofonedipcomputationisO(n)[15].Thecomputationofthep-valueforaunimodalitytestusesbootstrapsamplesandexpressestheprobabilityofdip(Fn)beinglessthanthedipvalueofacdfUrnofnobservationssampledfromtheU[0,1]Uniformdistribution:P=#[dip(Fn)\u2264dip(Urn)]/b,r=1,...,b.(2)ThenullhypothesisH0thatFnisunimodal,isacceptedatsigni\ufb01cancelevel\u03b1ifp-value>\u03b1,otherwiseH0isrejectedinfavorofthealternativehypothesisH1whichsuggestsmultimodality.LetadatasetX={xi:xi\u2208Rd}Ni=1then,inthepresentcontext,thediptestcanbeappliedonanysubsetc,e.g.adatacluster,andmorespeci\ufb01callyontheecdfFn(xi)(t)=1nPxj\u2208c{Dist(xi,xj)\u2264t}ofthedistancesbetweenareferenceviewerxiofcandthenmembersoftheset.Wecalltheviewersthatidentifymultimodalityandvoteforthesettosplitassplitviewers.Thedip-distcomputationforasetcwithndatapointmembersissummarizedasfollows:1.ComputeUrnandtherespectivedip(Urn),r=1,...,b,fortheUniformsampledistributions.2.ComputeF(xi)nanddip(F(xi)n),i=1,...,n,fordatapointviewersusingthesortedmatrixDist.3.Estimatethep-valuesP(xi),i=1,...,n,basedonEq.2usingasigni\ufb01cancelevel\u03b1andcomputethepercentageofviewersidentifyingmultimodality.SincetheascendingorderingoftherowsofDist,requiredforcomputingF(xi)n,canbedoneoncedur-ingo\ufb04inepreprocessing,andthatthesamebsamplesofUniformdistributioncanbeusedfortestingallviewers,thedip-distcomputationforasetwithndatapointshasO(bnlogn+n2)complexity.3\f00.20.40.60.811.200.20.40.60.8 no splitsplit 71%max dipmin dip(a)dataset100.20.40.60.8100.010.020.030.040.050.06best split viewer: p=0.00, dip=0.1097distancefrequency(b)strongestsplitviewer00.20.40.60.8100.010.020.030.040.050.06worst split viewer: p=0.00, dip=0.0335distancefrequency(c)weakestsplitviewer00.20.40.60.811.200.20.40.60.8 no splitsplit 24%max dipmin dip(d)dataset200.20.40.60.8100.010.020.030.040.050.06best split viewer: p=0.00, dip=0.0776distancefrequency(e)strongestsplitviewer00.20.40.60.8100.010.020.030.040.050.06worst split viewer: p=0.00, dip=0.0335distancefrequency(f)weakestsplitviewer00.20.40.60.811.200.20.40.60.8 no split(g)dataset3(h)densityplot00.20.40.60.811.200.20.40.60.8 no splitsplit 24%max dipmin dip(i)dataset4(j)densityplotFigure1:Applicationofdip-distcriterionon2dsyntheticdatawithtwostructuresof200datapointseach.Thesplitviewersaredenotedinredcolor.(a)OneUniformsphericalandoneellipticGaussianstructure.(b),(c)Thehistogramsofpairwisedistancesofthestrongestandweakestsplitviewer.(d)Thetwostructurescomecloser;thesplitviewersarereduced,sodoesthedipvalueforthesplitviewer.(g)Thetwostructuresarenolongerdistinguishableasthedensitymapin(h)showsonemode.(i)TheUniformsphericalisreplacedwithastructuregeneratedfromaStudent-tdistribution.Figure1illustratesanexampleofapplyingthedip-distcriteriononsyntheticdata.WegeneratedaUniformsphericalandaGaussianellipticstructure,andthenconstructedthreedi\ufb00erenttwo-dimensionaldatasetsbydecreasingthedistancebetweenthem.Thediptestparametersareset\u03b1=0andb=1000.Thehistogramsineachrowindicatetheresultofthediptest.Asthestructurescomecloser,thenumberofviewersthatobservemultimodalitydecreases.Eventually,thestructuresformaunimodaldistribution(Figure1(g)),whichmaybevisuallyveri\ufb01edfromthepresenteddensitymap.ThefourthdatasetofFigure1(j)wascreatedbyincludingastructuregeneratedbyaStudent-tdistributioncenteredatthesamelocationwherethesphereislocatedinFigure1(g).Therespectivedensitymapshowsclearlytwomodes,evidencethatjusti\ufb01eswhythedip-distcriteriondeterminesmultimodalitywith24%oftheviewerssuggestingthesplit.Moregenerally,ifthepercentageofsplitviewersisgreaterthanasmallthreshold,e.g.1%,wemaydecidethattheclusterismultimodal.3Thedip-meansalgorithmDip-meansisanincrementalclusteringalgorithmthatcombinesthreeindividualcomponents.The\ufb01rstisalocalsearchclusteringtechniquethattakesasinputamodelofkclustersandoptimizesthemodelparameters.Forthispurposek-meansisusedwheretheclustermodelsaretheircentroids.Thesecond,andmostimportant,decideswhetheradatasubsetcontainsmultipleclusterstructuresusingthedip-distpresentedinSection2.Thethirdcomponentisadivisiveprocedure(bisecting)that,givenadatasubset,performsthesplittingintotwoclustersandprovidesthetwocenters.Dip-meansmethodologytakesasinputthedatasetXandtwoparametersforthedip-distcriterion:thesigni\ufb01cancelevel\u03b1andthepercentagethresholdvthdofclustermembersthatshouldbesplitviewerstodecideforadivision(Algorithm1).Forthesakeofgenerality,weassumethatdip-means4\fAlgorithm1Dip-means(X,kinit,\u03b1,vthd)input:datasetX={xi}Ni=1,theinitialnumberofclusterskinit,astatisticsigni\ufb01cancelevel\u03b1fortheunimo-dalitytest,percentagevthdofsplitviewersrequiredforaclustertobeconsideredasasplitcandidate.output:thesetsofclustermembersC={cj}kj=1,themodelsM={mj}kj=1withthecentroidofeachcjset.let:score=unimodalityTest(c,\u03b1,vthd)returnsascorevaluefortheclusterc,{C,M}=kmeans(X,k)thek-meansclustering,{C,M}=kmeans(X,M)wheninitializedwithmodelM,{mL,mR}=splitCluster(c)thatsplitsaclustercandreturnstwocentersmL,mR.1:k\u2190kinit2:{C,M}\u2190kmeans(X,k)3:dowhilechangesinclusternumberoccur4:forj=1,...,k%foreachclusterj5:scorej\u2190unimodalityTest(cj,\u03b1,vthd)%computethescoreforunimodalitytest6:endfor7:ifmaxj(scorej)>0%thereexistsplitcandidates8:target\u2190argmaxj(scorej)%indexofclustertobesplitted9:{mL,mR}\u2190splitCluster(ctarget)10:M\u2190{M-mtarget,mL,mR}%replacetheoldcentroidwiththetwonewones11:{C,M}\u2190kmeans(X,M)%re\ufb01nesolution12:endif13:enddo14:return{C,M}maystartfromanyinitialpartitionwithkinit\u22651clusters.Ineachiteration,allkclustersareexaminedforunimodality,thesetofsplitviewersvjisfound,andtherespectiveclustercjischaracterizedassplitcandidateif|vj|/nj\u2265vthd.Inthiscase,anon-zeroscorevalueisassignedtoeachclusterbeingasplitcandidate,whilezeroscoreisassignedtoclustersthatdonothavesu\ufb03cientsplitviewers.Variousalternativescanbeemployedinordertocomputeascoreforasplitcandidatebasedonthepercentageofsplitviewers,oreventhesizeofclusters.Inourimplementationscorejofasplitcandidateclustercjiscomputedastheaveragevalueofthedipstatisticofitssplitviewers:scorej=\uf8f1\uf8f4\uf8f4\uf8f2\uf8f4\uf8f4\uf8f31|vj|Pxi\u2208vjdip(F(xi)),|vj|nj\u2265vthd0,otherwise.(3)Inordertoavoidtheoverestimationoftherealnumberofclusters,onlythecandidatewithmax-imumscoreissplitineachiteration.Aclusterissplitintotwoclustersusinga2-meanslocalsearchapproachstartingfromapairofsu\ufb03cientlydiversecentroidsmL,mRinsidetheclusterandconcerningonlythedatapointsofthatcluster.Weuseasimplewaytosetuptheinitialcentroids{mL,mR}\u2190{x,m\u2212(x\u2212m)},wherexaclustermemberselectedatrandomandmtheclustercen-troid.InthiswaymL,mRlayatequaldistancesfromm,thoughinoppositedirections.The2-meansprocedurecanberepeatedstartingfromdi\ufb00erentmL,mRinitializationsinordertodiscoveragoodsplit.Acomputationallymoreexpensivealternativecouldbethedeterministicprincipaldirectiondivisivepartitioning(PDDP)[16]thatsplitstheclusterbasedontheprincipalcomponent.Were\ufb01nethesolutionattheendofeachiterationusingk-means,which\ufb01ne-tunesthemodelofk+1clusters.Theprocedureterminateswhennosplitcandidatesareidenti\ufb01edamongthealreadyformedclusters.Theproposeddip-distcriterionusesonlythepairwisedistances,orsimilarities,betweendatapointsandnotthevectorrepresentationsthemselves.Thisenablesitsapplicationinkernelspace\u03a6,pro-videdakernelmatrixKwiththeN\u00d7Npairwisedatapointinnerproducts,Kij=\u03c6(xi)T\u03c6(xj).Al-gorithm1canbemodi\ufb01edappropriatelyforthispurpose.Morespeci\ufb01cally,kerneldip-meansuseskernelk-means[13]aslocalsearchtechnique,whichalsoimpliesthatcentroidscannotbecomputedinkernelspace,thuseachclusterisnowdescribedexplicitlybythesetofitsmemberscj.Inthiscase,sincethetransformeddatavectors\u03c6(x)arenotavailable,theclustersplittingprocedurecouldbeseededbytwoarbitraryclustermembers.However,weproposeamoree\ufb03cientapproach.AsdiscussedinSection2,thedistributionofpairwisedistancesbetweenareferenceviewerandthemembersofaclusterrevealsinformationaboutthemultimodalityofdatadistributionintheoriginalspace.Thisimpliesthatasplitoftheclustermembersbasedontheirdistancetoareferenceviewerconstitutesareasonablesplitintheoriginalspace,aswell.Tothisend,wemayuse2-meanstosplittheelementsoftheone-dimensionalsimilarityvector.Weconsiderasreferencesplitviewertheclustermemberwiththemaximumdipvalue.Here,2-meansisseededusingtwovalueslocated5\f00.20.40.60.8100.20.40.60.8  dip\u2212means: ke = 1no split00.20.40.60.8100.20.40.60.8x\u2212means: ke = 100.20.40.60.8100.20.40.60.8g\u2212means: ke = 1000.20.40.60.8100.20.40.60.8pg\u2212means: ke = 4(a)SinglestructuregeneratedbyaStudent-tdistribution00.20.40.60.8100.20.40.60.8  dip\u2212means: ke = 1no split00.20.40.60.8100.20.40.60.8x\u2212means: ke = 200.20.40.60.8100.20.40.60.8g\u2212means: ke = 200.20.40.60.8100.20.40.60.8pg\u2212means: ke = 2(b)SingleUniformrectanglestructure00.20.40.60.8100.20.40.60.8dip\u2212means: ke = 800.20.40.60.8100.20.40.60.8x\u2212means: ke = 2600.20.40.60.8100.20.40.60.8g\u2212means: ke = 3300.20.40.60.8100.20.40.60.8pg\u2212means: ke = 19(c)Eightclustersofvariousdensityandshape00.20.40.60.8100.20.40.60.8kernel dip\u2212means: ke = 200.20.40.60.8100.20.40.60.8kernel k\u2212means: k = 2(d)TwoUniformringstructures00.20.40.60.8100.20.40.60.8kernel dip\u2212means: ke = 300.20.40.60.8100.20.40.60.8kernel k\u2212means: k = 3(e)ThreeUniformringstructuresFigure2:Clusteringresultson2dsyntheticunimodalclusterstructureswith200datapointseach(thecentroidsaremarkedwith\u2297).(a),(b)Singleclusterstructures.(c)Variousstructuretypes.Basedontheleftmostsub\ufb01gure,itcontainsaUniformrectangle(green),aspherewithincreasingdensityatitsperiphery(lightgreen),twoGaussianstructures(black,pink),aUniformellipse(blue),atriangledenseratacorner(yellow),aStudent-t(lightblue),andaUniformarbitraryshape(red).(d),(e)Non-linearlyseparableringclusters(kernel-basedclusteringwithanRBFkernel).atoppositepositionswithrespecttothedistribution\u2019smean.Afterconvergence,theresultingtwo-waypartitionofthedatapoints,derivedbythepartitionofthecorrespondingsimilarityvaluestotheselectedreferencesplitviewer,initializesalocalsearchwithkernel2-means.4ExperimentsInourevaluationwecomparetheproposeddip-meansmethodwithx-means[1],g-means[4]andpg-means[5]thatarecloselyrelatedtopresentwork.Inallcomparedmethodswestartwithasinglecluster(kinit=1)andi)ateachiterationoneclusterisselectedforabisectingsplit,ii)10splittrialsareperformedwith2-meansinitializedwiththesimpletechniquedescribedinSection3,andthesplitwithlowerclusteringerror(thesumofsquareddi\ufb00erencesbetweenclustercentersandtheirassigneddatapoints)iskept,iii)there\ufb01nementisappliedaftereachiterationonallk+1clusters.Hence,onlythestatisticaltestthatdecideswhethertostopsplittingdi\ufb00ersineachcase.Exception6\fTable1:Resultsforsyntheticdatasetswith\ufb01xedk\u2217=20clusterswith200datapointsineachcluster.Case1,d=4Case1,d=16Case1,d=32MethodskeARIVIkeARIVIkeARIVIdip-means20.0\u00b10.01.00\u00b10.00.00\u00b10.020.0\u00b10.01.00\u00b10.00.00\u00b10.020.0\u00b10.01.00\u00b10.00.00\u00b10.0x-means7.3\u00b19.30.30\u00b10.52.07\u00b11.328.6\u00b17.80.88\u00b10.10.27\u00b10.231.3\u00b15.60.84\u00b10.10.36\u00b10.2g-means20.3\u00b10.50.99\u00b10.00.01\u00b10.020.3\u00b10.50.99\u00b10.00.01\u00b10.020.5\u00b10.60.99\u00b10.00.02\u00b10.0pg-means19.2\u00b12.50.90\u00b10.10.16\u00b10.219.0\u00b10.90.95\u00b10.10.07\u00b10.13.2\u00b15.10.09\u00b10.22.62\u00b10.9Case2,d=4Case2,d=16Case2,d=32MethodskeARIVIkeARIVIkeARIVIdip-means20.0\u00b10.00.99\u00b10.00.05\u00b10.020.0\u00b10.00.99\u00b10.00.02\u00b10.020.0\u00b10.00.99\u00b10.00.01\u00b10.0x-means24.8\u00b139.0.26\u00b10.42.26\u00b11.180.1\u00b115.0.75\u00b10.10.75\u00b10.271.6\u00b114.0.75\u00b10.10.66\u00b10.2g-means79.2\u00b122.0.77\u00b10.10.70\u00b10.2105.9\u00b130.0.83\u00b10.10.66\u00b10.2133.6\u00b142.0.83\u00b10.10.72\u00b10.2pg-means14.2\u00b14.70.67\u00b10.20.65\u00b10.510.4\u00b13.40.30\u00b10.21.26\u00b10.54.0\u00b11.50.06\u00b10.12.40\u00b10.2isthepg-meansmethodthatusesEMforlocalsearchanddoesnotrelyonclustersplittingtoaddanewcluster.Weusethemethodexactlyaspresentedin[5].Forthekernel-basedexperimentsweusethenecessarymodi\ufb01cationsdescribedattheendofSection3andcomparewithkernelk-means[13].Theparametersofthedip-distcriterionaresetas\u03b1=0forsigni\ufb01cancelevelofdiptestandb=1000forthenumberofbootstraps.Weconsiderassplitcandidatestheclustershavingatleastvthd=1%splitviewers.Thesevalueswere\ufb01xedinallexperiments.Forbothg-meansandpg-meanswesetthesigni\ufb01cancelevel\u03b1=0.001,whileweuse12randomprojectionsforthelatter.Inordertocomparethegroundtruthlabelingandthegroupingproducedbyclustering,weutilizetheVariationofInformation(VI)[17]metricandtheAdjustedRandIndex(ARI)[18].BetterclusteringisindicatedbylowervaluesofVIandhigherforARI.We\ufb01rstprovideclusteringresultsforsynthetic2ddatasetsinFigure2(kedenotestheestimatednumberofclusters).InFigures2(a),(b),weprovidetwoindicativeexamplesofsingleclusterstruc-tures.X-meansdecidescorrectlyforthestructuregeneratedfromStudent-tdistribution,butover\ufb01tsintheUniformrectanglecase,whiletheothertwomethodsover\ufb01tinbothcases.InthemulticlusterdatasetofFigure2(c),dip-meanssuccessfullydiscoversallclusters,incontrasttotheothermethodsthatsigni\ufb01cantlyoverestimate.Totestthekerneldip-meansextension,wecreatedtwo2dsyntheticdatasetcontainingtwoandthreeUniformringstructuresandweusedanRBFkerneltoconstructthekernelmatrixK.Itisclearthatx-means,g-means,andpg-meansarenotapplicableinthiscase.ThuswepresentinFigures2(d),2(e)theresultsusingkerneldip-meansandalsothebestsolutionfrom50randomlyinitializedrunsofkernelk-meanswiththetruenumberofclusters.Aswemayob-serve,dip-meansestimatesthetruenumberofclustersand\ufb01ndstheoptimalgroupingofdatapointsinbothcases,whereaskernelk-meansfailsinthethreeringcase.Furthermore,wecreatedsyntheticdatasetswithtruenumberk\u2217=20clusters,with200datapointseach,ind=4,16,32dimensionswithlowseparation[19].Twocaseswereconsidered:1)Gaussianmixturesofvaryingeccentricity,and2)datasetswithvariousclusterstructures,i.e.Gaussian(40%),Student-t(20%),Uniformellipses(20%)orUniformrectangles(20%).Foreachcaseanddimensions,wegenerated30datasetstotestthemethods.AstheresultsinTable1indicate,dip-meansprovidesexcellentclusteringperformanceinallcasesandestimatesaccuratelythetruenumberofclusters.Moreover,itperformsremarkablybetterthantheothermethods,especiallyforthedatasetsofCase2.Tworeal-worlddatasetswerealsoused,wheretheprovidedclasslabelswereconsideredasgroundtruth.HandwrittenPendigits(UCI)[20]contains16dimensionalvectors,eachonerepresentingadigitfrom0-9writtenbyahumansubject.ThedataprovideatrainingPDtrandatestingsetPDtewith7494and3498instances,respectively.Wealsoconsidertwosubsetsthatcontainthedigits{0,2,4}(PD3trandPD3te)and{3,6,8,9}(PD4trandPD4te).Wedonotapplyanypreprocessing.Coil-100istheseconddataset[21],whichcontains72imagestakenfromdi\ufb00erentanglesforeachoneofthe100includedobjects.WeusedtreesubsetsCoil3,Coil4,Coil5,withimagesfrom3,4and5objects,respectively.SIFTdescriptors[22]are\ufb01rstextractedfromthegreyscaleimagesthatare\ufb01nallyrepresentedbytheBagofVisualWordsmodelusing1000visualwords.AsreportedinTable2,dip-meanscorrectlydiscoversthenumberofclustersforthesubsetsofPendigits,whileprovidingareasonableunderestimationkeneartheoptimalforthefulldatasetsPD10trandPD10te.Apartfromtheexcessiveover\ufb01ttingofx-meansandg-means,pg-meansseemstoconcludesinoverestimatedke.InthehighdimensionalandsparsespaceoftheconsideredCoilsubsets,x-means7\fTable2:Clusteringresultsforreal-worlddata.Boldindicatesbestvalues.PD3te(k\u2217=3)PD4te(k\u2217=4)PD10te(k\u2217=10)MethodskeARIVIkeARIVIkeARIVIdip-means30.8790.33240.6260.54570.3431.587x-means1550.0313.7921940.0393.7235150.0413.825g-means210.2261.800360.2092.049730.2951.961pg-means40.8350.359100.5760.954130.4471.660PD3tr(k\u2217=3)PD4tr(k\u2217=4)PD10tr(k\u2217=10)MethodskeARIVIkeARIVIkeARIVIdip-means30.9630.11640.5220.84190.4351.452x-means2880.0184.3783810.0204.3729420.0244.387g-means520.1062.641580.1432.4641490.1602.605pg-means50.6550.74080.4391.320140.4941.504Coil3(k\u2217=3)Coil4(k\u2217=4)Coil5(k\u2217=5)MethodskeARIVIkeARIVIkeARIVIdip-means31.0000.00050.9120.17340.7720.308x-means80.4990.899110.4990.951150.6010.907g-means70.6690.650120.5020.977180.4341.204andg-meansprovidemorereasonablekeestimations,butstilloverestimations.Anexplanationforthisbehavioristhattheydiscoversmallergroupsofsimilarimages,i.e.imagestakenfromcloseanglestothesameobject,butfailtounifythesubclustersathigherlevel.Notealsothatwedidnotmanagetotestpg-meansinCoil-100subsets,sincecovariancematriceswerenotpositivede\ufb01nite.Thesuperiorityofdip-meansisalsoindicatedbythereportedvaluesforARIandVImeasures.5ConclusionsWehavepresentedanovelapproachfortestingwhethermultipleclusterstructuresarepresentinasetofdataobjects(e.g.adatacluster).Theproposeddip-distcriterionchecksforunimodalityoftheempiricaldatadensitydistribution,thusitismuchmoregeneralcomparedtoalternativesthattestforGaussianity.Dip-distusesastatisticalhypothesistest,namelyHartigans\u2019diptest,inordertoverifyunimodality.Ifadataobjectofthesetisconsideredasaviewer,thenthediptestcanbeappliedontheone-dimensionaldistance(orsimilarity)vectorwithcomponentsthedistancesbetweentheviewerandthemembersofthesameset.Weexploittheideathattheobservationofmultimodalityinthedistributionofdistancesindicatesmultimodalityoftheoriginaldatadistribution.Byconsideringallthedataobjectsofthesetasindividualviewersandbycombiningtherespectiveresultsofthetest,thepresenceofmultipleclusterstructuresinthesetcanbedetermined.Wehavealsoproposedanewincrementalclusteringalgorithmcalleddip-means,thatincorporatesdip-distcriterioninordertodecideforclustersplitting.Theprocedurestartswithonecluster,ititer-ativelysplitstheclusterindicatedbydip-distasmoreprobabletocontainmultipleclusterstructures,andterminateswhennonewclustersplitissuggested.Bytakingadvantageofthefactthatdip-distutilizesonlyinformationaboutthedistancesbetweendataobjects,wehavemodi\ufb01edappropriatelythemainalgorithmtoproposekerneldip-meanswhichcanbeappliedinkernelspace.Theproposedmethodisfast,easytoimplement,andworksverywellundera\ufb01xedparametersetting.Thereportedclusteringresultsindicatethatdip-meanscanprovidereasonableestimatesofthenumberofclusters,andproducemeaningfulclusteringsinbothdatasettypesinavarietyofarti\ufb01cialandrealdatasets.Apartfromtestingthemethodinreal-worldapplications,thereareseveralwaystoimprovetheimplementationdetailsofthemethod,especiallythekernel-basedversion.Wealsoplantotestitse\ufb00ectivenessinothersettings,suchasonlineclusteringofstreamdata.AcknowledgmentsWethankProf.GregHamerlyforprovidinghiscodeforpg-means.Thedescribedworkissupportedpartiallyandco-\ufb01nancedbytheEuropeanRegionalDevelopmentFund(ERDF)(2007-2013)oftheEuropeanUnionandNationalFunds(OperationalProgramme\u201cCompetitivenessandEntrepreneur-ship\u201d(OPCEII),ROPATTICA),undertheAction\u201cSYNERGASIA(COOPERATION)2009\u201d.8\fReferences[1]D.PellegandAndrewMoore.X-means:extendingk-meanswithe\ufb03cientestimationofthenumberofclusters.InternationalConferenceonMachineLearning(ICML),pp.727-734,2000.[2]R.E.KassandL.Wasserman.AreferenceBayesiantestfornestedhypothesesanditsrelationshiptotheSchwarzcriterion.JournaloftheAmericanStatisticalAssociation,90(431),pp.928-934,1995.[3]X.HuandL.Xu.Acomparativestudyofseveralclusternumberselectioncriteria.InJ.Liuetal.(eds.)IntelligentDataEngineeringandAutomatedLearning,pp.195\u2013202,Springer,2003.[4]G.HamerlyandC.Elkan.Learningthekink-means.AdvancesinNeuralInformationProcessingSystems(NIPS),pp.281-288,2003.[5]Y.FengandG.Hamerly.PG-means:learningthenumberofclustersindata.AdvancesinNeuralInforma-tionProcessingSystems(NIPS),pp.393\u2013400,2006.[6]K.KuriharaandM.Welling.Bayesiank-meansasamaximization-expectationalgorithm.NeuralCompu-tation,21(4),pp.1145\u20131172,2009.[7]R.Tibshirani,G.WaltherandT.Hastie.EstimatingthenumberofclustersinadatasetviatheGapstatistic.JournaloftheRoyalStatisticalSocietyB,63,pp.411-423,2001.[8]L.Zelnik-ManorandP.Perona.Self-tuningspectralclustering.AdvancesinNeuralInformationProcessingSystems(NIPS),pp.1601\u20131608,2004.[9]T.Shi,M.BelkinandB.Yu.DataSpectroscopy:eigenspacesofconvolutionoperatorsandclustering.TheAnnalsofStatistics,37(6B),pp.3960\u20133984,2009.[10]E.LevineandE.Domany.Resamplingmethodforunsupervisedestimationofclustervalidity.NeuralComputation,13(11),pp.2573-2593,2001.[11]RobertTibshiraniandG.Walther.Clustervalidationbypredictionstrength.Computational&GraphicalStatistics,14(3),pp.511-528,2005.[12]T.Lange,V.Roth,MikioL.Braun,andJ.M.Buhmann.Stability-basedvalidationofclusteringsolutions.NeuralComputation,16(6),pp.1299-1323,2004.[13]I.S.Dhillon,Y.GuanandB.Kulis.Kernelk-means:spectralclusteringandnormalizedcuts.InternationalConferenceonKnowledgeDiscoveryandDataMining(SIGKDD),pp.551\u2013556,2004.[14]B.W.Silverman.UsingKerneldensityestimatestoinvestigatemultimodality.JournalofRoyalStatisticSocietyB,43(1),pp.97-99,1981.[15]J.A.HartiganandP.M.Hartigan.Thediptestofunimodality.TheAnnalsofStatistics,13(1),pp.70-84,1985.[16]D.L.Boley.Principaldirectiondivisivepartitioning.DataMiningandKnowledgeDiscovery,2(4),pp.344,1998.[17]M.Meila.Comparingclusterings\u2013aninformationbaseddistance.MultivariateAnalysis,98(5),pp.873-895,2007.[18]L.HubertandP.Arabie.Comparingpartitions.JournalofClassi\ufb01cation,2(1),pp.193-218,1985.[19]J.J.Verbeek,N.Vlassis,andB.Kro\u201dse.E\ufb03cientGreedyLearningofGaussianMixtureModels.NeuralComputation,15(2),pp.469-485,2003.[20]A.AsuncionandD.Newman.UCIMachineLearningRepository.UniversityofCaliforniaatIrvine,Irvine,CA,2007.Availableonline:http://www.ics.uci.edu/mlearn/MLRepository.html[21]S.A.Nene,S.K.NayarandH.Murase.ColumbiaObjectImageLibrary(COIL-100).TechnicalReportCUCS-006-96,February1996.[22]D.Lowe.Distinctiveimagefeaturesfromscale-invariantkeypoints.JournalofComputerVision,60,pp.91-110,2004.9\f", "award": [], "sourceid": 1157, "authors": [{"given_name": "Argyris", "family_name": "Kalogeratos", "institution": null}, {"given_name": "Aristidis", "family_name": "Likas", "institution": null}]}