{"title": "Filtering Abstract Senses From Image Search Results", "book": "Advances in Neural Information Processing Systems", "page_first": 1589, "page_last": 1597, "abstract": "We propose an unsupervised method that, given a word, automatically selects non-abstract senses of that word from an online ontology and generates images depicting the corresponding entities. When faced with the task of learning a visual model based only on the name of an object, a common approach is to find images on the web that are associated with the object name, and then train a visual classifier from the search result. As words are generally polysemous, this approach can lead to relatively noisy models if many examples due to outlier senses are added to the model. We argue that images associated with an abstract word sense should be excluded when training a visual classifier to learn a model of a physical object. While image clustering can group together visually coherent sets of returned images, it can be difficult to distinguish whether an image cluster relates to a desired object or to an abstract sense of the word. We propose a method that uses both image features and the text associated with the images to relate latent topics to particular senses. Our model does not require any human supervision, and takes as input only the name of an object category. We show results of retrieving concrete-sense images in two available multimodal, multi-sense databases, as well as experiment with object classifiers trained on concrete-sense images returned by our method for a set of ten common office objects.", "full_text": "Filtering Abstract Senses From Image Search Results\n\nKate Saenko1,2 and Trevor Darrell2\n\n1 MIT CSAIL, Cambridge, MA\n\n2 UC Berkeley EECS and ICSI, Berkeley, CA\n\nsaenko@csail.mit.edu, trevor@eecs.berkeley.edu\n\nAbstract\n\nWe propose an unsupervised method that, given a word, automatically selects\nnon-abstract senses of that word from an online ontology and generates images\ndepicting the corresponding entities. When faced with the task of learning a vi-\nsual model based only on the name of an object, a common approach is to \ufb01nd\nimages on the web that are associated with the object name and train a visual clas-\nsi\ufb01er from the search result. As words are generally polysemous, this approach\ncan lead to relatively noisy models if many examples due to outlier senses are\nadded to the model. We argue that images associated with an abstract word sense\nshould be excluded when training a visual classi\ufb01er to learn a model of a physical\nobject. While image clustering can group together visually coherent sets of re-\nturned images, it can be dif\ufb01cult to distinguish whether an image cluster relates to\na desired object or to an abstract sense of the word. We propose a method that uses\nboth image features and the text associated with the images to relate latent topics\nto particular senses. Our model does not require any human supervision, and\ntakes as input only the name of an object category. We show results of retrieving\nconcrete-sense images in two available multimodal, multi-sense databases, as well\nas experiment with object classi\ufb01ers trained on concrete-sense images returned by\nour method for a set of ten common of\ufb01ce objects.\n\n1 Introduction\n\nMany practical scenarios call for robots or agents which can learn a visual model on the \ufb02y given\nonly a spoken or textual de\ufb01nition of an object category. A prominent example is the Semantic\nRobot Vision Challenge (SRVC)1 , which provides robot entrants with a text-\ufb01le list of categories to\nbe detected shortly before the competition begins. More generally, we would like a robot or agent\nto be able to engage in situated dialog with a human user and to understand what objects the user\nis refering to. It is generally unreasonable to expect users to refer only to objects covered by static,\nmanually annotated image databases. We therefore need a way to \ufb01nd images for an arbitrary object\nin an unsupervised manner.\nA common approach to learning a visual model based solely on the name of an object is to \ufb01nd\nimages on the web that co-occur with the object name by using popular web search services, and\ntrain a visual classi\ufb01er from the search results. As words are generally polysemous (e.g. mouse)\nand are often used in different contexts (e.g. mouse pad), this approach can lead to relatively noisy\nmodels. Early methods used manual intervention to identify clusters corresponding to the desired\nsense [2], or grouped together visually coherent sets of images using automatic image clustering\n(e.g. [9]). However, image clusters rarely exactly align with object senses because of the large\nvariation in appearance within most categories. Also, clutter from abstract senses of the word that\n\n1http://www.semantic-robot-vision-challenge.org\n\n1\n\n\fFigure 1: WISDOM separates the concrete (physical) senses from the abstract ones.\n\nare not associated with a physical object can further complicate matters (e.g. mouse as in \u201ca timid\nperson\u201d.) 2\nTo address these issues, we propose an unsupervised Web Image Sense DisambiguatiOn Model\n(WISDOM), illustrated in Figure 1. Given a word, WISDOM automatically selects concrete senses\nof that word from a semantic dictionary and generates images depicting the corresponding entities,\n\ufb01rst \ufb01nding coherent topics in both text and image domains, and then grounding the learned topics\nusing the selected word senses. Images corresponding to different visual manifestations of a single\nphysical sense are linked together based on the likelihood of their image content and surrounding\ntext (words in close proximity to the image link) being associated with the given sense.\nWe make use of a well-known semantic dictionary (WordNet [8]), which has been previously used\ntogether with a text-only latent topic model to construct a probabilistic model of individual word\nsenses for use with online images [17]. We build on this work by incorporating a visual term, and\nby using the Wordnet semantic hierarchy to automatically infer whether a particular sense describes\na physical entity or a non-physical concept. We show results of detecting such concrete senses in\ntwo available multimodal (text and image), multi-sense databases: the MIT-ISD dataset [17], and\nthe UIUC-ISD dataset [14]. We also experiment with object classi\ufb01cation in novel images, using\nclassi\ufb01ers trained on the images collected via our method for a set of ten common objects.\n\n2 Related Work\n\nSeveral approaches to building object models from image search results have been proposed. Some\nhave relied on visual similarity, either selecting a single inlier image cluster based on a small valida-\ntion set [9], or bootstrapping object classi\ufb01ers from existing labeled images [13]. In [18] a classi\ufb01er\nbased on text features (such as whether the keyword appears in the URL) was used re-rank the im-\nages before bootstrapping the image model. However, the text ranker was category-independent and\nthus unable to learn words predictive of a speci\ufb01c word sense. An approach most similar to ours [2]\ndiscovered topics in the textual context of images using Latent Dirichlet Allocation (LDA), however,\nmanual intervention by the user was required to sort the topics into positive and negative for each\ncategory. Also, the combination of image and text features is used in some web retrieval methods\n(e.g. [7]), however, our work is focused not on instance-based image retrieval, but on category-level\nmodeling.\nTwo recent papers have speci\ufb01cally addressed polysemous words. In Saenko and Darrell [17], the\nuse of dictionary de\ufb01nitions to train an unsupervised visual sense model was proposed. However, the\nuser was required to manually select the de\ufb01nition for which to build the model. Furthermore, the\n\n2While the \ufb01rst few pages of image search results returned by modern search engines generally have very\nfew abstract examples, possibly due to the sucess of reranking based on previous user\u2019s click-through history,\nresults from farther down the list are much less uniform, as our experimental results show.\n\n2\n\nObject Sense: drink containerInput Word: cupOnline DictionarycupWISDOM\u2022cup(a\u00a0contestin\u00a0which\u00a0a\u00a0cup\u00a0is\u00a0awarded)\u00a0\u201cthe\u00a0World\u00a0Cup\u00a0is\u00a0the\u00a0world's\u00a0most\u00a0widely\u00a0watched\u00a0sporting\u00a0event.\u201d\u2022cup\u00a0(a\u00a0small\u00a0open\u00a0containerusually\u00a0used\u00a0for\u00a0drinking;\u00a0usually\u00a0has\u00a0a\u00a0handle)\u00a0\"he\u00a0put\u00a0the\u00a0cup\u00a0back\u00a0in\u00a0the\u00a0saucer\";\u00a0\"the\u00a0handle\u00a0of\u00a0the\u00a0cup\u00a0was\u00a0missing\"Object Sense: trophyAbstract Sense: sporting event\u2026search\u2022cup,\u00a0loving\u00a0cup(a\u00a0large\u00a0metal\u00a0vessel\u00a0with\u00a0two\u00a0handles\u00a0that\u00a0is\u00a0awarded\u00a0as\u00a0a\u00a0trophyto\u00a0the\u00a0winner\u00a0of\u00a0a\u00a0competition)\u00a0\"the\u00a0school\u00a0kept\u00a0the\u00a0cups\u00a0is\u00a0a\u00a0special\u00a0glass\u00a0case\u201d\u2026cup\fsense model did not incorporate visual features, but rather used the text contexts to re-rank images,\nafter which an image classi\ufb01er was built on the top-ranked results. Loeff et al. [14] performed\nspectral clustering in both the text and image domain and evaluated how well the clusters matched\ndifferent senses. However, as a pure clustering approach, this method cannot assign sense labels.\nIn the text domain, Yarowsky [20] proposed an unsupervised method for traditional word sense\ndisambiguation (WSD), and suggested the use of dictionary de\ufb01nitions as an initial seed. Also, Boiy\net al. [4] determined which words are related to a visual domain using hypothesis testing on a target\n(visual) corpus, compared to a general (non-visual) corpus.\nA related problem is modeling word senses in images manually annotated with words, such as the\ncaption \u201csky, airplane\u201d [1]. Models of annotated images assume that there is a correspondence\nbetween each image region and a word in the caption (e.g. Corr-LDA, [5]). Such models predict\nwords, which serve as category labels, based on image content. In contrast, our model predicts a\ncategory label based on all of the words in the web image\u2019s text context, where a particular word\ndoes not necessarily have a corresponding image region, and vice versa. In work closely related to\nCorr-LDA, a People-LDA [11] model is used to guide topic formation in news photos and captions,\nusing a specialized face recognizer. The caption data is less constrained than annotations, including\nnon-category words, but still far more constrained than generic webpage text.\n\n3 Sense-Grounding with a Dictionary Model\n\nWe wish to estimate the probability that an image search result embedded in a web page is one of\na concrete or abstract concept. First, we determine whether the web image is related to a particular\nword sense, as de\ufb01ned by a dictionary. The dictionary model presented in [17] provides an estimate\nof word sense based on the text associated with the web image. We will \ufb01rst describe this model,\nand then extend it to include both an image component and an adaptation step to better re\ufb02ect word\nsenses present in images.\nThe dictionary model [17] uses LDA on a large collection of text related to the query word to learn\nlatent senses/uses of the word. LDA [6] discovers hidden topics, i.e. distributions over discrete\nobservations (such as words), in the data. Each document is modeled as a mixture of topics z \u2208\n{1, ..., K}. A given collection of M documents, each containing a bag of Nd words, is assumed\nto be generated by the following process: First, we sample the parameters \u03c6j of a multinomial\ndistribution over words from a Dirichlet prior with parameter \u03b2 for each topic j = 1, ..., K. For\neach document d, we sample the parameters \u03b8d of a multinomial distribution over topics from a\nDirichlet prior with parameter \u03b1. Finally, for each word token i, we choose a topic zi from the\nmultinomial \u03b8d, and then choose a word wi from the multinomial \u03c6zi.\nSince learning LDA topics directly from the images\u2019 text contexts can lead to poor results due to\nthe low quantity and irregular quality of such data, an additional dataset of text-only web pages\nis created for learning, using regular web search. The dictionary model then uses the limited text\navailable in the WordNet entries to relate dictionary sense to latent text topics. For example, sense\n1 of \u201cbass\u201d contains the de\ufb01nition \u201cthe lowest part of the musical range,\u201d as well as the hypernym\n(\u201cpitch\u201d) and other semantic relations. The bag-of-words extracted from such a semantic entry for\nsense s \u2208 {1, 2, ..., S} is denoted by the variable es = (e1, e2, ..., eEs), where Es is the total number\nof words. The dictionary model assumes that the sense is independent of the words conditioned on\nthe distribution of topics in the document. For a web image with an associated text document dt, the\nconditional probability of sense is given by\n\nKX\n\nP (s|dt) =\n\nP (s|z = j)P (z = j|dt),\n\n(1)\n\nj=1\n\nwhere the distribution of latent topics in the text context, P (z|dt) is given by the \u03b8dt variable,\ncomputed by generalizing the learned LDA model to the (unseen) text contexts. The likelihood of\na sense given latent topic z = j is de\ufb01ned as the normalized average likelihood of words in the\n\n3\n\n\fdictionary entry es, 3\n\nEsX\n\ni=1\n\nP (ei|z),\n\nP (s|z) \u221d 1\nEs\n\n(2)\n\nIncorporating Image Features. The dictionary model (1) does not take into account the image part\nof the image/text pair. Here, we extend it to include an image term, which can potentially provide\ncomplementary information. First, we estimate P (s|di), or the probability of a sense given an image\ndi. Similar to the text-only case, we learn an LDA model consisting of latent topics v \u2208 {1, ..., L},\nusing the visual bag-of-words extracted from the unlabeled images. The estimated \u03b8 variables give\nP (v|di). To compute the conditional probability of a sense given a visual topic, we marginalize the\njoint P (s, v) across all image and associated text documents {di, dt} in the collection\n\nP (s|dt = k)P (v|di = k)\n\n(3)\n\nP (s|v) \u221d MX\n\nk=1\n\nLX\n\nNote that the above assumes conditional independence of the sense and the visual topic given the\nobservations. Intuitively, this provides us with an estimate of the collocation of senses with visual\ntopic. We can now compute the probability of dictionary sense for a novel image di\u2217 as:\n\nP (s|di\u2217) =\n\nP (s|v = j)P (v = j|di\u2217)\n\n(4)\n\nj=1\n\nFinally, the joint text and image model is de\ufb01ned as the combination of the text-space and image-\nspace models via the sum rule,\n\nP (s|di, dt) = \u03bbP (s|di) + (1 \u2212 \u03bb)P (s|dt)\n\n(5)\nOur assumption in using the sum rule is that the combination can be modelled as a mixture of\nexperts, where the features of one modality are independent of sense given the other modality [3].\nAdaptation. Recall that we can estimate \u03b8dt for the unseen web image contexts by generalizing the\nweb-text LDA model using Gibbs sampling. However, web topics can be a poor match to image\nsearch data (e.g. the \u201cgenome research\u201d topic of mouse.) Our solution is to adapt the web topics to\nthe image search data. We do this by \ufb01xing the z assignments of the web documents and sampling\nthe z\u2019s of the image contexts for a few iterations. This procedure updates the topics to better re\ufb02ect\nthe latent dimensions present in the image search data, without the over\ufb01tting effect mentioned\nearlier.\n\n4 Filtering out Abstract Senses\n\nTo our knowledge, no previous work has considered the task of detecting concrete vs. abstract senses\nin general web images. We can do so by virtue of the multimodal sense grounding method presented\nin the previous section. Given a set of senses for a paricular word, our task is to classify each sense\nas being abstract or concrete. Fortunately, WordNet contains relatively direct metadata related to\nthe physicality of a word sense. In particular, one of the main functions of WordNet is to put words\nin semantic relation to each other using the concepts of hyponym and hypernym. For example,\n\u201cscarlet\u201d and \u201ccrimson\u201d are hyponyms of \u201cred\u201d, while \u201ccolor\u201d is a hypernym of \u201cred\u201d. One can\nfollow the chain of direct hypernyms all the way to the top of the tree, \u201centity\u201d. Thus, we can detect\na concrete sense by examining its hypernym tree to see if it contains one of the following nodes:\n\u2019article\u2019, \u2019instrumentality\u2019,\u2019article of clothing\u2019, \u2019animal\u2019, or \u2019body part\u2019. What\u2019s more, we can thus\nrestrict the model to speci\ufb01c types of physical entities: living things, artifacts, clothing, etc.\nIn addition, WordNet contains lexical \ufb01le information for each sense, marking it as a state, or an an-\nimal, etc. For example, the sense \u201cmouse, computer mouse\u201d is marked <artifact>. In this paper, we\nclassify a WordNet sense as being due to a concrete object when the lexical tag is one of <animal>,\n<artifact>, <body>, <plant> and <act>. We exclude people and proper nouns in the experiments\nin this paper, as well as prune away infrequent senses.\n\n3The average word likelihood was found to be a good indicator of how relevant a topic is to a sense. The\n\ntotal word likelihood could be used, but it would allow senses with longer entries to dominate.\n\n4\n\n\f5 Data\n\nWe evaluated the outlined algorithms on three datasets: the \ufb01ve-word MIT-ISD dataset [17], the\nthree-word UIUC-ISD dataset [14], and OFFICE dataset of ten common of\ufb01ce objects that we col-\nlected for the classi\ufb01cation experiment. 4 All datasets had been collected automatically by issuing\nqueries to the Yahoo Image SearchTM engine and downloading the returned images and correspond-\ning HTML web pages. For the MIT-ISD dataset, the query terms used were: BASS, FACE, MOUSE,\nSPEAKER and WATCH. For the UIUC-ISD dataset, three basic query terms were used: BASS,\nCRANE and SQUASH. To increase corpus size, the authors also used supplemental query terms\nfor each word. The search terms selected were those related to the concrete senses (e.g. \u201ccon-\nstruction cranes\u201d, \u201cwhooping crane\u201d, etc.) Since these human-selected search terms require human\ninput, while our method only requires a list of words, we exclude them from our experiments. The\nOFFICE dataset queries were: CELLPHONE, FORK, HAMMER, KEYBOARD, MUG, PLIERS,\nSCISSORS, STAPLER, TELEPHONE, WATCH.\nThe images were labeled by a human annotator with all concrete senses for each word. The annotator\nsaw only the images, and not the surrounding text or any dictionary de\ufb01nitions. For the MIT-ISD\ndataset, each concrete sense was labeled as core, related, and unrelated. Images where the object\nwas too small or too occluded were labeled as related. For the UIUC-ISD dataset, the labels for\neach concrete sense were similarly core, related and unrelated. In addition, a people label was\nused for unrelated images depicting faces or a crowd. 5 The OFFICE dataset was only labeled with\ncore and unrelated labels. We evaluated our models on two retrieval tasks: retrieval of only core\nimages of each sense, and retrieval of both core and related images. In the former case, core labels\nwere used as positive labels for each sense, with related, unrelated and people images labeled as\nnegative. In the latter case, core and related images were labeled as positive, and unrelated and\npeople as negative. Note that the labels were only used in testing, and not in training.\nTo provide training data for the web text topic model, we also collected an unlabeled corpus of text-\nonly webpages for each word. These additional webpages were collected via regular web search for\nthe single-word search term (e.g. CRANE), and were not labeled.\n\n6 Features\n\nWhen extracting words from web pages, all HTML tags are removed, and the remaining text is\ntokenized. A standard stop-word list of common English words, plus a few domain-speci\ufb01c words\nlike \u201cjpg\u201d, is applied, followed by a Porter stemmer [16]. Words that appear only once and the\nactual word used as the query are pruned. To extract text context words for an image, the image\nlink is located automatically in the corresponding HTML page. All word tokens in a 100-token\nwindow surrounding the location of the image link are extracted. The text vocabulary size used for\nthe dictionary model ranges between 12K-20K words for the different search words.\nTo extract image features, all images are resized to 300 pixels in width and converted to grayscale.\nTwo types of local feature points are detected in the image: edge features [9] and scale-invariant\nsalient points. To detect edge points, we \ufb01rst perform Canny edge detection, and then sample a \ufb01xed\nnumber of points along the edges from a distribution proportional to edge strength. The scales of the\nlocal regions around points are sampled uniformly from the range of 10-50 pixels. To detect scale-\ninvariant salient points, we use the Harris-Laplace [15] detector with the lowest strength threshold\nset to 10. Altogether, 400 edge points and approximately the same number of Harris-Laplace points\nare detected per image. A 128-dimensional SIFT descriptor is used to describe the patch surround-\ning each interest point. After extracting a bag of interest point descriptors for each image, vector\nquantization is performed. A codebook of size 800 is constructed by k-means clustering a randomly\nchosen subset of the database (300 images per keyword), and all images are converted to bags of the\nresulting visual words (cluster centers of the codebook.) No spatial information is included in the\nimage representation, rather it is treated as a bag-of-words.\n\n4The MIT-ISD and OFFICE datasets are available at http://people.csail.mit.edu/saenko\n5The UIUC-ISD dataset and its complete description can be obtained at\n\nhttp://visionpc.cs.uiuc.edu/isd/index.html\n\n5\n\n\f(a) Text Model\n\n(b) Image Model\n\nFigure 2: The top 25 images returned by the text and the image models for mouse-4 (device).\n\n7 Retrieval Experiments\n\nIn this section, we evaluate WISDOM on the task of retrieving concrete sense images from search\nresults. Below are the actual concrete senses that were automatically selected from WordNet by our\nmodel for each word in the datasets:\n\nMIT-ISD: bass-7 (instrument), bass-8 (\ufb01sh), face-1 (human face), face-13 (surface), mouse-1 (ro-\n\ndent), mouse-4 (device), speaker-2 (loudspeaker), watch-1 (timepiece)\n\nUIUC-ISD: bass-7 (instrument), bass-8 (\ufb01sh), crane-4 (machine), crane-5 (bird), squash-1 (plant),\n\nsquash-3 (game)\n\nOFFICE: cellphone-1 (mobile phone), fork-1 (utensil), hammer-2 (hand tool), keyboard-1 (any\nkeyboard), mug-1 (drinking vessel), mug-1 (drinking vessel), pliers-1 (tool), scissors-1\n(cutting tool), stapler-1 (stapling device), telephone-1 (landline phone), watch-1 (time-\npiece)\n\nWe train a separate web text LDA model and a separate image LDA model for each word in the\ndataset. The number of topics K is a parameter to the model that represents the dimensionality of\nthe latent space used by the model. We set K = 8 for all LDA models in the following experiments.\nThis was done so that the number of latent text topics is roughly equal to the number of senses. In the\nimage domain, it is less clear what the number of topics should be. Ideally, each topic would coincide\nwith a visually coherent class of images all belonging to the same sense. In practice, because images\nof an object class on the web are extremely varied, multiple visual clusters are needed to encompass\na single visual category. Our experiments have shown that the model is relatively insensitive to\nvalues of this parameter in the range of 8-32. To perform inference in LDA, we used the Gibbs\nsampling approach of [10], implemented in the Matlab Topic Modeling Toolbox [19]. We used\nsymmetric Dirichlet priors with scalar hyperparameters \u03b1 = 50/K and \u03b2 = 0.01, which have the\neffect of smoothing the empirical topic distribution, and 1000 iterations of Gibbs sampling.\nFigure 2 shows the images that were assigned the highest probability for mouse-4 (computer device)\nsense by the text-only model P (s|dt) (Figure 2(a)), and by the image-only model P (s|di) (Figure\n2(b)). Both models return high-precision results, but somewhat different and complementary im-\nages. As we expected, the image model\u2019s results are more visually coherent, while the text model\u2019s\nresults are more visually varied.\nNext, we evaluate retrieval of individual senses using the multimodal model (Eq. 5, with \u03bb = 0.5)\nand compare it to the Yahoo search engine baseline. This is somewhat unfair to the baseline, as here\nwe assume that our model knows which sense to retrieve (we will remove this assumption later.)\nThe recall-precision curves (RPCs) are shown in Figure 3. The \ufb01gure shows the RPCs for each\nword in the MIT-ISD (top row) and UIUC-ISD (bottom row) datasets, computed by thresholding\nP (s|di, dt). WISDOM\u2019s RPCs are shown as the green curves. The blue curves are the RPCs obtained\nby the original Yahoo image search retrieval order. For example, the top leftmost plot shows retrieval\nof bass-7 (musical instrument). These results demonstrate that we are able to greatly improve the\nretrieval of each concrete sense compared to the search engine.\n\n6\n\n\f(a) MIT-ISD data\n\n(b) UIUC-ISD data\n\nFigure 3: Recall-precision of each concrete sense (core labels) using the multimodal dictionary\nmodel (green) and the search engine (blue), evaluated on two datasets.\n\n(a) Core Senses, MIT-ISD\n\n(b) Core Senses, UIUC-ISD\n\n(c) Core+Related Senses, MIT-ISD\n\n(d) Core+Related Senses, UIUC-ISD\n\nFigure 4: Recall-precision of all concrete senses using WISDOM (green) and the search engine (blue).\n\nWISDOM does fail to retrieve one sense \u2013 face-13, de\ufb01ned as \u201ca vertical surface\u201d. This is a highly\nambiguous sense visually, although it has an <artifact> lexical tag. One possibility for the future\nis to exclude senses that are descendents of \u201csurface\u201d as being too ambiguous. Also, preliminary\ninvestigation indicates that weighting the text and image components of the model differently can\nresult in improved results; model weighting is therefore an important topic for future work.\nNext, we evaluate the ability of WISDOM to \ufb01lter out abstract senses. Here we no longer assume that\nthe correct senses are known. Figure 4 shows the result of \ufb01ltering out the abstract senses, which is\ndone by evaluating the probability of any of the concrete senses in a given search result. The ground\ntruth labels used to compute these RPCs are positive if an image was labeled either with any core\nsense (Fig.4 (a,b)), or with any core or related sense (Fig.4 (c,d)), and negative otherwise. These\nresults demonstrate that our model improves the retrieval of images of concrete (i.e. physical) senses\nof words, without any user input except for the word itself. Figure 5 shows how the model \ufb01lters out\ncertain images, including illustrations by an artist named Crane, from search results for CRANE.\n\n8 Classi\ufb01cation Experiments\n\nWe have shown that our method can improve retrieval of concrete senses, therefore providing higher-\nprecision image training data for object recognition algorithms. We have conjectured that this leads\nto better classi\ufb01cation results; in this section, we provide some initial experiments to support this\nclaim. We collected a dataset of ten of\ufb01ce objects, and trained ten-way SVM classi\ufb01ers using\nthe vocabulary-guided pyramid match kernel over bags of local SIFT features implemented in the\nLIBPMK library [12]. The training data for the SVM was either the \ufb01rst 100 images returned from\nthe search engine, or the top 100 images ranked by our model. Since we\u2019re interested in objects, we\nkeep only the <artifact> senses that descend from \u201cinstrumentality\u201d or \u201carticle\u201d. Figure 6 shows\nclassi\ufb01cation results on held-out test data, averaged over 10 runs on random 80% subsets of the\n\n7\n\n\f(a) Yahoo Image Search\n\n(b) WISDOM\n\nFigure 5: The top images returned by the search engine for CRANE, compared to our model.\n\nFigure 6: Classi\ufb01cation accuracy of ten objects in the OFFICE dataset.\n\ndata. Our method improves accuracy for most of the objects; in particular, classi\ufb01cation of \u201cmug\u201d\nimproves greatly due to the non-object senses being \ufb01ltered out. This is a very dif\ufb01cult task, as\nevidenced by the baseline performance; the average baseline accuracy is 27%. Training with our\nmethod achieves 35% accuracy, a 25% relative improvement. We believe that this relative improve-\nment is due to the higher precision of the training images and will persist even if the overall accuracy\nwere improved due to a better classi\ufb01er.\n\n9 Conclusion\n\nWe presented WISDOM, an architecture for clustering image search results for polysemous words\nbased on image and text co-occurrences and grounding latent topics according to dictionary word\nsenses. Our method distinguishes which senses are abstract from those that are concrete, allowing\nit to \ufb01lter out the abstract senses when constructing a classi\ufb01er for a particular object of interest to\na situated agent. This can be of particular utility to a mobile robot faced with the task of learning\na visual model based only on the name of an object provided on a target list or spoken by a human\nuser. Our method uses both image features and the text associated with the images to relate estimated\nlatent topics to particular senses in a semantic database. WISDOM does not require any human\nsupervision, and takes as input only an English noun. It estimates the probability that a search result\nis associated with an abstract word sense, rather than a sense that is tied to a physical object. We\nhave carried out experiments with image and text-based models to form estimates of abstract vs.\nconcrete senses, and have shown results detecting concrete-sense images in two multimodal, multi-\nsense databases. We also demonstrated a 25% relative improvement in accuracy when classi\ufb01ers are\ntrained with our method as opposed to the raw search results.\n\nAcknowledgments\n\nThis work was supported in part by DARPA, Google, and NSF grants IIS-0905647 and IIS-0819984.\n\n8\n\n\fReferences\n[1] K. Barnard, K. Yanai, M. Johnson, and P. Gabbur. Cross modal disambiguation. In Toward Category-Level\n\nObject Recognition, J. Ponce, M. Hebert, C. Schmidt, eds., Springer-Verlag LNCS Vol. 4170, 2006.\n\n[2] T. Berg and D. Forsyth. Animals on the web. In Proc. CVPR, 2006.\n[3] J. Bilmes and K. Kirchhoff. Directed graphical models of classi\ufb01er combination: application to phone\n\nrecognition. In Proc. ICSLP, 2000.\n\n[4] E. Boiy, K. Deschacht, and M.-F. Moens. Learning Visual Entities and Their Visual Attributes from Text\n\nCorpora. In Proc. DEXA, 2008.\n\n[5] D. Blei and M. Jordan. Modeling annotated data. In Proc. International ACM SIGIR Conference on Re-\n\nsearch and Development in Information Retrieval, pages 127-134. ACM Press, 2003.\n\n[6] D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. J. Machine Learning Research, 3:993-1022,\n\n2003.\n\n[7] Z. Chen, L. Wenyin, F. Zhang and M. Li. Web mining for web image retrieval. J. of the American Society\n\nfor Information Science and Technology, 51:10, pages 831-839, 2001.\n\n[8] C. Fellbaum. Wordnet: An Electronic Lexical Database. Bradford Books, 1998.\n[9] R. Fergus, L. Fei-Fei, P. Perona, and A. Zisserman. Learning Object Categories from Google\u2019s Image\n\nSearch. In Proc. ICCV 2005.\n\n[10] T. Grif\ufb01ths and M. Steyvers. Finding Scienti\ufb01c Topics. In Proc. of the National Academy of Sciences,\n\n101 (suppl. 1), pages 5228-5235, 2004.\n\n[11] V. Jain, E. Learned-Miller, A. McCallum. People-LDA: Anchoring Topics to People using Face Recogni-\n\ntion. In Proc. ICCV, 2007.\n\n[12] J. Lee. LIBPMK: A Pyramid Match Toolkit. MIT Tech Report MIT-CSAIL-TR-2008-17, available online\n\nat http://hdl.handle.net/1721.1/41070. 2008\n\n[13] J. Li, G. Wang, and L. Fei-Fei. OPTIMOL: automatic Object Picture collecTion via Incremental MOdel\n\nLearning. In Proc. CVPR, 2007.\n\n[14] N. Loeff, C. Ovesdotter Alm, D. Forsyth. Discriminating Image Senses by Clustering with Multimodal\n\nFeatures. In Proc. ACL 2006.\n\n[15] K. Mikolajczyk and C. Schmid. Scale and af\ufb01ne invariant interest point detectors. In Proc. IJCV, 2004.\n[16] M. Porter, An algorithm for suf\ufb01x stripping, Program, 14(3) pp 130-137, 1980.\n[17] K, Saenko and T. Darrell. Unsupervised Learning of Visual Sense Models for Polysemous Words. In Proc.\n\nNIPS, 2008.\n\n[18] F. Schroff, A. Criminisi and A. Zisserman. Harvesting image databases from the web. In Proc. ICCV,\n\n2007.\n\n[19] M. Steyvers and T. Grif\ufb01ths. Matlab Topic Modeling Toolbox.\n\nhttp://psiexp.ss.uci.edu/research/software.htm\n\n[20] D. Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. ACL, 1995.\n\n9\n\n\f", "award": [], "sourceid": 1143, "authors": [{"given_name": "Kate", "family_name": "Saenko", "institution": null}, {"given_name": "Trevor", "family_name": "Darrell", "institution": null}]}