Using a Markov Random Field for Image Re-ranking Based on Visual and Textual Features

Abstract

Abstract. We propose a novel method to re-order the list of images returned by an image retrieval system (IRS). The method combines the original order obtained by the IRS, the similarity between images obtained with visual and textual features, and a relevance feedback approach, all of them with the purpose of separating relevant from irrelevant images, and thus, obtaining a more appropriate order. The method is based on a Markov random field (MRF) model, in which each image in the list is represented as a random variable that could be relevant or irrelevant. The energy function proposed for the MRF combines two factors: the similarity between the images in the list (internal similarity); and information obtained from the original order and the similarity of each image with the query (external similarity). Experiments were conducted with resources from the Image CLEF 2008 forum for the photo retrieval track, taking into account textual and visual features. The results show that the proposed method improves, according to the MAP measure, the order of the original list up to 63% (in the textual case) and up to 55% (in the visual case); and suggest future work using a combination of both kind of features.