Heterogeneous Manifold Ranking for Image Retrieval

Abstract-Graph-based ranking models, such as manifold ranking (MR), have been widely used in various image retrieval applications. To further improve such models, a current trend is to fuse the ranking results from multiple feature sets. Most of existing methods mainly concentrate on fusing the homogeneous feature sets derived from a single information channel, like the multiple modalities of image visual content, but little is known in fusing such heterogeneous feature sets derived from multiple information channels as the click-through data associated with images and their visual content. The primary challenge is how to effectively exploit the complementary properties of the heterogeneous feature sets. Another tough issue is the low-quality nature of the click-through data, which makes the exploration of such complementary properties more difficult. In this paper, we propose a heterogeneous MR (HMR) model, in which a couple of graphs built on the click and visual feature sets are fused to simultaneously encode the image ranking results. Specifically, our HMR model applies different solutions to fuse the heterogeneous feature sets in terms of whether the relevance feedback mechanism is available or not. In addition, we develop a click refinement technique to address the noiseness and sparseness problems inherent in the click-through data. Concretely, it prunes the inaccurate clicks from the click-through data using a neighbor voting strategy, and then enriches the pruned data with novel yet accurate clicks based on a novel collaborative filtering (CF) approach, which is devised by integrating the merits of three popularly used CF methods, thus called TriCF algorithm. Extensive experiments on the tasks of click refinement and image retrieval demonstrate the superior performance of the proposed algorithms over several representative methods, especially when the click-through data is highly noisy and sparse.