A Web Search Engine-Based Approach to Measure Semantic Similarity between Words

A Web Search Engine-Based Approach to Measure Semantic Similarity between Words

Abstract— A Web Search Engine-Based Approach to Measure Semantic Similarity between Words. Measuring the semantic similarity between words is an important component in various tasks on the web such as relation extraction, community mining, < Final Year Projects > document clustering, and automatic metadata extraction. Despite the usefulness of semantic similarity measures in these applications, accurately measuring semantic similarity between two words (or entities) remains a challenging task. We propose an empirical method to estimate semantic similarity using page counts and text snippets retrieved from a web search engine for two words. Specifically, we define various word co-occurrence measures using page counts and integrate those with lexical patterns extracted from text snippets. To identify the numerous semantic relations that exist between two given words, we propose a novel pattern extraction algorithm and a pattern clustering algorithm. The optimal combination of page counts-based co-occurrence measures and lexical pattern clusters is learned using support vector machines. The proposed method outperforms various baselines and previously proposed web-based semantic similarity measures on three benchmark data sets showing a high correlation with human ratings. Moreover, the proposed method significantly improves the accuracy in a community mining task.