基于用户行为分析的搜索引擎评价研究
|
摘要
评价是万维网搜索引擎的重要组成部分,是搜索引擎算法改进、系统优化以及日常运营维护的重要保障。传统的评价方式由于大量人力物力资源的消耗,难于满足搜索引擎评价快速全面的要求。如何准确、快速、全面地实现搜索引擎的评价,是急需解决的问题。本文针对万维网用户的信息需求,结合用户行为分析和搜索引擎评价展开相关研究,实现用户行为信息的有效挖掘和搜索引擎快速全面的评价。
本文的研究工作包括:
(1)对用户行为进行宏观统计分析,包括用户的查询分析和点击分析,挖掘用户行为和信息需求之间的联系。同时,区分用户的查询意图,考察不同信息需求下,用户行为的差异性。
(2)针对用户行为中存在的偏置和噪音问题,以及传统方法无法处理长尾查询的不足,提出基于点击粒度的搜索用户行为模型,实现对点击可靠性的评估。实验和分析表明,基于用户思维决策过程导出的行为特征能够区分不同的点击,所提的用户行为模型能够有效实现点击质量的评估,并对长尾查询词有效。
(3)结合用户行为分析方法和传统的Cranfield评价体系,构建基于用户行为分析的搜索引擎搜索性能评价的框架结构,实现相关评价系统。同时,针对单搜索引擎用户行为信息存在的不足,提出基于多搜索引擎用户行为信息的MCTR模型,实现对查询的自动标注。相关实验结果表明,该自动标注方法具有一定的准确性,能够完成自动评价搜索引擎结果的任务。
(4)针对万维网信息数据规模大的特点,提出了用户访问万维网(User Accessed Web,简称UA Web)的概念。结合用户浏览行为信息,利用蒙特卡洛随机采样过程实现页面数据的均一采样。考察和分析用户访问万维网的特点,以及搜索引擎的索引规模和索引结构。
(5)针对搜索引擎评价的具体目标和内容,整合用户行为分析和搜索引擎评价的相关研究成果,提出一个搜索引擎整体评价系统的设计方案,希望能够满足搜索引擎快速、全面的评价要求。
Performance evaluation is an important issue for Web search engines in terms of algorithm improvement, system optimization, and maintenance. Traditional methods cannot satisfy the request of search engine evaluation due to huge amount of human efforts and an extremely time-consuming process in practice. This paper study user behehavior and mine useful information to evaluation Web search engine’s performance fully and automatically. The contributions of this paper are:
(1) Based on interactive process between user and search engine, we present an analysis of user behaviors about querying and clicking, and mined the relationship between user behavior and need. We also analyses different user behaviors for different user need based on several types of query needs.
(2) Due to the bias and noisy in user behavior, we propose a behavior model to estimate click reliability. Experimental results show that the proposed features can be separating reliable clicks from other ones, and the model effectively identifies click quality which works well for hot queries and long-tail ones.
(3) This paper presents a performance evaluation method under Cranfield framework fully and automatically, and constructs an evaluation system based on user click-through behavior. MCTR model is proposed to eliminate potential and inherent bias. The results show our method produces evaluation results similar to those gained by traditional human annotation.
(4) The UA Web is proposed to describe the userful information on the Web and Monte Carlo simulation process is adopted to generate near-uniform sampling page sets. Experimental results reveal some properties of the UA Web and the index profile of four commercial search engines.
(5) Considering the goal and content of Web search engine evaluation, we combine the research results about user behavior analysis and search engine evaluation, propose an improved evaluation system to work fully and automatically.
本文的研究工作包括:
(1)对用户行为进行宏观统计分析,包括用户的查询分析和点击分析,挖掘用户行为和信息需求之间的联系。同时,区分用户的查询意图,考察不同信息需求下,用户行为的差异性。
(2)针对用户行为中存在的偏置和噪音问题,以及传统方法无法处理长尾查询的不足,提出基于点击粒度的搜索用户行为模型,实现对点击可靠性的评估。实验和分析表明,基于用户思维决策过程导出的行为特征能够区分不同的点击,所提的用户行为模型能够有效实现点击质量的评估,并对长尾查询词有效。
(3)结合用户行为分析方法和传统的Cranfield评价体系,构建基于用户行为分析的搜索引擎搜索性能评价的框架结构,实现相关评价系统。同时,针对单搜索引擎用户行为信息存在的不足,提出基于多搜索引擎用户行为信息的MCTR模型,实现对查询的自动标注。相关实验结果表明,该自动标注方法具有一定的准确性,能够完成自动评价搜索引擎结果的任务。
(4)针对万维网信息数据规模大的特点,提出了用户访问万维网(User Accessed Web,简称UA Web)的概念。结合用户浏览行为信息,利用蒙特卡洛随机采样过程实现页面数据的均一采样。考察和分析用户访问万维网的特点,以及搜索引擎的索引规模和索引结构。
(5)针对搜索引擎评价的具体目标和内容,整合用户行为分析和搜索引擎评价的相关研究成果,提出一个搜索引擎整体评价系统的设计方案,希望能够满足搜索引擎快速、全面的评价要求。
Performance evaluation is an important issue for Web search engines in terms of algorithm improvement, system optimization, and maintenance. Traditional methods cannot satisfy the request of search engine evaluation due to huge amount of human efforts and an extremely time-consuming process in practice. This paper study user behehavior and mine useful information to evaluation Web search engine’s performance fully and automatically. The contributions of this paper are:
(1) Based on interactive process between user and search engine, we present an analysis of user behaviors about querying and clicking, and mined the relationship between user behavior and need. We also analyses different user behaviors for different user need based on several types of query needs.
(2) Due to the bias and noisy in user behavior, we propose a behavior model to estimate click reliability. Experimental results show that the proposed features can be separating reliable clicks from other ones, and the model effectively identifies click quality which works well for hot queries and long-tail ones.
(3) This paper presents a performance evaluation method under Cranfield framework fully and automatically, and constructs an evaluation system based on user click-through behavior. MCTR model is proposed to eliminate potential and inherent bias. The results show our method produces evaluation results similar to those gained by traditional human annotation.
(4) The UA Web is proposed to describe the userful information on the Web and Monte Carlo simulation process is adopted to generate near-uniform sampling page sets. Experimental results reveal some properties of the UA Web and the index profile of four commercial search engines.
(5) Considering the goal and content of Web search engine evaluation, we combine the research results about user behavior analysis and search engine evaluation, propose an improved evaluation system to work fully and automatically.
引文
Adamic L, Huberman B. 2000. The nature of markets on the World Wide Web. Quart. J. Electron. Comm. 1(1):5-12.
Agichtein E, Brill E, Dumais S. 2006a. Improving Web search ranking by incorporating user behavior information. Proceedings of the 29th Annual international ACM SIGIR
Conference on Research and Development in information Retrieval (Seattle, Washington, USA, August 06 - 11, 2006). SIGIR '06. ACM Press, New York, NY, 19-26.
Agichtein E, Brill E, Dumais S, Ragno R. 2006b. Learning user interaction models for predicting Web search result preferences. Proceedings of the 29th Annual international ACM SIGIR Conference on Research and Development in information
Retrieval (Seattle, Washington, USA, August 06 - 11, 2006). SIGIR '06. ACM Press, New York, NY, 3-10.
Agrawal R, Halverson A, Kenthapadi K, Mishra N, Tsaparas P. 2009. Generating labels from clicks. Proceedings of the 2nd ACM international Conference on Web Search and Data Mining (Barcelona, Spain, February 09 - 12, 2009). R. Baeza-Yates, P.
Boldi, B. Ribeiro-Neto, and B. B. Cambazoglu, Eds. WSDM '09. ACM Press, New York, NY, 172-181.
Amitay E, Carmel D, Lempel R, Soffer A. 2004. Scaling IR-system evaluation using term relevance sets. Proceedings of the 27th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Sheffield, United Kingdom, July 25 - 29, 2004). SIGIR '04. ACM Press, New York, NY, 10-17.
Baeza-Yates R, Hurtado C, Mendoza M, Dupret G. 2005. Modeling user search behavior. Proceedings of the 3rd Latin American Web Congress. LA-WEB. IEEE Computer Society, Washington, DC, 242.
Baeza-Yates R, Ribeiro-Neto B. 1999. Modern information retrieval. Addison-Wesley (ACM Press series).
Baeza-Yates R, Tiberi A. 2007. Extracting semantic relations from query logs. Proceedings of the 13th ACM SIGKDD international Conference on Knowledge
Discovery and Data Mining (San Jose, California, USA, August 12 - 15, 2007). KDD '07. ACM Press, New York, NY, 76-85.
Bar-Yossef Z, Berg A, Chien S, Fakcharoenphol J, Weitz D. 2000. Approximating aggregate queries about Web pages via random walks. Proceedings of the 26th international Conference on Very Large Data Bases (September 10 - 14, 2000). Very
Large Data Bases. Morgan Kaufmann Publishers, San Francisco, CA, 535-544.
Bar-Yossef Z, Gurevich M. 2006. Random sampling from a search engine's index. Proceedings of the 15th international Conference on World Wide Web (Edinburgh, Scotland, May 23 - 26, 2006). WWW '06. ACM Press, New York, NY, 367-376.
Battelle J. 2005. John Battelle's searchblog. Retrieved from http://battellemedia.com/archives/001889.php.
Bharat K, Broder A. 1998. A technique for measuring the relative size and overlap of public Web search engines. Proceedings of the 7th international Conference on
World Wide Web 7 (Brisbane, Australia). P. H. Enslow and A. Ellis, Eds. Elsevier Science Publishers B. V., Amsterdam, The Netherlands, 379-388.
Bilenko M, White R. 2008. Mining the search trails of surfing crowds: identifying relevant websites from user activity. Proceeding of the 17th international Conference on World Wide Web (Beijing, China, April 21 - 25, 2008). WWW '08. ACM Press, New York, NY, 51-60.
Boldi P, Bonchi F, Castillo C, Vigna S. 2009. From "Dango" to "Japanese Cakes": Query Reformulation Models and Patterns. Proceedings of the 2009 IEEE/WIC/ACM international Joint Conference on Web intelligence and intelligent Agent Technology - Volume 01 (September 15 - 18, 2009). Web Intelligence & Intelligent Agent. IEEE Computer Society, Washington, DC, 183-190.
Bradlow E, Schmittlein D. 2000. The little engines that could: modeling the performance of World Wide Web search engines. Marketing Science, 19:43–62.
Brin S, Page L. 1998. The anatomy of a large-scale hypertextual Web search engine. Proceedings of the 7th international Conference on World Wide Web 7 (Brisbane, Australia). P. H. Enslow and A. Ellis, Eds. Elsevier Science Publishers B. V., Amsterdam, The Netherlands, 107-117.
Broder A, Kumar R, Maghoul F, Raghavan P, Rajagopalan S, Stata R, Tomkins A, Wiener J. 2000. Graph structure in the Web. Comput Netw:309-320.
Broder A. 2002. A taxonomy of Web search. SIGIR Forum 36, 2 (Sep. 2002), 3-10. Buckley C, Dimmick D, Soboroff I, Voorhees E. 2007. Bias and the limits of pooling for large collections. Inf. Retr. 10, 6, 491-508.
Buckley C, Voorhees E. 2004. Retrieval evaluation with incomplete information, Proceedings of the 30th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Amsterdam, The Netherlands, July 23 - 27, 2007). SIGIR '07. ACM Press, New York, NY, 63-70.
Büttcher S, Clarke C, Yeung P, Soboroff I. 2007. Reliable information retrieval evaluation with incomplete and biased judgments. Proceedings of the 30th Annual international ACM SIGIR Conference on Research and Development in informationRetrieval (Amsterdam, The Netherlands, July 23 - 27, 2007). SIGIR '07. ACM Press, New York, NY, 63-70.
Carletta J. 1996. Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics 22(2):249-254.
Chowdhury A, Soboroff I. 2002. Automatic evaluation of World Wide Web search services. Proceedings of the 25th Annual international ACM SIGIR Conference on
Research and Development in information Retrieval (Tampere, Finland, August 11 - 15, 2002). SIGIR '02. ACM Press, New York, NY, 421-422.
Cleverdon C, Mills J, Keen M. 1966. Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 1, Design; Part 1. Cockburn A, Jones S. 1996. Which way now? Analysing and easing inadequacies in
WWW navigation. International Journal of Human-Computer Studies, 45:105-129. Craswell M, Hawking D. 2004. Overview of the TREC 2003 Web track. In E. M.
Voorhees and Lori P. Buckland, eds. NIST Special Publication 500-261: TREC 2004. Washington: Department of Commerce and National Institute of Standards and Technology.
Craswell N, Zoeter O, Taylor M, Ramsey B. 2008. An experimental comparison of click position-bias models. Proceedings of the international Conference on Web Search and Web Data Mining (Palo Alto, California, USA, February 11 - 12, 2008). WSDM '08. ACM Press, New York, NY, 87-94.
Dobra A, Fienberg S. 2004. How large is the World Wide Web? M. Levene and A. Poulovassilis (eds), Web Dynamics: Adapting to Change in Content, Size, Topology and Use (Springer, Berlin/ Heidelberg) , 23-44.
Dou Z, Song R, Yuan X, Wen J. 2008. Are clickthrough data adequate for learning web search rankings?. Proceedings of the 18th ACM Conference on information and Knowledge Management (Hong Kong, China, November 02 - 06, 2009). CIKM '09. ACM Press, New York, NY, 1077-1086.
Downey D, Dumais S, Liebling D, Horvitz E. 2008. Understanding the relationship between searchers' queries and information goals. Proceeding of the 17th ACM Conference on information and Knowledge Management (Napa Valley, California, USA, October 26 - 30, 2008). CIKM '08. ACM Press, New York, NY, 449-458.
Fawcett T. 2006. An introduction to ROC analysis. Pattern Recogn. Lett. 27(8):861-874.
French B. 2009. Google semantic upgrade enhances long tail query results. Retrieved from http://blogsite.com/public/item/230059.
Fuxman A, Tsaparas P, Achan K, Agrawal R. 2008. Using the wisdom of the crowds for keyword generation. Proceeding of the 17th international Conference on World WideWeb (Beijing, China, April 21 - 25, 2008). WWW '08. ACM Press, New York, NY, 61-70.
Gao J, Yuan W, Li X, Deng K, Nie J. 2009. Smoothing clickthrough data for web search ranking. Proceedings of the 32nd international ACM SIGIR Conference on Research and Development in information Retrieval (Boston, MA, USA, July 19 - 23, 2009). SIGIR '09. ACM Press, New York, NY, 355-362.
Glassman S. 1994. A caching relay for the World Wide Web. Comput. Netw. ISDN Syst. 27(2):165-173.
Google. 2008. Official Google blog: We knew the web was big. Retrieved from http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html.
Gulli A, Signorini A. 2005. The indexable Web is more than 11.5 billion pages. In Special interest Tracks and Posters of the 14th international Conference on World Wide Web (Chiba, Japan, May 10 - 14, 2005). WWW '05. ACM Press, New York, NY, 902-903. Guo F, Liu C, Wang Y. 2009. Efficient multiple-click models in web search. Proceedings of the 2nd ACM international Conference on Web Search and Data Mining (Barcelona, Spain, February 09 - 12, 2009). R. Baeza-Yates, P. Boldi, B. Ribeiro-Neto, and B. B. Cambazoglu, Eds. WSDM '09. ACM Press, New York, NY, 124-131.
Han I, Lee S, Lee S. 2007. Graph structure of the Korea Web. Proceedings of the 12th DASFAA, 930–935.
Hawking D, Craswell N, Thistlewaite P, Harman D. 1999. Results and challenges in Web search evaluation. Proceedings of the 8th international World Wide Web Conference. North-Holland Publishing Co., Amsterdam, The Netherlands, 1321-1330.
Hawking D, Craswell N. 2002. Overview of the TREC-2002 web track. In The 11th Text Retrieval Conference (TREC-2002), volume 11. National Institute of Standards and Technology, NIST.
Hawking D, Craswell N. 2003. Overview of the TREC-2003 Web track. In NIST Special Publication: SP 500-255, The 12th Text Retrieval Conference (TREC 2003). Hedger J. 2005. Google takes backhanded bow out of size war with Yahoo. Retrieved from http://www.searchenginejournal.com/?p=2277.
Henzinger M, Heydon A, Mitzenmacher M, Najork M. 2000. On near-uniform URL sampling. Proceedings of the 9th international World Wide Web Conference on Computer Networks : the international Journal of Computer and Telecommunications Netowrking (Amsterdam, The Netherlands). North-Holland Publishing Co., Amsterdam, The Netherlands, 295-308.
Henzinger M, Motwani R, Silverstein C. 2002. Challenges in web search engines. SIGIR Forum 36, 2, 11-22.
Huberman B, Pirolli P, Pitkow J, Lukose R. 1998. Strong regularities in World Wide Web surfing. Science, 280(3):95-97.
Jarvelin K, Kekalainen J. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transaction on Information Systems (TOIS), 20(4):422-446.
Joachims T, Freitag D, Mitchell T. 1997. WebWatcher: a tour guide for the World Wide Web. Proceedings of the 15th International Joint Conference on Artificial Intelligence. IJCAI’97. Morgan Kaufmann, 1, 770– 777.
Joachims T, Granka L, Pan B, Hembrooke H, Gay G. 2005. Accurately interpreting clickthrough data as implicit feedback. Proceedings of the 28th Annual international
ACM SIGIR Conference on Research and Development in information Retrieval (Salvador, Brazil, August 15 - 19, 2005). SIGIR '05. ACM Press, New York, NY, 154-161.
Joachims T, Granka L, Pan B, Hembrooke H, Radlinski F, Gay G. 2007. Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search. ACM Transactions on Information Systems, 25(2):7.
Joachims T. 2002a. Evaluating retrieval performance using clickthrough data. Proceedings of the SIGIR Workshop on Mathematical/FormalMethods in Information Retrieval.
Joachims T. 2002b. Optimizing search engines using clickthrough data. Proceedings of the 8th ACM SIGKDD international Conference on Knowledge Discovery and Data Mining (Edmonton, Alberta, Canada, July 23 - 26, 2002). KDD '02. ACM Press, New York, NY, 133-142.
Jolliffe I. 1986. Principal Component Analysis. Springer-Verlag, New York. Kammenhuber N, Luxenburger J, Feldmann A, Weikum G. 2006. Web search clickstreams. Proceedings of the 6th ACM SIGCOMM Conference on internet Measurement. IMC '06. ACM Press, New York, NY, 245-250.
Kent A, Berry M, Leuhrs F, Perry, J. 1955. Machine literature searching VIII. Operational criteria for designing information retrieval systems. American Documentation, 6(2): 93-101.
Lawrence S, Giles C. 1998. Searching the World Wide Web. Science, 5360(280):98.
Lawrence S, Giles C. 1999. Accessibility of information on the Web. Nature, 400:107–109.
Lee U, Liu Z, Cho J. 2005. Automatic identification of user goals in Web search. Proceedings of the 14th international Conference on World Wide Web (Chiba, Japan, May 10 - 14, 2005). WWW '05. ACM Press, New York, NY, 391-400.
Li Y, Wang B, Xu S, Li J, Li P. 2009. QueryTrans: finding similar queries based on query trace graph. Proceedings of the 2009 IEEE/WIC/ACM International Conference on Web Intelligence. WI 2009. Universitàdegli Studi di Milano Bicocca, Milano, 260-263.
Liu J. 2001. Monte carlo strategies in scientific computing. Springer.
Liu Y, Gao B, Liu T, Zhang Y, Ma Z, He S, Li H. 2008. BrowseRank: letting web users vote for page importance. Proceedings of the 31st Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Singapore, Singapore, July 20 - 24, 2008). SIGIR '08. ACM Press, New York, NY, 451-458.
Liu Y, Jin Y, Zhang M, Ma S, Ru L. 2009. User browsing graph: structure, evolution and application. Late breaking result session in 2nd ACM International Conference on Web Search and Data Mining (WSDM 2009).
Liu Y, Zhang M, Ru L, Ma S. 2006. Automatic query type identification based on click through information. In H.T. Ng et al. (Eds.): AIRS 2006, LNCS 4182, pp. 593–600.
Lyman P, Hal R, Varian, 2003. How much information [EB/OL]. Retrieved from http://www.sims.berkeley.edu/how-much-info-2003.
Martindale C, Konopka A. 1996. Oligonucleotide frequencies in DNA follow a Yule distribution. Computers and Chemistry, 20(1):35-38.
Mei Q, Church K. 2008. Entropy of search logs: how hard is search? with personalization? with backoff?. Proceedings of the international Conference on Web Search and Web Data Mining (Palo Alto, California, USA, February 11 - 12, 2008). WSDM '08. ACM Press, New York, NY, 45-54.
Nuray R, Can F. 2003. Automatic ranking of retrieval systems in imperfect environments. Proceedings of the 26th Annual international ACM SIGIR Conference on Research and Development in informaion Retrieval (Toronto, Canada, July 28 - August 01, 2003). SIGIR '03. ACM Press, New York, NY, 379-380.
Pearson K. 1901. On lines and planes of closest fit to system of points in space. Philosophical Magazine, B(2):559-572.
Radlinski F, Kurup M, Joachims T. 2007. Active exploration for learning rankings from clickthrough data. Proceedings of the 13th ACM SIGKDD international Conference on Knowledge Discovery and Data Mining (San Jose, California, USA, August 12 - 15, 2007). KDD '07. ACM Press, New York, NY, 570-579.
Rose D, Levinson D. 2004. Understanding user goals in web search. Proceedings of the 13th international Conference on World Wide Web (New York, NY, USA, May 17 - 20, 2004). WWW '04. ACM Press, New York, NY, 13-19.
Sadagopan N, Li J. 2008. Characterizing typical and atypical user sessions in clickstreams. Proceeding of the 17th international Conference on World Wide Web (Beijing, China, April 21 - 25, 2008). WWW '08. ACM Press, New York, NY, 885-894.
Saracevic T. 1995. Evaluation of evaluation in information retrieval. Proceedings of the 18th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Seattle, Washington, United States, July 09 - 13, 1995). E. A.
Fox, P. Ingwersen, and R. Fidel, Eds. SIGIR '95. ACM Press, New York, NY, 138-146.
Selberg E. 1999. Towards comprehensive Web search. Doctoral Thesis. UMI Order Number: AAI9936480, University of Washington.
Shannon C. 1948. A mathematical theory of communication. Bell System Technical Journal, 27:379-423.
Silverstein C, Marais H, Henzinger M, Moricz M. 1999. Analysis of a very large web search engine query log. SIGIR Forum 33, 1, 6-12.
Soboroff I, Nicholas C, Cahan P. 2001. Ranking retrieval systems without relevance judgments. Proceedings of the 24th Annual international ACM SIGIR Conference on
Research and Development in information Retrieval (New Orleans, Louisiana, United States). SIGIR '01. ACM Press, New York, NY, 66-73.
Soboroff I, Voorhees E, Craswell N. 2003. Summary of the SIGIR 2003 workshop on defining evaluation methodologies for terabyte-scale test collections. SIGIR Forum 37, 2, 55-58.
Soboroff I. 2004. On evaluating web search with very few relevant documents. Proceedings of the 27th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Sheffield, U.K., July, 2004). SIGIR '04. ACM Press, New York, NY, 530-531.
Sullivan D. 2005. Search engine sizes [EB/OL]. Retrieved from search engine watch Web site http://searchenginewatch.com/reports/article.php/2156481.
Svore K, Wu Q, Burges C, Raman A. 2007. Improving Web spam classification using rank-time features. Proceedings of the 3rd international Workshop on Adversarial information Retrieval on the Web (Banff, Alberta, Canada, May 08 - 08, 2007). AIRWeb '07, vol. 215. ACM Press, New York, NY, 9-16
Tan P, Kumar V. 2000. Modeling of Web robot navigational patterns. In Procings of ACM WebKDD Workshop. ACM Press, New York, NY.
Tan P, Kumar V. 2002. Discovery of Web robot sessions based on their navigational patterns. Data Mining and Knowledge Discovery, 6(1):9-35.
Tauscher L, Greenberg S. 1997. How people revisit web pages: Empirical findings and implications for the design of history systems. International Journal of Human-Computer Studies, 47:97-137.
Von-Neumann J. 1951. Various technique used in connection with random digits. Washington: NBS Appl. Math, 12: 36-38.
Voorhees E. 2002. The philosophy of information retrieval evaluation. In Revised Papers
From the 2nd Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language information Retrieval Systems (September 03 - 04, 2001). C. Peters, M. Braschler, J. Gonzalo, and M. Kluck, Eds. Lecture Notes In Computer Science, vol. 2406. Springer-Verlag, London, 355-370.
Yule G. 1944. Statistical study of literary vocabulary. Cambridge, Cambridge University Press.
Zipf G. 1949. Human behavior and the principle of least effort. Addison-Wesley Press.
百度百科. 2009.搜索引擎. http://baike.baidu.com/view/2647196.htm.
刘奕群,岑荣伟,张敏,茹立云,马少平. 2007.基于用户行为分析的搜索引擎自动性能评价.软件学报, 19(11), 3023-3032.
刘奕群. 2010.搜索引擎技术基础.
薛宇飞,刘奕群,张敏,马少平,茹立云. 2009.基于用户浏览图的网页质量评估方法的比较分析.全国第十届计算语言学学术会议(CNCCL-2009).山东, 482-488.
余慧佳,刘奕群,张敏,茹立云,马少平. 2007.基于大规模日志分析的网络搜索引擎用户行为研究.中文信息学报, 21(1), 109-114.
张磊,李亚楠,王斌,李鹏,蒋在帆. 2008.网页搜索引擎查询日志的session划分研究.第四届全国信息检索与内容安全学术会议(NCIRCS2008).北京, 335-345.
中国互联网络信息中心(CNNIC). 1997.中国互联网络发展状况统计报告. http://www.cnnic.net.cn/download/2003/10/13/93603.pdf.
中国互联网络信息中心(CNNIC). 2009. 2009年中国搜索引擎用户行为研究报告. http://www.cnnic.net.cn/uploadfiles/doc/2009/9/21/104148.doc.
中国互联网络信息中心(CNNIC). 2010.第25次中国互联网络发展状况统计报告. http://www.cnnic.net.cn/uploadfiles/pdf/2010/1/15/101600.pdf.
Agichtein E, Brill E, Dumais S. 2006a. Improving Web search ranking by incorporating user behavior information. Proceedings of the 29th Annual international ACM SIGIR
Conference on Research and Development in information Retrieval (Seattle, Washington, USA, August 06 - 11, 2006). SIGIR '06. ACM Press, New York, NY, 19-26.
Agichtein E, Brill E, Dumais S, Ragno R. 2006b. Learning user interaction models for predicting Web search result preferences. Proceedings of the 29th Annual international ACM SIGIR Conference on Research and Development in information
Retrieval (Seattle, Washington, USA, August 06 - 11, 2006). SIGIR '06. ACM Press, New York, NY, 3-10.
Agrawal R, Halverson A, Kenthapadi K, Mishra N, Tsaparas P. 2009. Generating labels from clicks. Proceedings of the 2nd ACM international Conference on Web Search and Data Mining (Barcelona, Spain, February 09 - 12, 2009). R. Baeza-Yates, P.
Boldi, B. Ribeiro-Neto, and B. B. Cambazoglu, Eds. WSDM '09. ACM Press, New York, NY, 172-181.
Amitay E, Carmel D, Lempel R, Soffer A. 2004. Scaling IR-system evaluation using term relevance sets. Proceedings of the 27th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Sheffield, United Kingdom, July 25 - 29, 2004). SIGIR '04. ACM Press, New York, NY, 10-17.
Baeza-Yates R, Hurtado C, Mendoza M, Dupret G. 2005. Modeling user search behavior. Proceedings of the 3rd Latin American Web Congress. LA-WEB. IEEE Computer Society, Washington, DC, 242.
Baeza-Yates R, Ribeiro-Neto B. 1999. Modern information retrieval. Addison-Wesley (ACM Press series).
Baeza-Yates R, Tiberi A. 2007. Extracting semantic relations from query logs. Proceedings of the 13th ACM SIGKDD international Conference on Knowledge
Discovery and Data Mining (San Jose, California, USA, August 12 - 15, 2007). KDD '07. ACM Press, New York, NY, 76-85.
Bar-Yossef Z, Berg A, Chien S, Fakcharoenphol J, Weitz D. 2000. Approximating aggregate queries about Web pages via random walks. Proceedings of the 26th international Conference on Very Large Data Bases (September 10 - 14, 2000). Very
Large Data Bases. Morgan Kaufmann Publishers, San Francisco, CA, 535-544.
Bar-Yossef Z, Gurevich M. 2006. Random sampling from a search engine's index. Proceedings of the 15th international Conference on World Wide Web (Edinburgh, Scotland, May 23 - 26, 2006). WWW '06. ACM Press, New York, NY, 367-376.
Battelle J. 2005. John Battelle's searchblog. Retrieved from http://battellemedia.com/archives/001889.php.
Bharat K, Broder A. 1998. A technique for measuring the relative size and overlap of public Web search engines. Proceedings of the 7th international Conference on
World Wide Web 7 (Brisbane, Australia). P. H. Enslow and A. Ellis, Eds. Elsevier Science Publishers B. V., Amsterdam, The Netherlands, 379-388.
Bilenko M, White R. 2008. Mining the search trails of surfing crowds: identifying relevant websites from user activity. Proceeding of the 17th international Conference on World Wide Web (Beijing, China, April 21 - 25, 2008). WWW '08. ACM Press, New York, NY, 51-60.
Boldi P, Bonchi F, Castillo C, Vigna S. 2009. From "Dango" to "Japanese Cakes": Query Reformulation Models and Patterns. Proceedings of the 2009 IEEE/WIC/ACM international Joint Conference on Web intelligence and intelligent Agent Technology - Volume 01 (September 15 - 18, 2009). Web Intelligence & Intelligent Agent. IEEE Computer Society, Washington, DC, 183-190.
Bradlow E, Schmittlein D. 2000. The little engines that could: modeling the performance of World Wide Web search engines. Marketing Science, 19:43–62.
Brin S, Page L. 1998. The anatomy of a large-scale hypertextual Web search engine. Proceedings of the 7th international Conference on World Wide Web 7 (Brisbane, Australia). P. H. Enslow and A. Ellis, Eds. Elsevier Science Publishers B. V., Amsterdam, The Netherlands, 107-117.
Broder A, Kumar R, Maghoul F, Raghavan P, Rajagopalan S, Stata R, Tomkins A, Wiener J. 2000. Graph structure in the Web. Comput Netw:309-320.
Broder A. 2002. A taxonomy of Web search. SIGIR Forum 36, 2 (Sep. 2002), 3-10. Buckley C, Dimmick D, Soboroff I, Voorhees E. 2007. Bias and the limits of pooling for large collections. Inf. Retr. 10, 6, 491-508.
Buckley C, Voorhees E. 2004. Retrieval evaluation with incomplete information, Proceedings of the 30th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Amsterdam, The Netherlands, July 23 - 27, 2007). SIGIR '07. ACM Press, New York, NY, 63-70.
Büttcher S, Clarke C, Yeung P, Soboroff I. 2007. Reliable information retrieval evaluation with incomplete and biased judgments. Proceedings of the 30th Annual international ACM SIGIR Conference on Research and Development in informationRetrieval (Amsterdam, The Netherlands, July 23 - 27, 2007). SIGIR '07. ACM Press, New York, NY, 63-70.
Carletta J. 1996. Assessing agreement on classification tasks: the kappa statistic. Computational Linguistics 22(2):249-254.
Chowdhury A, Soboroff I. 2002. Automatic evaluation of World Wide Web search services. Proceedings of the 25th Annual international ACM SIGIR Conference on
Research and Development in information Retrieval (Tampere, Finland, August 11 - 15, 2002). SIGIR '02. ACM Press, New York, NY, 421-422.
Cleverdon C, Mills J, Keen M. 1966. Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 1, Design; Part 1. Cockburn A, Jones S. 1996. Which way now? Analysing and easing inadequacies in
WWW navigation. International Journal of Human-Computer Studies, 45:105-129. Craswell M, Hawking D. 2004. Overview of the TREC 2003 Web track. In E. M.
Voorhees and Lori P. Buckland, eds. NIST Special Publication 500-261: TREC 2004. Washington: Department of Commerce and National Institute of Standards and Technology.
Craswell N, Zoeter O, Taylor M, Ramsey B. 2008. An experimental comparison of click position-bias models. Proceedings of the international Conference on Web Search and Web Data Mining (Palo Alto, California, USA, February 11 - 12, 2008). WSDM '08. ACM Press, New York, NY, 87-94.
Dobra A, Fienberg S. 2004. How large is the World Wide Web? M. Levene and A. Poulovassilis (eds), Web Dynamics: Adapting to Change in Content, Size, Topology and Use (Springer, Berlin/ Heidelberg) , 23-44.
Dou Z, Song R, Yuan X, Wen J. 2008. Are clickthrough data adequate for learning web search rankings?. Proceedings of the 18th ACM Conference on information and Knowledge Management (Hong Kong, China, November 02 - 06, 2009). CIKM '09. ACM Press, New York, NY, 1077-1086.
Downey D, Dumais S, Liebling D, Horvitz E. 2008. Understanding the relationship between searchers' queries and information goals. Proceeding of the 17th ACM Conference on information and Knowledge Management (Napa Valley, California, USA, October 26 - 30, 2008). CIKM '08. ACM Press, New York, NY, 449-458.
Fawcett T. 2006. An introduction to ROC analysis. Pattern Recogn. Lett. 27(8):861-874.
French B. 2009. Google semantic upgrade enhances long tail query results. Retrieved from http://blogsite.com/public/item/230059.
Fuxman A, Tsaparas P, Achan K, Agrawal R. 2008. Using the wisdom of the crowds for keyword generation. Proceeding of the 17th international Conference on World WideWeb (Beijing, China, April 21 - 25, 2008). WWW '08. ACM Press, New York, NY, 61-70.
Gao J, Yuan W, Li X, Deng K, Nie J. 2009. Smoothing clickthrough data for web search ranking. Proceedings of the 32nd international ACM SIGIR Conference on Research and Development in information Retrieval (Boston, MA, USA, July 19 - 23, 2009). SIGIR '09. ACM Press, New York, NY, 355-362.
Glassman S. 1994. A caching relay for the World Wide Web. Comput. Netw. ISDN Syst. 27(2):165-173.
Google. 2008. Official Google blog: We knew the web was big. Retrieved from http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html.
Gulli A, Signorini A. 2005. The indexable Web is more than 11.5 billion pages. In Special interest Tracks and Posters of the 14th international Conference on World Wide Web (Chiba, Japan, May 10 - 14, 2005). WWW '05. ACM Press, New York, NY, 902-903. Guo F, Liu C, Wang Y. 2009. Efficient multiple-click models in web search. Proceedings of the 2nd ACM international Conference on Web Search and Data Mining (Barcelona, Spain, February 09 - 12, 2009). R. Baeza-Yates, P. Boldi, B. Ribeiro-Neto, and B. B. Cambazoglu, Eds. WSDM '09. ACM Press, New York, NY, 124-131.
Han I, Lee S, Lee S. 2007. Graph structure of the Korea Web. Proceedings of the 12th DASFAA, 930–935.
Hawking D, Craswell N, Thistlewaite P, Harman D. 1999. Results and challenges in Web search evaluation. Proceedings of the 8th international World Wide Web Conference. North-Holland Publishing Co., Amsterdam, The Netherlands, 1321-1330.
Hawking D, Craswell N. 2002. Overview of the TREC-2002 web track. In The 11th Text Retrieval Conference (TREC-2002), volume 11. National Institute of Standards and Technology, NIST.
Hawking D, Craswell N. 2003. Overview of the TREC-2003 Web track. In NIST Special Publication: SP 500-255, The 12th Text Retrieval Conference (TREC 2003). Hedger J. 2005. Google takes backhanded bow out of size war with Yahoo. Retrieved from http://www.searchenginejournal.com/?p=2277.
Henzinger M, Heydon A, Mitzenmacher M, Najork M. 2000. On near-uniform URL sampling. Proceedings of the 9th international World Wide Web Conference on Computer Networks : the international Journal of Computer and Telecommunications Netowrking (Amsterdam, The Netherlands). North-Holland Publishing Co., Amsterdam, The Netherlands, 295-308.
Henzinger M, Motwani R, Silverstein C. 2002. Challenges in web search engines. SIGIR Forum 36, 2, 11-22.
Huberman B, Pirolli P, Pitkow J, Lukose R. 1998. Strong regularities in World Wide Web surfing. Science, 280(3):95-97.
Jarvelin K, Kekalainen J. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transaction on Information Systems (TOIS), 20(4):422-446.
Joachims T, Freitag D, Mitchell T. 1997. WebWatcher: a tour guide for the World Wide Web. Proceedings of the 15th International Joint Conference on Artificial Intelligence. IJCAI’97. Morgan Kaufmann, 1, 770– 777.
Joachims T, Granka L, Pan B, Hembrooke H, Gay G. 2005. Accurately interpreting clickthrough data as implicit feedback. Proceedings of the 28th Annual international
ACM SIGIR Conference on Research and Development in information Retrieval (Salvador, Brazil, August 15 - 19, 2005). SIGIR '05. ACM Press, New York, NY, 154-161.
Joachims T, Granka L, Pan B, Hembrooke H, Radlinski F, Gay G. 2007. Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search. ACM Transactions on Information Systems, 25(2):7.
Joachims T. 2002a. Evaluating retrieval performance using clickthrough data. Proceedings of the SIGIR Workshop on Mathematical/FormalMethods in Information Retrieval.
Joachims T. 2002b. Optimizing search engines using clickthrough data. Proceedings of the 8th ACM SIGKDD international Conference on Knowledge Discovery and Data Mining (Edmonton, Alberta, Canada, July 23 - 26, 2002). KDD '02. ACM Press, New York, NY, 133-142.
Jolliffe I. 1986. Principal Component Analysis. Springer-Verlag, New York. Kammenhuber N, Luxenburger J, Feldmann A, Weikum G. 2006. Web search clickstreams. Proceedings of the 6th ACM SIGCOMM Conference on internet Measurement. IMC '06. ACM Press, New York, NY, 245-250.
Kent A, Berry M, Leuhrs F, Perry, J. 1955. Machine literature searching VIII. Operational criteria for designing information retrieval systems. American Documentation, 6(2): 93-101.
Lawrence S, Giles C. 1998. Searching the World Wide Web. Science, 5360(280):98.
Lawrence S, Giles C. 1999. Accessibility of information on the Web. Nature, 400:107–109.
Lee U, Liu Z, Cho J. 2005. Automatic identification of user goals in Web search. Proceedings of the 14th international Conference on World Wide Web (Chiba, Japan, May 10 - 14, 2005). WWW '05. ACM Press, New York, NY, 391-400.
Li Y, Wang B, Xu S, Li J, Li P. 2009. QueryTrans: finding similar queries based on query trace graph. Proceedings of the 2009 IEEE/WIC/ACM International Conference on Web Intelligence. WI 2009. Universitàdegli Studi di Milano Bicocca, Milano, 260-263.
Liu J. 2001. Monte carlo strategies in scientific computing. Springer.
Liu Y, Gao B, Liu T, Zhang Y, Ma Z, He S, Li H. 2008. BrowseRank: letting web users vote for page importance. Proceedings of the 31st Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Singapore, Singapore, July 20 - 24, 2008). SIGIR '08. ACM Press, New York, NY, 451-458.
Liu Y, Jin Y, Zhang M, Ma S, Ru L. 2009. User browsing graph: structure, evolution and application. Late breaking result session in 2nd ACM International Conference on Web Search and Data Mining (WSDM 2009).
Liu Y, Zhang M, Ru L, Ma S. 2006. Automatic query type identification based on click through information. In H.T. Ng et al. (Eds.): AIRS 2006, LNCS 4182, pp. 593–600.
Lyman P, Hal R, Varian, 2003. How much information [EB/OL]. Retrieved from http://www.sims.berkeley.edu/how-much-info-2003.
Martindale C, Konopka A. 1996. Oligonucleotide frequencies in DNA follow a Yule distribution. Computers and Chemistry, 20(1):35-38.
Mei Q, Church K. 2008. Entropy of search logs: how hard is search? with personalization? with backoff?. Proceedings of the international Conference on Web Search and Web Data Mining (Palo Alto, California, USA, February 11 - 12, 2008). WSDM '08. ACM Press, New York, NY, 45-54.
Nuray R, Can F. 2003. Automatic ranking of retrieval systems in imperfect environments. Proceedings of the 26th Annual international ACM SIGIR Conference on Research and Development in informaion Retrieval (Toronto, Canada, July 28 - August 01, 2003). SIGIR '03. ACM Press, New York, NY, 379-380.
Pearson K. 1901. On lines and planes of closest fit to system of points in space. Philosophical Magazine, B(2):559-572.
Radlinski F, Kurup M, Joachims T. 2007. Active exploration for learning rankings from clickthrough data. Proceedings of the 13th ACM SIGKDD international Conference on Knowledge Discovery and Data Mining (San Jose, California, USA, August 12 - 15, 2007). KDD '07. ACM Press, New York, NY, 570-579.
Rose D, Levinson D. 2004. Understanding user goals in web search. Proceedings of the 13th international Conference on World Wide Web (New York, NY, USA, May 17 - 20, 2004). WWW '04. ACM Press, New York, NY, 13-19.
Sadagopan N, Li J. 2008. Characterizing typical and atypical user sessions in clickstreams. Proceeding of the 17th international Conference on World Wide Web (Beijing, China, April 21 - 25, 2008). WWW '08. ACM Press, New York, NY, 885-894.
Saracevic T. 1995. Evaluation of evaluation in information retrieval. Proceedings of the 18th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Seattle, Washington, United States, July 09 - 13, 1995). E. A.
Fox, P. Ingwersen, and R. Fidel, Eds. SIGIR '95. ACM Press, New York, NY, 138-146.
Selberg E. 1999. Towards comprehensive Web search. Doctoral Thesis. UMI Order Number: AAI9936480, University of Washington.
Shannon C. 1948. A mathematical theory of communication. Bell System Technical Journal, 27:379-423.
Silverstein C, Marais H, Henzinger M, Moricz M. 1999. Analysis of a very large web search engine query log. SIGIR Forum 33, 1, 6-12.
Soboroff I, Nicholas C, Cahan P. 2001. Ranking retrieval systems without relevance judgments. Proceedings of the 24th Annual international ACM SIGIR Conference on
Research and Development in information Retrieval (New Orleans, Louisiana, United States). SIGIR '01. ACM Press, New York, NY, 66-73.
Soboroff I, Voorhees E, Craswell N. 2003. Summary of the SIGIR 2003 workshop on defining evaluation methodologies for terabyte-scale test collections. SIGIR Forum 37, 2, 55-58.
Soboroff I. 2004. On evaluating web search with very few relevant documents. Proceedings of the 27th Annual international ACM SIGIR Conference on Research and Development in information Retrieval (Sheffield, U.K., July, 2004). SIGIR '04. ACM Press, New York, NY, 530-531.
Sullivan D. 2005. Search engine sizes [EB/OL]. Retrieved from search engine watch Web site http://searchenginewatch.com/reports/article.php/2156481.
Svore K, Wu Q, Burges C, Raman A. 2007. Improving Web spam classification using rank-time features. Proceedings of the 3rd international Workshop on Adversarial information Retrieval on the Web (Banff, Alberta, Canada, May 08 - 08, 2007). AIRWeb '07, vol. 215. ACM Press, New York, NY, 9-16
Tan P, Kumar V. 2000. Modeling of Web robot navigational patterns. In Procings of ACM WebKDD Workshop. ACM Press, New York, NY.
Tan P, Kumar V. 2002. Discovery of Web robot sessions based on their navigational patterns. Data Mining and Knowledge Discovery, 6(1):9-35.
Tauscher L, Greenberg S. 1997. How people revisit web pages: Empirical findings and implications for the design of history systems. International Journal of Human-Computer Studies, 47:97-137.
Von-Neumann J. 1951. Various technique used in connection with random digits. Washington: NBS Appl. Math, 12: 36-38.
Voorhees E. 2002. The philosophy of information retrieval evaluation. In Revised Papers
From the 2nd Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language information Retrieval Systems (September 03 - 04, 2001). C. Peters, M. Braschler, J. Gonzalo, and M. Kluck, Eds. Lecture Notes In Computer Science, vol. 2406. Springer-Verlag, London, 355-370.
Yule G. 1944. Statistical study of literary vocabulary. Cambridge, Cambridge University Press.
Zipf G. 1949. Human behavior and the principle of least effort. Addison-Wesley Press.
百度百科. 2009.搜索引擎. http://baike.baidu.com/view/2647196.htm.
刘奕群,岑荣伟,张敏,茹立云,马少平. 2007.基于用户行为分析的搜索引擎自动性能评价.软件学报, 19(11), 3023-3032.
刘奕群. 2010.搜索引擎技术基础.
薛宇飞,刘奕群,张敏,马少平,茹立云. 2009.基于用户浏览图的网页质量评估方法的比较分析.全国第十届计算语言学学术会议(CNCCL-2009).山东, 482-488.
余慧佳,刘奕群,张敏,茹立云,马少平. 2007.基于大规模日志分析的网络搜索引擎用户行为研究.中文信息学报, 21(1), 109-114.
张磊,李亚楠,王斌,李鹏,蒋在帆. 2008.网页搜索引擎查询日志的session划分研究.第四届全国信息检索与内容安全学术会议(NCIRCS2008).北京, 335-345.
中国互联网络信息中心(CNNIC). 1997.中国互联网络发展状况统计报告. http://www.cnnic.net.cn/download/2003/10/13/93603.pdf.
中国互联网络信息中心(CNNIC). 2009. 2009年中国搜索引擎用户行为研究报告. http://www.cnnic.net.cn/uploadfiles/doc/2009/9/21/104148.doc.
中国互联网络信息中心(CNNIC). 2010.第25次中国互联网络发展状况统计报告. http://www.cnnic.net.cn/uploadfiles/pdf/2010/1/15/101600.pdf.
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |