爬虫系统中标签删除功能的设计及优化
|
摘要
在用爬虫爬取到大型商品网站的大规模网页数据集后,要将网页数据集作进一步筛选以得到目标数据集,筛选之前要做的一项准备工作就是删除网页中多余的标签.为此,用递归算法的思想给出了标签删除的算法,提出了标签删除功能的软件设计思想,对设计进行了2次设计改进及性能优化,最终采用了1个缓冲区维系线程1个标签删除线程的双线程设计思想.实验表明,优化后的标签删除功能在单机上每1000个网页的平均处理时间只需19.7 s,处理20万个网页只需1.1小时.
After crawling to obtain a data set of large web pages on a large commodity site,the data set is screened to further get the target data set.Before screening,preparation must to be done is to delete the redundant tags in the web pages.Therefore,the algorithm of deletion tag is given with the idea of a recursive algorithm.The design idea of tag deletion function is put forward.2 time design improvements are carried out to optimize the performance.Finally,the design idea of dual thread is adopted.The dual threads are 1 maintain buffer thread and 1 tag deletion thread.In single computer environment,experiments show that the optimized tag deletion function only takes 19.7 seconds for each 1000 pages,and only 1.1 hours for 200 000 web pages.
After crawling to obtain a data set of large web pages on a large commodity site,the data set is screened to further get the target data set.Before screening,preparation must to be done is to delete the redundant tags in the web pages.Therefore,the algorithm of deletion tag is given with the idea of a recursive algorithm.The design idea of tag deletion function is put forward.2 time design improvements are carried out to optimize the performance.Finally,the design idea of dual thread is adopted.The dual threads are 1 maintain buffer thread and 1 tag deletion thread.In single computer environment,experiments show that the optimized tag deletion function only takes 19.7 seconds for each 1000 pages,and only 1.1 hours for 200 000 web pages.
引文
1黄仁,王良伟.基于主题相关概念和网页分块的主题爬虫研究.计算机应用研究,2013, 30(8):2377-2380, 2409.[doi:10.3969/j.issn.1001-3695.2013.08.034]
2孟繁疆,姬祥,袁琦,等.农产品价格主题搜索引擎的研究与实现.东北农业大学学报,2016, 47(9):64-71.[doi:10.3969/j.issn. 1005-9369.2016.09.009]
3吴洁明,冀单单,韩云辉.基于Web的DCI垂直搜索引擎的研究与设计.计算机工程与设计,2013, 34(4):1481-1487.[doi:10.3969/j.issn. 1000-7024.2013.04.066]
4 Sahami M, Heilman T D. A web-based kernel function for measuring the similarity of short text snippets. Proceedings of the 15th International Conference on World Wide Web.Edinburgh, Scotland. 2006. 377-386.
5 Ilbahar E, Cebi S. Classification of design parameters for ecommerce websites:A novel fuzzy Kano approach.Telematics and Informatics, 2017, 34(8):1814-1825.[doi:10.1016/j.tele.2017.09.004]
6 Deng ZY, Zhang J, He TQ. Automatic combination technology of fuzzy CPN for OWL-S web services in supercomputing cloud platform. International Journal of Pattern Recognition and Artificial Intelligence, 2017, 31(7):1759010.[doi:10.1142/S0218001417590108]
2孟繁疆,姬祥,袁琦,等.农产品价格主题搜索引擎的研究与实现.东北农业大学学报,2016, 47(9):64-71.[doi:10.3969/j.issn. 1005-9369.2016.09.009]
3吴洁明,冀单单,韩云辉.基于Web的DCI垂直搜索引擎的研究与设计.计算机工程与设计,2013, 34(4):1481-1487.[doi:10.3969/j.issn. 1000-7024.2013.04.066]
4 Sahami M, Heilman T D. A web-based kernel function for measuring the similarity of short text snippets. Proceedings of the 15th International Conference on World Wide Web.Edinburgh, Scotland. 2006. 377-386.
5 Ilbahar E, Cebi S. Classification of design parameters for ecommerce websites:A novel fuzzy Kano approach.Telematics and Informatics, 2017, 34(8):1814-1825.[doi:10.1016/j.tele.2017.09.004]
6 Deng ZY, Zhang J, He TQ. Automatic combination technology of fuzzy CPN for OWL-S web services in supercomputing cloud platform. International Journal of Pattern Recognition and Artificial Intelligence, 2017, 31(7):1759010.[doi:10.1142/S0218001417590108]