结合深度学习与相关反馈的遥感图像检索
|
摘要
目的针对基于内容的图像检索存在低层视觉特征与用户对图像理解的高层语义不一致、图像检索的精度较低以及传统的分类方法准确度低等问题,提出一种基于卷积神经网络和相关反馈支持向量机的遥感图像检索方法。方法通过对比度受限直方图均衡化算法对遥感图像进行预处理,限制遥感图像噪声的放大,采用自学习能力良好的卷积神经网络对遥感图像进行多层神经网络的监督学习提取丰富的图像特征,并将支持向量机作为基分类器,根据测试样本数据到分类超平面的距离进行排序得到检索结果,最后采用相关反馈策略对检索结果进行重新调整。结果在UC Merced Land-Use遥感图像数据集上进行图像检索实验,在mAP(mean average precision)精度指标上,当检索返回图像数为100时,本文方法比LSH(locality sensitive Hashing)方法提高了29. 4%,比DSH(density sensitive Hashing)方法提高了37. 2%,比EMR(efficient manifold ranking)方法提高了68. 8%,比未添加反馈和训练集筛选的SVM(support vector machine)方法提高了3. 5%,对于平均检索速度,本文方法比对比方法中m AP精度最高的方法提高了4倍,针对复杂的遥感图像数据,本文方法的检索效果较其他方法表现出色。结论本文提出了一种以距离评价标准为核心的反馈策略,以提高检索精度,并采用多距离结合的Top-k排序方法合理筛选训练集,以提高检索速度,本文方法可以广泛应用于人脸识别和目标跟踪等领域,对提升检索性能具有重要意义。
Objective The traditional content-based image retrieval method can only retrieve and analyze the features of low layers,such as color,texture,and shape,which exist in an image. Therefore,a low level of visual features exist in such method,which is inconsistent with the high-level semantic meaning of the user's understanding of the image. This inconsistency results in the"semantic gap"phenomenon,which leads to the low accuracy of image retrieval. Moreover,the traditional method cannot meet the user's demand for high-accuracy retrieval,while remote sensing images have rich information,complex content,and high dimensionality. Analyzing only the low-level features greatly reduces the accuracy of image retrieval. Therefore,selecting an appropriate image feature extraction is the key step to achieving high-accuracy retrieval.At the same time,the traditional classification method is insignificantly accurate in image classification. Determining how to select a high-accuracy image classification method is also essential. A remote sensing image retrieval method based on convolutional neural network and relevance feedback support vector machine( SVM) is proposed in this research. Method The proposed method can preprocess remote sensing images by contrasting the limited histogram equalization algorithm,limiting the noise magnification of the remote sensing images,and avoiding the influence of noise interference on the retrieval precision. On the basis of the Goog Le Net convolutional neural network model with good self-learning capability,a multilayer neural network of remote sensing images is supervised and studied,the rich features of the remote sensing images are extracted,and the problem of"semantic gap"in the content-based image retrieval method is solved. The original dataset is divided into training and test sets,and selecting the training set reasonably is the basis for the best classification. If too many samples exist in other categories in the training set,then determining the hyperplane classification will be greatly affected. A multi-distance combined top-k sorting method is proposed to rationally screen the original training set. The image closest to the query one will be used as the training set. On the one hand,the method saves considerable time for subsequent determination of the optimal hyperplane. On the other hand,most dissimilar images are filtered out to avoid the influences of more dissimilar images on the classification results. The SVM is used as the basic classifier,and the optimal hyperplane is trained according to the training set samples. The retrieval results are sorted according to the distance between the test sample data and the classified hyperplane. A feedback of the distance evaluation standard is proposed to update the retrieval results with the distance evaluation standard. The strategy readjusts the experimental results. The method uses a small-sample marking method to mark the counterexample images to avoid too many markers and lose the meaning of the retrieval. In addition,the optimal hyperplane of the SVM does not need to be retrained to avoid unnecessary time waste.Only multiple iterations are used to update the retrieval results,and one feedback can achieve the desired results. Result The image retrieval experiments are performed on the remote sensing image dataset of UC Merced Land-use dataset. Experimental results show that the mean average precision( m AP) of the proposed method is increased by 29.4% compared with that of the locality-sensitive Hashing method,is 37.2% higher than that of the density-sensitive Hashing method,which is68.8% higher than the efficient manifold ranking,and is 3.5% higher than that of the SVM method without feedback and training set screening. The number of retrieved images is 100. For the average retrieval speed,this method is four times higher than the method with the highest m AP accuracy in the comparison method. For the average recall rate and the average precision rate,this method is also higher than the comparison method,which shows that this method can improve not only the retrieval accuracy but also the retrieval speed. For complex remote sensing image data,the retrieval effect of this method is better than those of other methods. Conclusion A new feedback strategy is proposed in this study to improve the retrieval accuracy. Small-sample markers are used for the poor retrieval results and the distance evaluation standard as the core to perform many iterations. One time feedback can achieve good retrieval results. In terms of speed increase,this study proposes a multi-distance combined Top-k sorting method,which reduces the time of SVMs to train the optimal hyperplane by rationally selecting the training sample set and then improving the retrieval speed. This method can be widely applied to face recognition,target tracking,and other fields,and it is significant to improving retrieval performance.
Objective The traditional content-based image retrieval method can only retrieve and analyze the features of low layers,such as color,texture,and shape,which exist in an image. Therefore,a low level of visual features exist in such method,which is inconsistent with the high-level semantic meaning of the user's understanding of the image. This inconsistency results in the"semantic gap"phenomenon,which leads to the low accuracy of image retrieval. Moreover,the traditional method cannot meet the user's demand for high-accuracy retrieval,while remote sensing images have rich information,complex content,and high dimensionality. Analyzing only the low-level features greatly reduces the accuracy of image retrieval. Therefore,selecting an appropriate image feature extraction is the key step to achieving high-accuracy retrieval.At the same time,the traditional classification method is insignificantly accurate in image classification. Determining how to select a high-accuracy image classification method is also essential. A remote sensing image retrieval method based on convolutional neural network and relevance feedback support vector machine( SVM) is proposed in this research. Method The proposed method can preprocess remote sensing images by contrasting the limited histogram equalization algorithm,limiting the noise magnification of the remote sensing images,and avoiding the influence of noise interference on the retrieval precision. On the basis of the Goog Le Net convolutional neural network model with good self-learning capability,a multilayer neural network of remote sensing images is supervised and studied,the rich features of the remote sensing images are extracted,and the problem of"semantic gap"in the content-based image retrieval method is solved. The original dataset is divided into training and test sets,and selecting the training set reasonably is the basis for the best classification. If too many samples exist in other categories in the training set,then determining the hyperplane classification will be greatly affected. A multi-distance combined top-k sorting method is proposed to rationally screen the original training set. The image closest to the query one will be used as the training set. On the one hand,the method saves considerable time for subsequent determination of the optimal hyperplane. On the other hand,most dissimilar images are filtered out to avoid the influences of more dissimilar images on the classification results. The SVM is used as the basic classifier,and the optimal hyperplane is trained according to the training set samples. The retrieval results are sorted according to the distance between the test sample data and the classified hyperplane. A feedback of the distance evaluation standard is proposed to update the retrieval results with the distance evaluation standard. The strategy readjusts the experimental results. The method uses a small-sample marking method to mark the counterexample images to avoid too many markers and lose the meaning of the retrieval. In addition,the optimal hyperplane of the SVM does not need to be retrained to avoid unnecessary time waste.Only multiple iterations are used to update the retrieval results,and one feedback can achieve the desired results. Result The image retrieval experiments are performed on the remote sensing image dataset of UC Merced Land-use dataset. Experimental results show that the mean average precision( m AP) of the proposed method is increased by 29.4% compared with that of the locality-sensitive Hashing method,is 37.2% higher than that of the density-sensitive Hashing method,which is68.8% higher than the efficient manifold ranking,and is 3.5% higher than that of the SVM method without feedback and training set screening. The number of retrieved images is 100. For the average retrieval speed,this method is four times higher than the method with the highest m AP accuracy in the comparison method. For the average recall rate and the average precision rate,this method is also higher than the comparison method,which shows that this method can improve not only the retrieval accuracy but also the retrieval speed. For complex remote sensing image data,the retrieval effect of this method is better than those of other methods. Conclusion A new feedback strategy is proposed in this study to improve the retrieval accuracy. Small-sample markers are used for the poor retrieval results and the distance evaluation standard as the core to perform many iterations. One time feedback can achieve good retrieval results. In terms of speed increase,this study proposes a multi-distance combined Top-k sorting method,which reduces the time of SVMs to train the optimal hyperplane by rationally selecting the training sample set and then improving the retrieval speed. This method can be widely applied to face recognition,target tracking,and other fields,and it is significant to improving retrieval performance.
引文
[1]Hang Y,Yang Y B,Chen Z Q.A survey of content-based image retrieval[J].Application Research of Computers,2002,(9):9-13,29.[杭燕,杨育彬,陈兆乾.基于内容的图像检索综述[J].计算机应用研究,2002,(9):9-13,29.][DOI:10.3969/j.issn.1001-3695.2002.09.003]
[2]Liu L,Kuang G Y.Overview of image textural feature extraction methods[J].Journal of Image and Graphics,2009,14(4):622-635.[刘丽,匡纲要.图像纹理特征提取方法综述[J].中国图象图形学报,2009,14(4):622-635.][DOI:10.11834/jig.20090409]
[3]Tanase M,Veltkamp R C.Part-based shape retrieval with relevance feedback[C]//Proceedings of 2005 IEEE International Conference on Multimedia and Expo.Amsterdam,Netherlands:IEEE,2005:936-939.[DOI:10.1109/ICME.2005.1521578]
[4]Zhang H Q,Liu X Y,Yang S,et al.Retrieval ofremote sensing images based on semisupervised deep learning[J].Journal of Remote Sensing,2017,21(3):406-414.[张洪群,刘雪莹,杨森,等.深度学习的半监督遥感图像检索[J].遥感学报,2017,21(3):406-414.][DOI:10.11834/jrs.20176105]
[5]Liu B,Zhang H.Image retrieval algorithm based on convolutional neural network and manifold ranking[J].Journal of Computer Applications,2016,36(2):531-534.[刘兵,张鸿.基于卷积神经网络和流形排序的图像检索算法[J].计算机应用,2016,36(2):531-534.][DOI:10.11772/j.issn.1001-9081.2016.02.0531]
[6]Gong Z T,Chen G X,Ren X L,et al.An image retrieval method based on a convolutional neural network and hash coding[J].CAAI transactions on intelligent Systems,2016,11(3):391-400.[龚震霆,陈光喜,任夏荔,等.基于卷积神经网络和哈希编码的图像检索方法[J].智能系统学报,2016,11(3):391-400.][DOI:10.11992/tis.201603028]
[7]Li D M,Gai M Y,Li C R,et al.Research on adaptive optics image denoising algorithm based on the wavelet-based Contourlet transform[J].Laser&Optoelectronics Progress,2015,52(11):#111001.[李东明,盖梦野,李超然,等.基于小波域的Contourlet变换法的自适应光学图像去噪算法研究[J].激光与光电子学进展,2015,52(11):#111001.][DOI:10.3788/LOP52.111001]
[8]Xu X,Chen Q,Sun H J,et al.Fast Retinex-based color image enhancement[J].Computer Engineering and Applications,2010,46(5):4-6,72.[许欣,陈强,孙怀江,等.快速Retinex彩色图像增强[J].计算机工程与应用,2010,46(5):4-6,72.][DOI:10.3778/j.issn.1002-8331.2010.05.002]
[9]Li Y,Zhang Y F,Zhang Q,et al.Infrared image contrast enhancement based on haze remove method[J].Chinese Journal of Lasers,2015,42(1):#0113004.[李毅,张云峰,张强,等.基于去雾模型的红外图像对比度增强[J].中国激光,2015,42(1):#0113004.]
[10]Hai Y,Li L,Gu J.Imageenhancement based on contrast limited adaptive histogram equalization for 3D images of stereoscopic endoscopy[C]//Proceedings of 2015 IEEE International Conference on Information and Automation.Lijiang,China:IEEE,2015:668-672.[DOI:10.1109/ICInfA.2015.7279370]
[11]Szegedy C,Liu W,Jia Y Q,et al.Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA,USA:IEEE,2015:1-9.[DOI:10.1109/CVPR.2015.7298594]
[12]Melgani F,Bruzzone L.Classification of hyperspectral remote sensing images with support vector machines[J].IEEE Transactions on Geoscience and Remote Sensing,2004,42(8):1778-1790.[DOI:10.1109/TGRS.2004.831865]
[13]Chen Y S,Zhao X,Lin Z H.Optimizing subspaceSVM ensemble for hyperspectral imagery classification[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,2014,7(4):1295-1305.[DOI:10.1109/JSTARS.2014.2307356]
[14]Wang X J,Yang L L.Application of SVM relevance feedback algorithms in image retrieval[C]//2008 International Symposium on Information Science and Engineering.Shanghai,China:IEEE,2008:210-213.[DOI:10.1109/ISISE.2008.275]
[15]Fu R G,Li B,Gao Y H,et al.Content-based image retrieval based on CNN and SVM[C]//Proceedings of the 2nd IEEEInternational Conference on Computer and Communications.Chengdu,China:IEEE.2016:638-642.[DOI:10.1109/CompComm.2016.7924779]
[16]Bai J W,Zhao Z C.Novel relevance feedback method based on SVM in image retrieval[J].Software Guide,2010,9(10):49-51.[白婧文,赵志诚.一种新的基于SVM相关反馈的图像检索算法[J].软件导刊,2010,9(10):49-51.]
[17]Jiang N N,Qi M,Hao C Y.A SVM based relevance feedback algorithm for image retrieval[J].Computer Simulation,2009,26(1):219-221,330.[姜楠楠,齐敏,郝重阳.一种基于SVM的相关反馈图像检索算法[J].计算机仿真,2009,26(1):219-221,330.][DOI:10.3969/j.issn.1006-9348.2009.01.059]
[18]Feng G H.Research on large scale SVM classification based on boundary K-nearest[J].Computer Engineering and Applications,2009,45(23):15-17.[奉国和.边界K邻近大样本支持向量机分类[J].计算机工程与应用,2009,45(23):15-17.][DOI:10.3778/j.issn.1002-8331.2009.23.005]
[19]Ju H Y,Zhang J B,Li C F,et al.Automated Remote Sensing Image Classification Method Based on K-means and SVM[J].Application Research of Computers,2007,24(11):318-320.[居红云,张俊本,李朝峰,等.基于K-means与SVM结合的遥感图像全自动分类方法[J].计算机应用研究,2007,24(11):318-320.][DOI:10.3969/j.issn.1001-3695.2007.11.098]
[20]Meng X F,Ma Z M,Li X,et al.A Top-K query results ranking approach based on contextual preferences for web database[J].Chinese Journal of Computers,2014,37(9):1986-1998.[孟祥福,马宗民,李昕,等.基于上下文偏好的Web数据库查询结果Top-K排序方法[J].计算机学报,2014,37(9):1986-1998.][DOI:10.3724/SP.J.1016.2014.01986]
[21]Yang Y,Newsam S.Bag-of-visual-words and spatial extensions for land-use classification[C]//Proceedings of the 18th SIGSPA-TIAL International Conference on Advances in Geographic Information Systems.San Jose,California:ACM,2010:270-279.[DOI:10.1145/1869790.1869829]
[22]Chafik S,Daoudi I,Ouardi H E,et al.Locality sensitive hashing for content based image retrieval:a comparative experimental study[C]//Proceedings of 2014 International Conference on Next Generation Networks and Services.Casablanca,Morocco:IEEE,2014:38-43.[DOI:10.1109/NGNS.2014.6990224]
[23]Jin Z M,Li C,Lin Y,et al.Density sensitive hashing[J].IEEE Transactions on Cybernetics,2014,44(8):1362-1371.[DOI:10.1109/TCYB.2013.2283497]
[2]Liu L,Kuang G Y.Overview of image textural feature extraction methods[J].Journal of Image and Graphics,2009,14(4):622-635.[刘丽,匡纲要.图像纹理特征提取方法综述[J].中国图象图形学报,2009,14(4):622-635.][DOI:10.11834/jig.20090409]
[3]Tanase M,Veltkamp R C.Part-based shape retrieval with relevance feedback[C]//Proceedings of 2005 IEEE International Conference on Multimedia and Expo.Amsterdam,Netherlands:IEEE,2005:936-939.[DOI:10.1109/ICME.2005.1521578]
[4]Zhang H Q,Liu X Y,Yang S,et al.Retrieval ofremote sensing images based on semisupervised deep learning[J].Journal of Remote Sensing,2017,21(3):406-414.[张洪群,刘雪莹,杨森,等.深度学习的半监督遥感图像检索[J].遥感学报,2017,21(3):406-414.][DOI:10.11834/jrs.20176105]
[5]Liu B,Zhang H.Image retrieval algorithm based on convolutional neural network and manifold ranking[J].Journal of Computer Applications,2016,36(2):531-534.[刘兵,张鸿.基于卷积神经网络和流形排序的图像检索算法[J].计算机应用,2016,36(2):531-534.][DOI:10.11772/j.issn.1001-9081.2016.02.0531]
[6]Gong Z T,Chen G X,Ren X L,et al.An image retrieval method based on a convolutional neural network and hash coding[J].CAAI transactions on intelligent Systems,2016,11(3):391-400.[龚震霆,陈光喜,任夏荔,等.基于卷积神经网络和哈希编码的图像检索方法[J].智能系统学报,2016,11(3):391-400.][DOI:10.11992/tis.201603028]
[7]Li D M,Gai M Y,Li C R,et al.Research on adaptive optics image denoising algorithm based on the wavelet-based Contourlet transform[J].Laser&Optoelectronics Progress,2015,52(11):#111001.[李东明,盖梦野,李超然,等.基于小波域的Contourlet变换法的自适应光学图像去噪算法研究[J].激光与光电子学进展,2015,52(11):#111001.][DOI:10.3788/LOP52.111001]
[8]Xu X,Chen Q,Sun H J,et al.Fast Retinex-based color image enhancement[J].Computer Engineering and Applications,2010,46(5):4-6,72.[许欣,陈强,孙怀江,等.快速Retinex彩色图像增强[J].计算机工程与应用,2010,46(5):4-6,72.][DOI:10.3778/j.issn.1002-8331.2010.05.002]
[9]Li Y,Zhang Y F,Zhang Q,et al.Infrared image contrast enhancement based on haze remove method[J].Chinese Journal of Lasers,2015,42(1):#0113004.[李毅,张云峰,张强,等.基于去雾模型的红外图像对比度增强[J].中国激光,2015,42(1):#0113004.]
[10]Hai Y,Li L,Gu J.Imageenhancement based on contrast limited adaptive histogram equalization for 3D images of stereoscopic endoscopy[C]//Proceedings of 2015 IEEE International Conference on Information and Automation.Lijiang,China:IEEE,2015:668-672.[DOI:10.1109/ICInfA.2015.7279370]
[11]Szegedy C,Liu W,Jia Y Q,et al.Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA,USA:IEEE,2015:1-9.[DOI:10.1109/CVPR.2015.7298594]
[12]Melgani F,Bruzzone L.Classification of hyperspectral remote sensing images with support vector machines[J].IEEE Transactions on Geoscience and Remote Sensing,2004,42(8):1778-1790.[DOI:10.1109/TGRS.2004.831865]
[13]Chen Y S,Zhao X,Lin Z H.Optimizing subspaceSVM ensemble for hyperspectral imagery classification[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,2014,7(4):1295-1305.[DOI:10.1109/JSTARS.2014.2307356]
[14]Wang X J,Yang L L.Application of SVM relevance feedback algorithms in image retrieval[C]//2008 International Symposium on Information Science and Engineering.Shanghai,China:IEEE,2008:210-213.[DOI:10.1109/ISISE.2008.275]
[15]Fu R G,Li B,Gao Y H,et al.Content-based image retrieval based on CNN and SVM[C]//Proceedings of the 2nd IEEEInternational Conference on Computer and Communications.Chengdu,China:IEEE.2016:638-642.[DOI:10.1109/CompComm.2016.7924779]
[16]Bai J W,Zhao Z C.Novel relevance feedback method based on SVM in image retrieval[J].Software Guide,2010,9(10):49-51.[白婧文,赵志诚.一种新的基于SVM相关反馈的图像检索算法[J].软件导刊,2010,9(10):49-51.]
[17]Jiang N N,Qi M,Hao C Y.A SVM based relevance feedback algorithm for image retrieval[J].Computer Simulation,2009,26(1):219-221,330.[姜楠楠,齐敏,郝重阳.一种基于SVM的相关反馈图像检索算法[J].计算机仿真,2009,26(1):219-221,330.][DOI:10.3969/j.issn.1006-9348.2009.01.059]
[18]Feng G H.Research on large scale SVM classification based on boundary K-nearest[J].Computer Engineering and Applications,2009,45(23):15-17.[奉国和.边界K邻近大样本支持向量机分类[J].计算机工程与应用,2009,45(23):15-17.][DOI:10.3778/j.issn.1002-8331.2009.23.005]
[19]Ju H Y,Zhang J B,Li C F,et al.Automated Remote Sensing Image Classification Method Based on K-means and SVM[J].Application Research of Computers,2007,24(11):318-320.[居红云,张俊本,李朝峰,等.基于K-means与SVM结合的遥感图像全自动分类方法[J].计算机应用研究,2007,24(11):318-320.][DOI:10.3969/j.issn.1001-3695.2007.11.098]
[20]Meng X F,Ma Z M,Li X,et al.A Top-K query results ranking approach based on contextual preferences for web database[J].Chinese Journal of Computers,2014,37(9):1986-1998.[孟祥福,马宗民,李昕,等.基于上下文偏好的Web数据库查询结果Top-K排序方法[J].计算机学报,2014,37(9):1986-1998.][DOI:10.3724/SP.J.1016.2014.01986]
[21]Yang Y,Newsam S.Bag-of-visual-words and spatial extensions for land-use classification[C]//Proceedings of the 18th SIGSPA-TIAL International Conference on Advances in Geographic Information Systems.San Jose,California:ACM,2010:270-279.[DOI:10.1145/1869790.1869829]
[22]Chafik S,Daoudi I,Ouardi H E,et al.Locality sensitive hashing for content based image retrieval:a comparative experimental study[C]//Proceedings of 2014 International Conference on Next Generation Networks and Services.Casablanca,Morocco:IEEE,2014:38-43.[DOI:10.1109/NGNS.2014.6990224]
[23]Jin Z M,Li C,Lin Y,et al.Density sensitive hashing[J].IEEE Transactions on Cybernetics,2014,44(8):1362-1371.[DOI:10.1109/TCYB.2013.2283497]