遥感影像建筑物提取的卷积神经元网络与开源数据集方法
|
摘要
从遥感图像中自动化地检测和提取建筑物在城市规划、人口估计、地形图制作和更新等应用中具有极为重要的意义。本文提出和展示了建筑物提取的数个研究进展。由于遥感成像机理、建筑物自身、背景环境的复杂性,传统的经验设计特征的方法一直未能实现自动化,建筑物提取成为30余年尚未解决的挑战。先进的深度学习方法带来新的机遇,但目前存在两个困境:①尚缺少高精度的建筑物数据库,而数据是深度学习必不可少的"燃料";②目前国际上的方法都采用像素级的语义分割,目标级、矢量级的提取工作亟待开展。针对于此,本文进行以下工作:①与目前同类数据集相比,建立了一套目前国际上范围最大、精度最高、涵盖多种样本形式(栅格、矢量)、多类数据源(航空、卫星)的建筑物数据库(WHU building dataset),并实现开源;②提出一种基于全卷积网络的建筑物语义分割方法,与当前国际上的最新算法相比达到了领先水平;③将建筑物提取的范围从像素级的语义分割推广至目标实例分割,实现以目标(建筑物)为对象的识别和提取。通过试验,验证了WHU数据库在国际上的领先性和本文方法的先进性。
Automatic extraction of buildings from remote sensing images is significant to city planning, popular estimation, map making and updating.We report several important developments in building extraction. Automatic building recognition from remote sensing data has been a scientific challenge of more than 30 years. Traditional methods based on empirical feature design can hardly realize automation. Advanced deep learning based methods show prospects but have two limitations now. Firstly, large and accurate building datasets are lacking while such dataset is the necessary fuel for deep learning. Secondly, the current researches only concern building's pixel wise semantic segmentation and the further extractions on instance-level and vector-level are urgently required. This paper proposes several solutions. First, we create a large, high-resolution, accurate and open-source building dataset, which consists of aerial and satellite images with both raster and vector labels. Second,we propose a novel structure based on fully neural network which achieved the best accuracy of semantic segmentation compared to most recent studies. Third, we propose a building instance segmentation method which expands the current studies of pixel-level segmentation to building-level segmentation. Experiments proved our dataset's superiority in accuracy and multi-usage and our methods' advancement. It is expected that our researches might push forward the challenging building extraction study.
Automatic extraction of buildings from remote sensing images is significant to city planning, popular estimation, map making and updating.We report several important developments in building extraction. Automatic building recognition from remote sensing data has been a scientific challenge of more than 30 years. Traditional methods based on empirical feature design can hardly realize automation. Advanced deep learning based methods show prospects but have two limitations now. Firstly, large and accurate building datasets are lacking while such dataset is the necessary fuel for deep learning. Secondly, the current researches only concern building's pixel wise semantic segmentation and the further extractions on instance-level and vector-level are urgently required. This paper proposes several solutions. First, we create a large, high-resolution, accurate and open-source building dataset, which consists of aerial and satellite images with both raster and vector labels. Second,we propose a novel structure based on fully neural network which achieved the best accuracy of semantic segmentation compared to most recent studies. Third, we propose a building instance segmentation method which expands the current studies of pixel-level segmentation to building-level segmentation. Experiments proved our dataset's superiority in accuracy and multi-usage and our methods' advancement. It is expected that our researches might push forward the challenging building extraction study.
引文
[1] LIOW Y T,PAVLIDIS T.Use of shadows for extracting buildings in aerial images[J].Computer Vision,Graphics and Image Processing,1990,49(2):242-277.
[2] SIRMACEK B,UNSALAN C.Building detection from aerial images using invariant color features and shadow information[C]//Proceedings of the 23rd International Symposium on Computer and Information Sciences.Istanbul,Turkey:IEEE,2008:1-5.
[3] ZHONG Shenghua,HUANG Jianjun,XIE Weixin.A new method of building detection from a single aerial photograph[C]//Proceedings of the 9th International Conference on Signal Processing.Beijing,China:IEEE,2008:1219-1222.
[4] ZHANG Yun.Optimisation of building detection in satellite images by combining multispectral classification and texture filtering[J].ISPRS Journal of Photogrammetry and Remote Sensing,1999,54(1):50-60.
[5] 艾泽天.LiDAR辅助遥感影像建筑物分类识别和提取研究[D].焦作:河南理工大学,2011.AI Zetian.Research on classification recognition and extraction of buildings by RS image assisted by LiDAR data[D].Jiaozuo:Henan Polytechnic University,2011.
[6] LI Yong,WU Huayi.Adaptive building edge detection by combining LiDAR data and aerial images[J].The International Archives of the Photogrammetry,Remote Sensing and Spatial Information Science,2008(37):197-202.
[7] FERRAIOLI G.Multichannel InSAR building edge detection[J].IEEE Transactions on Geoscience and Remote Sensing,2010,48(3):1224-1231.
[8] 吕凤华,舒宁,龚龑,等.利用多特征进行航空影像建筑物提取[J].武汉大学学报(信息科学版),2017,42(5):656-660.Lü Fenghua,SHU Ning,GONG Yan,et al.Regular building extraction from high resolution image based on multilevel-features[J].Geomatics and Information Science of Wuhan University,2017,42(5):656-660.
[9] DUNAEVA A V,KORNILOV F A.Specific shape building detection from aerial imagery in infrared range[J].Vychislitelnaya Matematika Inform.,2017,6(3):84-100.
[10] 吴炜,骆剑承,沈占锋,等.光谱和形状特征相结合的高分辨率遥感图像的建筑物提取方法[J].武汉大学学报(信息科学版),2012,37(7):800-805.WU Wei,LUO Jiancheng,SHEN Zhanfeng,et al.Building extraction from high resolution remote sensing imagery based on spatial-spectral method[J].Geomatics and Information Science of Wuhan University,2012,37(7):800-805.
[11] AWRANGJEB M,ZHANG C,FRASER C S.Improved building detection using texture information[J].International Archives of the Photogrammetry,2011,ⅩⅩⅩⅧ-3/W22:143-148.
[12] 何春阳,曹鑫,史培军,等.基于Landsat7 ETM+全色数据纹理和结构信息复合的城市建筑信息提取[J].武汉大学学报(信息科学版),2004,29(9):800-804.HE Chunyang,CAO Xin,SHI Peijun,et al.Incorporation of texture and structure information for urban building detection by using Landsat7 ETM+ panchromatic image[J].Geomatics and Information Science of Wuhan University,2004,29(9):800-804.
[13] CHEN Dongyue,SHANG Shibo,WU Chengdong.Shadow-based building detection and segmentation in high-resolution remote sensing image[J].Journal of Multimedia,2014,9(1):181-188.
[14] 黄明.联合LiDAR数据和遥感影像的建筑物三维模型提取方法[D].太原:太原理工大学,2014.HUANG Ming.The three-dimentional model of extraction method of buildings combined LiDAR data and remote sensing images[D].Taiyuan:Taiyuan University of Technology,2014.
[15] 穆超,余洁,许磊,等.基于高分辨率遥感影像的DSM建筑物点的提取研究[J].武汉大学学报(信息科学版),2009,34(4):414-417.MU Chao,YU Jie,XU Lei,et al.Research on extracting building points from the DSM data combining the high-resolution remote sensing image[J].Geomatics and Information Science of Wuhan University,2009,34(4):414-417.
[16] 高春霞,谢明鸿,宋纳.基于DSM与阴影的彩色遥感影像建筑物提取[J].郑州轻工业学院学报(自然科学版),2015,30(5-6):108-112.GAO Chunxia,XIE Minghong,SONG Na.Building extraction from color remote sensing imagery based on DSM and shadows[J].Journal of Zhengzhou University of Light Industry(Natural Science Edition),2015,30(5-6):108-112.
[17] ZHONG Chen,XU Qizhi,YANG Feng,et al.Building change detection for high-resolution remotely sensed images based on a semantic dependency[C]//Proceedings of 2015 IEEE International Geoscience and Remote Sensing Symposium.Milan,Italy:IEEE,2015:3345-3348.
[18] 虞欣,郑肇葆.基于语义特征和贝叶斯网络的航空影像建筑物及灌木分类[J].测绘学报,2010,39(3):245-250.YU Xin,ZHENG Zhaobao.Classification of aerial images’ building and shrub based on semantic features and Bayesian networks[J].Acta Geodaetica et Cartographica Sinica,2010,39(3):245-250.
[19] GUO Jiayi,PAN Zongxu,LEI Bin,et al.Automatic color correction for multisource remote sensing images with wasserstein CNN[J].Remote Sensing,2017,9(5):483.
[20] YAO Yuan,JIANG Zhiguo,ZHANG Haopeng,et al.Chimney and condensing tower detection based on faster R-CNN in high resolution remote sensing images[C]//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium.Fort Worth,TX:IEEE,2017:3329-3332.
[21] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[C]∥Proceedings of the 25th International Conference on Neural Information Processing Systems.Lake Tahoe,Nevada:Curran Associates Inc.,2012:1097-1105.
[22] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv preprintarXiv:1409.1556 ,2014.
[23] SZEGEDY C,LIU Wei,JIA Yangqing,et al.Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA:IEEE,2015:1-9.
[24] HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al.Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV:IEEE,2016:770-778.
[25] DENG Jia,DONG Wei,SOCHER R,et al.ImageNet:a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami,FL:IEEE,2009.
[26] LIN T Y,MAIRE M,BELONGIE S,et al.Microsoft COCO:common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision.Zurich,Switzerland:Springer,2014:740-755.
[27] LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA:IEEE,2015:3431-3440.
[28] BADRINARAYANAN V,KENDALL A,CIPOLLA R.SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(12):2481-2495.
[29] NOH H,HONG S,HAN B.Learning deconvolution network for semantic segmentation[C]//Proceedings of 2015 IEEE International Conference on Computer Vision.Santiago,Chile:IEEE,2015:1520-1528.
[30] RONNEBERGER O,FISCHER P,BROX T.U-Net:convolutional networks for biomedical image segmentation[C]//Proceedings of 18th International Conference on Medical Image Computing and Computer-Assisted Intervention.Munich,Germany:Springer,2015:234-241.
[31] MAGGIORI E,TARABALKA Y,CHARPIAT G,et al.Convolutional neural networks for large-scale remote-sensing image classification[J].IEEE Transactions on Geoscience and Remote Sensing,2017,55(2):645-657.
[32] YUAN Jiangye.Learning building extraction in aerial scenes with convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,40(11):2793-2798.
[33] GIRSHICK R,DONAHUE J,DARRELL T,et al.Region-based convolutional networks for accurate object detection and segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(1):142-158.
[34] GIRSHICK R.Fast R-CNN[C]//Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV).Santiago,Chile:IEEE,2015.
[35] REN Shaoqing,HE Kaiming,GIRSHICK R,et al.Faster R-CNN:towards real-time object detection with region proposal networks[C]// International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2015:91-99.
[36] REDMON J,DIVVALA S,GIRSHICK R,et al.You only look once:unified,real-time object detection[J].arXiv preprint arXiv:1506.02640,2015:779-788.
[37] HE Kaiming,GKIOXARI G,DOLLáR P,et al.Mask R-CNN[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018.DOI:10.1109/TPAMI.2018.2844175.
[38] MNIH V.Machine learning for aerial image labeling[D].Toronto:University of Toronto,2013.
[39] MAGGIORI E,TARABALKA Y,CHARPIAT G,et al.Can semantic labeling methods generalize to any city?the inria aerial image labeling benchmark[C]//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium.Fort Worth,TX:IEEE,2017:3226-3229.
[40] WU Guangming,SHAO Xiaowei,GUO Zhiling,et al.Automatic building segmentation of aerial imagery using multi-constraint fully convolutional networks[J].Remote Sensing,2018,10(3):407.
[2] SIRMACEK B,UNSALAN C.Building detection from aerial images using invariant color features and shadow information[C]//Proceedings of the 23rd International Symposium on Computer and Information Sciences.Istanbul,Turkey:IEEE,2008:1-5.
[3] ZHONG Shenghua,HUANG Jianjun,XIE Weixin.A new method of building detection from a single aerial photograph[C]//Proceedings of the 9th International Conference on Signal Processing.Beijing,China:IEEE,2008:1219-1222.
[4] ZHANG Yun.Optimisation of building detection in satellite images by combining multispectral classification and texture filtering[J].ISPRS Journal of Photogrammetry and Remote Sensing,1999,54(1):50-60.
[5] 艾泽天.LiDAR辅助遥感影像建筑物分类识别和提取研究[D].焦作:河南理工大学,2011.AI Zetian.Research on classification recognition and extraction of buildings by RS image assisted by LiDAR data[D].Jiaozuo:Henan Polytechnic University,2011.
[6] LI Yong,WU Huayi.Adaptive building edge detection by combining LiDAR data and aerial images[J].The International Archives of the Photogrammetry,Remote Sensing and Spatial Information Science,2008(37):197-202.
[7] FERRAIOLI G.Multichannel InSAR building edge detection[J].IEEE Transactions on Geoscience and Remote Sensing,2010,48(3):1224-1231.
[8] 吕凤华,舒宁,龚龑,等.利用多特征进行航空影像建筑物提取[J].武汉大学学报(信息科学版),2017,42(5):656-660.Lü Fenghua,SHU Ning,GONG Yan,et al.Regular building extraction from high resolution image based on multilevel-features[J].Geomatics and Information Science of Wuhan University,2017,42(5):656-660.
[9] DUNAEVA A V,KORNILOV F A.Specific shape building detection from aerial imagery in infrared range[J].Vychislitelnaya Matematika Inform.,2017,6(3):84-100.
[10] 吴炜,骆剑承,沈占锋,等.光谱和形状特征相结合的高分辨率遥感图像的建筑物提取方法[J].武汉大学学报(信息科学版),2012,37(7):800-805.WU Wei,LUO Jiancheng,SHEN Zhanfeng,et al.Building extraction from high resolution remote sensing imagery based on spatial-spectral method[J].Geomatics and Information Science of Wuhan University,2012,37(7):800-805.
[11] AWRANGJEB M,ZHANG C,FRASER C S.Improved building detection using texture information[J].International Archives of the Photogrammetry,2011,ⅩⅩⅩⅧ-3/W22:143-148.
[12] 何春阳,曹鑫,史培军,等.基于Landsat7 ETM+全色数据纹理和结构信息复合的城市建筑信息提取[J].武汉大学学报(信息科学版),2004,29(9):800-804.HE Chunyang,CAO Xin,SHI Peijun,et al.Incorporation of texture and structure information for urban building detection by using Landsat7 ETM+ panchromatic image[J].Geomatics and Information Science of Wuhan University,2004,29(9):800-804.
[13] CHEN Dongyue,SHANG Shibo,WU Chengdong.Shadow-based building detection and segmentation in high-resolution remote sensing image[J].Journal of Multimedia,2014,9(1):181-188.
[14] 黄明.联合LiDAR数据和遥感影像的建筑物三维模型提取方法[D].太原:太原理工大学,2014.HUANG Ming.The three-dimentional model of extraction method of buildings combined LiDAR data and remote sensing images[D].Taiyuan:Taiyuan University of Technology,2014.
[15] 穆超,余洁,许磊,等.基于高分辨率遥感影像的DSM建筑物点的提取研究[J].武汉大学学报(信息科学版),2009,34(4):414-417.MU Chao,YU Jie,XU Lei,et al.Research on extracting building points from the DSM data combining the high-resolution remote sensing image[J].Geomatics and Information Science of Wuhan University,2009,34(4):414-417.
[16] 高春霞,谢明鸿,宋纳.基于DSM与阴影的彩色遥感影像建筑物提取[J].郑州轻工业学院学报(自然科学版),2015,30(5-6):108-112.GAO Chunxia,XIE Minghong,SONG Na.Building extraction from color remote sensing imagery based on DSM and shadows[J].Journal of Zhengzhou University of Light Industry(Natural Science Edition),2015,30(5-6):108-112.
[17] ZHONG Chen,XU Qizhi,YANG Feng,et al.Building change detection for high-resolution remotely sensed images based on a semantic dependency[C]//Proceedings of 2015 IEEE International Geoscience and Remote Sensing Symposium.Milan,Italy:IEEE,2015:3345-3348.
[18] 虞欣,郑肇葆.基于语义特征和贝叶斯网络的航空影像建筑物及灌木分类[J].测绘学报,2010,39(3):245-250.YU Xin,ZHENG Zhaobao.Classification of aerial images’ building and shrub based on semantic features and Bayesian networks[J].Acta Geodaetica et Cartographica Sinica,2010,39(3):245-250.
[19] GUO Jiayi,PAN Zongxu,LEI Bin,et al.Automatic color correction for multisource remote sensing images with wasserstein CNN[J].Remote Sensing,2017,9(5):483.
[20] YAO Yuan,JIANG Zhiguo,ZHANG Haopeng,et al.Chimney and condensing tower detection based on faster R-CNN in high resolution remote sensing images[C]//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium.Fort Worth,TX:IEEE,2017:3329-3332.
[21] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[C]∥Proceedings of the 25th International Conference on Neural Information Processing Systems.Lake Tahoe,Nevada:Curran Associates Inc.,2012:1097-1105.
[22] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv preprintarXiv:1409.1556 ,2014.
[23] SZEGEDY C,LIU Wei,JIA Yangqing,et al.Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA:IEEE,2015:1-9.
[24] HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al.Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas,NV:IEEE,2016:770-778.
[25] DENG Jia,DONG Wei,SOCHER R,et al.ImageNet:a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami,FL:IEEE,2009.
[26] LIN T Y,MAIRE M,BELONGIE S,et al.Microsoft COCO:common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision.Zurich,Switzerland:Springer,2014:740-755.
[27] LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA:IEEE,2015:3431-3440.
[28] BADRINARAYANAN V,KENDALL A,CIPOLLA R.SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(12):2481-2495.
[29] NOH H,HONG S,HAN B.Learning deconvolution network for semantic segmentation[C]//Proceedings of 2015 IEEE International Conference on Computer Vision.Santiago,Chile:IEEE,2015:1520-1528.
[30] RONNEBERGER O,FISCHER P,BROX T.U-Net:convolutional networks for biomedical image segmentation[C]//Proceedings of 18th International Conference on Medical Image Computing and Computer-Assisted Intervention.Munich,Germany:Springer,2015:234-241.
[31] MAGGIORI E,TARABALKA Y,CHARPIAT G,et al.Convolutional neural networks for large-scale remote-sensing image classification[J].IEEE Transactions on Geoscience and Remote Sensing,2017,55(2):645-657.
[32] YUAN Jiangye.Learning building extraction in aerial scenes with convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,40(11):2793-2798.
[33] GIRSHICK R,DONAHUE J,DARRELL T,et al.Region-based convolutional networks for accurate object detection and segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,38(1):142-158.
[34] GIRSHICK R.Fast R-CNN[C]//Proceedings of 2015 IEEE International Conference on Computer Vision (ICCV).Santiago,Chile:IEEE,2015.
[35] REN Shaoqing,HE Kaiming,GIRSHICK R,et al.Faster R-CNN:towards real-time object detection with region proposal networks[C]// International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2015:91-99.
[36] REDMON J,DIVVALA S,GIRSHICK R,et al.You only look once:unified,real-time object detection[J].arXiv preprint arXiv:1506.02640,2015:779-788.
[37] HE Kaiming,GKIOXARI G,DOLLáR P,et al.Mask R-CNN[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018.DOI:10.1109/TPAMI.2018.2844175.
[38] MNIH V.Machine learning for aerial image labeling[D].Toronto:University of Toronto,2013.
[39] MAGGIORI E,TARABALKA Y,CHARPIAT G,et al.Can semantic labeling methods generalize to any city?the inria aerial image labeling benchmark[C]//Proceedings of 2017 IEEE International Geoscience and Remote Sensing Symposium.Fort Worth,TX:IEEE,2017:3226-3229.
[40] WU Guangming,SHAO Xiaowei,GUO Zhiling,et al.Automatic building segmentation of aerial imagery using multi-constraint fully convolutional networks[J].Remote Sensing,2018,10(3):407.