多模型集成的弱监督语义分割算法
|
摘要
为减小池化操作造成空间信息丢失的影响,提高基于迁移学习的弱监督语义分割算法的性能,提出一种多模型集成的弱监督图像语义分割算法.该算法在迁移学习算法的基础上,利用多尺度图像的高层语义特征和单尺度图像的高中层相结合的卷积特征,分别训练2个差异化的同质型基分割模型,并与原迁移学习训练的分割模型进行加权平均,集成构造最后的分割模型.同时结合预测类别可信度调整语义分割中对应类别像素的可信度,抑制分割图中的假正例区域,提高分割的精度.在VOC2012数据集上进行实验的结果表明,验证集上的平均重叠率为55.3%,测试集上的平均重叠率为56.9%,比原迁移学习算法分别提升6.1%和11.1%,也优于其他以类标为弱监督信息的语义分割算法.
In order to reduce the impact of loss of spatial information generated by pooling operator and improve the performance of transfer learning for weakly-supervised semantic segmentation algorithm with deep convolutional neural network, this paper designs a weakly-supervised image semantic segmentation algorithm based on multi-model ensemble. Based on transfer learning algorithm, the method firstly utilizes the semantic features from last convolutional layer of a multi-scale image and the convolutional features from the middle and deep layers of a single-scale image to respectively train two different homogeneous segmentation models. And then these models are weighted integrating with the original transfer-learning model to get the final segmentation model. In addition, the algorithm combines the confidence of categories to adjust the pixels' confidence expecting to suppress the false positive regions in the segmented image to improve the accuracy. Finally, the proposed algorithm is tested in challenging VOC2012 dataset. The results show that the mean intersection-over-union of the proposed algorithm is 55.3% on validation dataset and 56.9% on test set, outperforming the original transfer-learning algorithm by 6.1% and 11.1%, respectively. And the method performs favorably against other segmentation methods using weakly-supervised information based on class labels as well.
In order to reduce the impact of loss of spatial information generated by pooling operator and improve the performance of transfer learning for weakly-supervised semantic segmentation algorithm with deep convolutional neural network, this paper designs a weakly-supervised image semantic segmentation algorithm based on multi-model ensemble. Based on transfer learning algorithm, the method firstly utilizes the semantic features from last convolutional layer of a multi-scale image and the convolutional features from the middle and deep layers of a single-scale image to respectively train two different homogeneous segmentation models. And then these models are weighted integrating with the original transfer-learning model to get the final segmentation model. In addition, the algorithm combines the confidence of categories to adjust the pixels' confidence expecting to suppress the false positive regions in the segmented image to improve the accuracy. Finally, the proposed algorithm is tested in challenging VOC2012 dataset. The results show that the mean intersection-over-union of the proposed algorithm is 55.3% on validation dataset and 56.9% on test set, outperforming the original transfer-learning algorithm by 6.1% and 11.1%, respectively. And the method performs favorably against other segmentation methods using weakly-supervised information based on class labels as well.
引文
[1]Chen L C,Papandreou G,Kokkinos I,et al.DeepLab:semantic image segmentation with deep convolutional nets,atrous convolution,and fully connected CRFs[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2018,40(4):834-848
[2]Everingham M,van Gool L,Williams C K I,et al.The pascal visual object classes(VOC)challenge[J].International Journal of Computer Vision,2010,88(2):303-338
[3]Cordts M,Omran M,Ramos S,et al.The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:3213-3223
[4]Lin T Y,Maire M,Belongie S,et al.Microsoft COCO:common objects in context[C]//Proceedings of European Conference on Computer Vision.Heidelberg:Springer,2014:740-755
[5]Papandreou G,Chen L C,Murphy K P,et al.Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2015:1742-1750
[6]Dai J F,He K M,Sun J.BoxSup:exploiting bounding boxes to supervise convolutional networks for semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2015:1635-1643
[7]Lin D,Dai J F,Jia J Y,et al.ScribbleSup:scribble-supervised convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:3159-3167
[8]Bearman A,Russakovsky O,Ferrari V,et al.What’s the point:semantic segmentation with point supervision[C]//Proceedings of European Conference on Computer Vision.Heidelberg:Springer,2016:549-565
[9]Souly N,Spampinato C,Shah M.Semi supervised semantic segmentation using generative adversarial network[C]//Proceedings of the IEEE International Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2017:5689-5697
[10]Hung W C,Tsai Y H,Liou Y T,et al.Adversarial learning for semi-supervised semantic segmentation[OL].[2018-06-29].https://arxiv.org/pdf/1802.07934v1.pdf
[11]Pinheiro P O,Collobert R.From image-level to pixel-level labeling with convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2015:1713-1721
[12]Hong S,Oh J,Lee H,et al.Learning transferrable knowledge for semantic segmentation with deep convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:3204-3212
[13]Kolesnikov A,Lampert C H.Seed,expand and constrain:three principles for weakly-supervised image segmentation[C]//Proceedings of European Conference on Computer Vision.Heidelberg:Springer,2016:695-711
[14]Wei Y C,Liang X D,Chen Y P,et al.STC:a simple to complex framework for weakly-supervised semantic segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(11):2314-2320
[15]Qi X J,Liu Z Z,Shi J P,et al.Augmented feedback in sematic segmentation under image level supervision[C]//Proceedings of European Conference on Computer Vision.Heidelberg:Springer,2016:90-105
[16]Chen L C,Yang Y,Wang J,et al.Attention to scale:scale-aware semantic image segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:3640-3649
[17]Badrinarayanan V,Kendall A,Cipolla R.SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(12):2481-2495
[18]Yu F,Koltun V.Multi-scale context aggregation by dilated convolutions[OL].[2018-06-29].https://arxiv.org/pdf/1511.07-122v2.pdf
[19]Zhao H S,Shi J P,Qi X J,et al.Pyramid scene parsing network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2017:6230-6239
[20]Zhang Wenwen,Han Yusheng,Huang Qinchao,et al.The fast multi-scale convolutional sparse coding based super-resolution for infrared image[J].Journal of Computer-Aided Design&Computer Graphics,2018,30(10):1935-1942(in Chinese)(张雯雯,韩裕生,黄勤超,等.基于多尺度卷积稀疏编码的红外图像快速超分辨率[J].计算机辅助设计与图形学学报,2018,30(10):1935-1942)
[21]Zhou Zhihua.Machine learning[M].Beijing:Tsinghua University Press,2016:171-184(in Chinese)(周志华.机器学习[M].北京:清华大学出版社,2016:171-184)
[22]Chen L C,Barron J T,Papandreou G,et al.Semantic image segmentation with task-specific edge detection using CNNs and a discriminatively trained domain transform[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:4545-4554
[23]Wei Y C,Feng J S,Liang X D,et al.Object region mining with adversarial erasing:a simple classification to semantic segmentation approach[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2017:6488-6496
[2]Everingham M,van Gool L,Williams C K I,et al.The pascal visual object classes(VOC)challenge[J].International Journal of Computer Vision,2010,88(2):303-338
[3]Cordts M,Omran M,Ramos S,et al.The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:3213-3223
[4]Lin T Y,Maire M,Belongie S,et al.Microsoft COCO:common objects in context[C]//Proceedings of European Conference on Computer Vision.Heidelberg:Springer,2014:740-755
[5]Papandreou G,Chen L C,Murphy K P,et al.Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2015:1742-1750
[6]Dai J F,He K M,Sun J.BoxSup:exploiting bounding boxes to supervise convolutional networks for semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2015:1635-1643
[7]Lin D,Dai J F,Jia J Y,et al.ScribbleSup:scribble-supervised convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:3159-3167
[8]Bearman A,Russakovsky O,Ferrari V,et al.What’s the point:semantic segmentation with point supervision[C]//Proceedings of European Conference on Computer Vision.Heidelberg:Springer,2016:549-565
[9]Souly N,Spampinato C,Shah M.Semi supervised semantic segmentation using generative adversarial network[C]//Proceedings of the IEEE International Conference on Computer Vision.Los Alamitos:IEEE Computer Society Press,2017:5689-5697
[10]Hung W C,Tsai Y H,Liou Y T,et al.Adversarial learning for semi-supervised semantic segmentation[OL].[2018-06-29].https://arxiv.org/pdf/1802.07934v1.pdf
[11]Pinheiro P O,Collobert R.From image-level to pixel-level labeling with convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2015:1713-1721
[12]Hong S,Oh J,Lee H,et al.Learning transferrable knowledge for semantic segmentation with deep convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:3204-3212
[13]Kolesnikov A,Lampert C H.Seed,expand and constrain:three principles for weakly-supervised image segmentation[C]//Proceedings of European Conference on Computer Vision.Heidelberg:Springer,2016:695-711
[14]Wei Y C,Liang X D,Chen Y P,et al.STC:a simple to complex framework for weakly-supervised semantic segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(11):2314-2320
[15]Qi X J,Liu Z Z,Shi J P,et al.Augmented feedback in sematic segmentation under image level supervision[C]//Proceedings of European Conference on Computer Vision.Heidelberg:Springer,2016:90-105
[16]Chen L C,Yang Y,Wang J,et al.Attention to scale:scale-aware semantic image segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:3640-3649
[17]Badrinarayanan V,Kendall A,Cipolla R.SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,39(12):2481-2495
[18]Yu F,Koltun V.Multi-scale context aggregation by dilated convolutions[OL].[2018-06-29].https://arxiv.org/pdf/1511.07-122v2.pdf
[19]Zhao H S,Shi J P,Qi X J,et al.Pyramid scene parsing network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2017:6230-6239
[20]Zhang Wenwen,Han Yusheng,Huang Qinchao,et al.The fast multi-scale convolutional sparse coding based super-resolution for infrared image[J].Journal of Computer-Aided Design&Computer Graphics,2018,30(10):1935-1942(in Chinese)(张雯雯,韩裕生,黄勤超,等.基于多尺度卷积稀疏编码的红外图像快速超分辨率[J].计算机辅助设计与图形学学报,2018,30(10):1935-1942)
[21]Zhou Zhihua.Machine learning[M].Beijing:Tsinghua University Press,2016:171-184(in Chinese)(周志华.机器学习[M].北京:清华大学出版社,2016:171-184)
[22]Chen L C,Barron J T,Papandreou G,et al.Semantic image segmentation with task-specific edge detection using CNNs and a discriminatively trained domain transform[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2016:4545-4554
[23]Wei Y C,Feng J S,Liang X D,et al.Object region mining with adversarial erasing:a simple classification to semantic segmentation approach[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Los Alamitos:IEEE Computer Society Press,2017:6488-6496