基于YOLO深度卷积神经网络的复杂背景下机器人采摘苹果定位
|
摘要
为提高苹果采摘机器人的工作效率和环境适应性,使其能全天候的在不同光线环境下对遮挡、粘连和套袋等多种情况下的果实进行识别定位,该文提出了基于YOLOv3(you only look once)深度卷积神经网络的苹果定位方法。该方法通过单个卷积神经网络(one-stage)遍历整个图像,回归目标的类别和位置,实现了直接端到端的目标检测,在保证效率与准确率兼顾的情况下实现了复杂环境下苹果的检测。经过训练的模型在验证集下的m AP(meanaverageprecision)为87.71%,准确率为97%,召回率为90%,IOU(intersection over union)为83.61%。通过比较YOLOv3与Faster RCNN算法在不同数目、不同拍摄时间、不同生长阶段、不同光线下对苹果的实际检测效果,并以F1为评估值对比分析了4种算法的差异,试验结果表明YOLOv3在密集苹果的F1高于YOLOv2算法4.45个百分点,在其他环境下高于Faster RCNN将近5个百分点,高于HOG+SVM(histogram of oriented gradient+support vector machine)将近10个百分点。并且在不同硬件环境验证了该算法的可行性,一幅图像在GPU下的检测时间为16.69 ms,在CPU下的检测时间为105.21 ms,实际检测视频的帧率达到了60帧/s和15帧/s。该研究可为机器人快速长时间高效率在复杂环境下识别苹果提供理论基础。
Automatic recognition of apple is one of the important aspects for apple harvest robots.Fast apple recognition can improve the efficiency of picking robots.In the actual scene of the orchard,the recognition conditions for apple are complex such as daytime,night,overlap apples,occlusion,bagged,backlighting,reflected light and dense apple,considering which a highly robust and fast visual recognition scheme is required.A fast and stable apple recognition scheme was proposed based on improved YOLOv3 in this paper.The entire image was traversed by a single convolutional neural network(one-stage),dividing an image into a plurality of sub-regions with the same size,and predicting the class of the target and its bounding box in each sub-region.Finally,the non-maximum value suppression was merged into the outer frame of the whole target,and the category and position of the target were returned.In order to improve the detection efficiency,the VGG-like network model was used to replace the original residual network of YOLOv3,and the model size was reduced,in which the 53-layer neural network was compressed into a 13-layer neural network without affecting the detection effect.Taking into account the size of the smallest apple in dense apples images,the anchor points of 3 different sizes were reduced to 2,reducing the final predicted tensor and ensuring that the smallest anchor point could still include the minimum target.The steps in this paper were stated as follows:Firstly,the data set was manually marked,including 400 images for the training set and 115 images for the verification set,including a total of 1 158 apple samples.In addition,in order to increase the generalization ability of the model,the data set was enhanced by adjusting the hue,color amount and exposure of the image,and a total of 51 500 images were generated.Then the initial value of the anchor points was calculated through K-means.Secondly,training the data set,output a model every 100 iterations.For the verification set,the mean average precision(mAP) value of each weight in batches was calculated,selecting the model with the highest mAP value,and finding the appropriate threshold to ensure most preferred precision,recall rate and intersection over union(IOU).The trained model had a m AP which reached up to 87.71%,an accuracy rate up to 97%,a recall rate up to 90%,and an IOU up to 83.61%.Thirdly,the specific performance of the model under image conditions for different fruit number,illumination angle,fruit growth stage and shooting time were verified in additional experimental data sets.The experimental data set consisted of 336 pictures containing 1 410 apple samples.The comparison was performed with algorithms of HOG+SVM,Faster RCNN,YOLOv2,and YOLOv3,with the evaluated index of F1 value.The experimental results showed that YOLOv3 performed significantly better than YOLOv2 in dense apples image,and better in other environments than Faster RCNN and HOG+SVM.Finally,the detection accuracy of the algorithm was verified in different hardware environments.The detection time of an image under the GPU was 16.69 ms with 60 frame/s for the actual video,and under the CPU was 105.21 ms with 15 frame/s for the actual video.Since it was positioned only at the beginning of the picking process and it did not require frequently refreshing during the picking process,in which the detection time in this paper was qualified.A reference was provided for the rapid,long-term high efficiency of robots to locate apples in complex environments in this research.
Automatic recognition of apple is one of the important aspects for apple harvest robots.Fast apple recognition can improve the efficiency of picking robots.In the actual scene of the orchard,the recognition conditions for apple are complex such as daytime,night,overlap apples,occlusion,bagged,backlighting,reflected light and dense apple,considering which a highly robust and fast visual recognition scheme is required.A fast and stable apple recognition scheme was proposed based on improved YOLOv3 in this paper.The entire image was traversed by a single convolutional neural network(one-stage),dividing an image into a plurality of sub-regions with the same size,and predicting the class of the target and its bounding box in each sub-region.Finally,the non-maximum value suppression was merged into the outer frame of the whole target,and the category and position of the target were returned.In order to improve the detection efficiency,the VGG-like network model was used to replace the original residual network of YOLOv3,and the model size was reduced,in which the 53-layer neural network was compressed into a 13-layer neural network without affecting the detection effect.Taking into account the size of the smallest apple in dense apples images,the anchor points of 3 different sizes were reduced to 2,reducing the final predicted tensor and ensuring that the smallest anchor point could still include the minimum target.The steps in this paper were stated as follows:Firstly,the data set was manually marked,including 400 images for the training set and 115 images for the verification set,including a total of 1 158 apple samples.In addition,in order to increase the generalization ability of the model,the data set was enhanced by adjusting the hue,color amount and exposure of the image,and a total of 51 500 images were generated.Then the initial value of the anchor points was calculated through K-means.Secondly,training the data set,output a model every 100 iterations.For the verification set,the mean average precision(mAP) value of each weight in batches was calculated,selecting the model with the highest mAP value,and finding the appropriate threshold to ensure most preferred precision,recall rate and intersection over union(IOU).The trained model had a m AP which reached up to 87.71%,an accuracy rate up to 97%,a recall rate up to 90%,and an IOU up to 83.61%.Thirdly,the specific performance of the model under image conditions for different fruit number,illumination angle,fruit growth stage and shooting time were verified in additional experimental data sets.The experimental data set consisted of 336 pictures containing 1 410 apple samples.The comparison was performed with algorithms of HOG+SVM,Faster RCNN,YOLOv2,and YOLOv3,with the evaluated index of F1 value.The experimental results showed that YOLOv3 performed significantly better than YOLOv2 in dense apples image,and better in other environments than Faster RCNN and HOG+SVM.Finally,the detection accuracy of the algorithm was verified in different hardware environments.The detection time of an image under the GPU was 16.69 ms with 60 frame/s for the actual video,and under the CPU was 105.21 ms with 15 frame/s for the actual video.Since it was positioned only at the beginning of the picking process and it did not require frequently refreshing during the picking process,in which the detection time in this paper was qualified.A reference was provided for the rapid,long-term high efficiency of robots to locate apples in complex environments in this research.
引文
[1]Kapach K,Barnea E,Mairon R,et al.Computer vision for fruit harvesting robots-state of the art and challenges ahead[J].International Journal of Computational Vision and Robotics,2012,3(1/2):4-34.
[2]吕继东,赵德安.苹果采摘机器人目标果实快速跟踪识别方法[J].农业机械学报,2014,45(1):65-72.Lu Jidong,Zhao De'an.Fast tracking and recognition method for target fruits of apple picking robots[J].Transactions of the Chinese Society for Agricultural Machinery,2014,45(1):65-72.(in Chinese with English abstract)
[3]赵德安,刘晓洋,陈玉,等.苹果采摘机器人夜间识别方法[J].农业机械学报,2015,46(3):15-22.Zhao Dean,Liu Xiaoyang,Chen Yu,et al.Night recognition method of apple picking robot[J].Transactions of the Chinese Society for Agricultural Machinery,2015,46(3):15-22.(in Chinese with English abstract)
[4]Ji W,Zhao D,Cheng F,et al.Automatic recognition vision system guided for apple harvesting robot[J].Computers&Electrical Engineering,2012,38(5):1186-1195.
[5]Stajnko D,Lakota M,Ho?evar M.Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging[J].Computers and Electronics in Agriculture,2004,42(1):31-42.
[6]Wachs J P,Stern H I,Burks T,et al.Low and high-level visual feature-based apple detection from multi-modal images[J].Precision Agriculture,2010,11(6):717-735.
[7]Rakun J,Stajnko D,Zazula D.Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry[J].Computers and Electronics in Agriculture,2011,76(1):80-88.
[8]Aggelopoulou A D,Bochtis D,Fountas S,et al.Yield prediction in apple orchards based on image processing[J].Precision Agriculture,2011,12(3):448-456.
[9]宋怀波,张卫园,张欣欣,等.基于模糊集理论的苹果表面阴影去除方法[J].农业工程学报,2014,30(3):135-141.Song Huaibo,Zhang Weiyuan,Zhang Xinxin,et al.Apple surface shadow removal method based on fuzzy set theory[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2014,30(3):135-141.(in Chinese with English abstract)
[10]Kurtulmus F,Lee W S,Vardar A.Green citrus detection using‘eigenfruit’,color and circular Gabor texture features under natural outdoor conditions[J].Computers and Electronics in Agriculture,2011,78(2):140-149.
[11]Linker R,Cohen O,Naor A.Determination of the number of green apples in RGB images recorded in orchards[J].Computers and Electronics in Agriculture,2012,81:45-57.
[12]Arivazhagan S,Shebiah R N,Nidhyanandhan S S,et al.Fruit recognition using color and texture features[J].Journal of Emerging Trends in Computing and Information Sciences,2010,1(2):90-94.
[13]Xu Y,Imou K,Kaizu Y,et al.Two-stage approach for detecting slightly overlapping strawberries using HOGdescriptor[J].Biosystems engineering,2013,115(2):144-153.
[14]卢军,桑农.变化光照下树上柑橘目标检测与遮挡轮廓恢复技术[J].农业机械学报,2014,45(4):76-81.Lu Jun,Sang Nong.Detection of citrus targets and restoration of concealed contours in trees under changing light[J].Transactions of the Chinese Society for Agricultural Machinery,2014,45(4):76-81.(in Chinese with English abstract)
[15]Zhao C,Lee W S,He D.Immature green citrus detection based on colour feature and sum of absolute transformed difference(SATD)using colour images in the citrus grove[J].Computers and Electronics in Agriculture,2016,124:243-253.
[16]谢忠红,姬长英,郭小清,等.基于改进Hough变换的类圆果实目标检测[J].农业工程学报,2010,26(7):157-162.Xie Zhonghong,Ji Changying,Guo Xiaoqing,et al.Target detection of fruit-like fruit based on improved Hough transform[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2010,26(7):157-162.(in Chinese with English abstract)
[17]马翠花,张学平,李育涛,等.基于显著性检测与改进Hough变换方法识别未成熟番茄[J].农业工程学报,2016,32(14):219-226.Ma Cuihua,Zhang Xueping,Li Yutao,et al.Identification of immature tomatoes based on saliency detection and improved Hough transform method[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2016,32(14):219-226.(in Chinese with English abstract)
[18]Girshick R,Donahue J,Darrell T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.2014:580-587.
[19]Girshick R.Fast R-CNN[C]//Proceedings of the IEEEinternational conference on computer vision.2015:1440-1448.
[20]Ren S,He K,Girshick R,et al.Faster R-CNN:Towards real-time object detection with region proposal networks[C]//Advances in neural information processing systems.2015:91-99.
[21]熊俊涛,刘振,汤林越,等.自然环境下绿色柑橘视觉检测技术研究[J].农业机械学报,2018,49(4):45-52.Xiong Juntao,Liu Zhen,Tang Linyue,et al.Research on green citrus vision detection technology in natural environment[J].Transactions of the Chinese Society for Agricultural Machinery,2018,49(4):45-52.(in Chinese with English abstract)
[22]Liu W,Anguelov D,Erhan D,et al.Ssd:Single shot multibox detector[C]//European conference on computer vision.Springer,Cham,2016:21-37.
[23]Redmon J,Divvala S,Girshick R,et al.You only look once:Unified,real-time object detection[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.2016:779-788.
[24]薛月菊,黄宁,涂淑琴,等.未成熟芒果的改进YOLOv2识别方法[J].农业工程学报,2018,34(7):173-179.Xue Yueju,Huang Ning,Tu Shuqin,et al.Immature mango detection based on improved YOLOv2[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2018,34(7):173-179.(in Chinese with English abstract)
[25]Redmon J,Farhadi A.YOLO9000:Better,Faster,Stronger[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2017:6517-6525.
[26]He K,Zhang X,Ren S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.2016:770-778.
[27]Ioffe S,Szegedy C.Batch normalization:Accelerating deep network training by reducing internal covariate shift[C]//International Conference on Machine Learning.2015:448-456.
[28]Bargoti S,Underwood J.Deep fruit detection in orchards[C]//IEEE International Conference on Robotics and Automation(ICRA).IEEE,2017:3626-3633.
[29]Ding W,Taylor G.Automatic moth detection from trap images for pest management[J].Computers and Electronics in Agriculture,2016,123:17-28.
[30]Hripcsak G,Rothschild A S.Agreement,the f-measure,and reliability in information retrieval[J].Journal of the American Medical Informatics Association,2005,12(3):296-298.
[31]Dalal N,Triggs B.Histograms of oriented gradients for human detection[C]//Computer Vision and Pattern Recognition,2005.CVPR 2005.IEEE Computer Society Conference on.IEEE,2005:886-893.
[2]吕继东,赵德安.苹果采摘机器人目标果实快速跟踪识别方法[J].农业机械学报,2014,45(1):65-72.Lu Jidong,Zhao De'an.Fast tracking and recognition method for target fruits of apple picking robots[J].Transactions of the Chinese Society for Agricultural Machinery,2014,45(1):65-72.(in Chinese with English abstract)
[3]赵德安,刘晓洋,陈玉,等.苹果采摘机器人夜间识别方法[J].农业机械学报,2015,46(3):15-22.Zhao Dean,Liu Xiaoyang,Chen Yu,et al.Night recognition method of apple picking robot[J].Transactions of the Chinese Society for Agricultural Machinery,2015,46(3):15-22.(in Chinese with English abstract)
[4]Ji W,Zhao D,Cheng F,et al.Automatic recognition vision system guided for apple harvesting robot[J].Computers&Electrical Engineering,2012,38(5):1186-1195.
[5]Stajnko D,Lakota M,Ho?evar M.Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging[J].Computers and Electronics in Agriculture,2004,42(1):31-42.
[6]Wachs J P,Stern H I,Burks T,et al.Low and high-level visual feature-based apple detection from multi-modal images[J].Precision Agriculture,2010,11(6):717-735.
[7]Rakun J,Stajnko D,Zazula D.Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry[J].Computers and Electronics in Agriculture,2011,76(1):80-88.
[8]Aggelopoulou A D,Bochtis D,Fountas S,et al.Yield prediction in apple orchards based on image processing[J].Precision Agriculture,2011,12(3):448-456.
[9]宋怀波,张卫园,张欣欣,等.基于模糊集理论的苹果表面阴影去除方法[J].农业工程学报,2014,30(3):135-141.Song Huaibo,Zhang Weiyuan,Zhang Xinxin,et al.Apple surface shadow removal method based on fuzzy set theory[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2014,30(3):135-141.(in Chinese with English abstract)
[10]Kurtulmus F,Lee W S,Vardar A.Green citrus detection using‘eigenfruit’,color and circular Gabor texture features under natural outdoor conditions[J].Computers and Electronics in Agriculture,2011,78(2):140-149.
[11]Linker R,Cohen O,Naor A.Determination of the number of green apples in RGB images recorded in orchards[J].Computers and Electronics in Agriculture,2012,81:45-57.
[12]Arivazhagan S,Shebiah R N,Nidhyanandhan S S,et al.Fruit recognition using color and texture features[J].Journal of Emerging Trends in Computing and Information Sciences,2010,1(2):90-94.
[13]Xu Y,Imou K,Kaizu Y,et al.Two-stage approach for detecting slightly overlapping strawberries using HOGdescriptor[J].Biosystems engineering,2013,115(2):144-153.
[14]卢军,桑农.变化光照下树上柑橘目标检测与遮挡轮廓恢复技术[J].农业机械学报,2014,45(4):76-81.Lu Jun,Sang Nong.Detection of citrus targets and restoration of concealed contours in trees under changing light[J].Transactions of the Chinese Society for Agricultural Machinery,2014,45(4):76-81.(in Chinese with English abstract)
[15]Zhao C,Lee W S,He D.Immature green citrus detection based on colour feature and sum of absolute transformed difference(SATD)using colour images in the citrus grove[J].Computers and Electronics in Agriculture,2016,124:243-253.
[16]谢忠红,姬长英,郭小清,等.基于改进Hough变换的类圆果实目标检测[J].农业工程学报,2010,26(7):157-162.Xie Zhonghong,Ji Changying,Guo Xiaoqing,et al.Target detection of fruit-like fruit based on improved Hough transform[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2010,26(7):157-162.(in Chinese with English abstract)
[17]马翠花,张学平,李育涛,等.基于显著性检测与改进Hough变换方法识别未成熟番茄[J].农业工程学报,2016,32(14):219-226.Ma Cuihua,Zhang Xueping,Li Yutao,et al.Identification of immature tomatoes based on saliency detection and improved Hough transform method[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2016,32(14):219-226.(in Chinese with English abstract)
[18]Girshick R,Donahue J,Darrell T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.2014:580-587.
[19]Girshick R.Fast R-CNN[C]//Proceedings of the IEEEinternational conference on computer vision.2015:1440-1448.
[20]Ren S,He K,Girshick R,et al.Faster R-CNN:Towards real-time object detection with region proposal networks[C]//Advances in neural information processing systems.2015:91-99.
[21]熊俊涛,刘振,汤林越,等.自然环境下绿色柑橘视觉检测技术研究[J].农业机械学报,2018,49(4):45-52.Xiong Juntao,Liu Zhen,Tang Linyue,et al.Research on green citrus vision detection technology in natural environment[J].Transactions of the Chinese Society for Agricultural Machinery,2018,49(4):45-52.(in Chinese with English abstract)
[22]Liu W,Anguelov D,Erhan D,et al.Ssd:Single shot multibox detector[C]//European conference on computer vision.Springer,Cham,2016:21-37.
[23]Redmon J,Divvala S,Girshick R,et al.You only look once:Unified,real-time object detection[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.2016:779-788.
[24]薛月菊,黄宁,涂淑琴,等.未成熟芒果的改进YOLOv2识别方法[J].农业工程学报,2018,34(7):173-179.Xue Yueju,Huang Ning,Tu Shuqin,et al.Immature mango detection based on improved YOLOv2[J].Transactions of the Chinese Society of Agricultural Engineering(Transactions of the CSAE),2018,34(7):173-179.(in Chinese with English abstract)
[25]Redmon J,Farhadi A.YOLO9000:Better,Faster,Stronger[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2017:6517-6525.
[26]He K,Zhang X,Ren S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.2016:770-778.
[27]Ioffe S,Szegedy C.Batch normalization:Accelerating deep network training by reducing internal covariate shift[C]//International Conference on Machine Learning.2015:448-456.
[28]Bargoti S,Underwood J.Deep fruit detection in orchards[C]//IEEE International Conference on Robotics and Automation(ICRA).IEEE,2017:3626-3633.
[29]Ding W,Taylor G.Automatic moth detection from trap images for pest management[J].Computers and Electronics in Agriculture,2016,123:17-28.
[30]Hripcsak G,Rothschild A S.Agreement,the f-measure,and reliability in information retrieval[J].Journal of the American Medical Informatics Association,2005,12(3):296-298.
[31]Dalal N,Triggs B.Histograms of oriented gradients for human detection[C]//Computer Vision and Pattern Recognition,2005.CVPR 2005.IEEE Computer Society Conference on.IEEE,2005:886-893.