使用线约束运动最小二乘法的视差图像拼接
|
摘要
目的图像配准是影响拼接质量的关键因素。已有的视差图象拼接方法没有解决匹配特征点对间的错误配准问题,容易引起不自然的拼接痕迹。针对这一问题,提出了使用线约束运动最小二乘法的配准算法,减少图像的配准误差,提高拼接质量。方法首先,计算目标图像和参考图像的SIFT(scale-invariant feature transform)特征点,应用RANSAC(random sample consensus)方法建立特征点的匹配关系,由此计算目标到参考图像的最佳单应变换。然后,使用线约束运动最小二乘法分别配准两组图像:1)第1组是目标图像和参考图像;2)第2组是经单应变换后的目标图像和参考图像。第1组用逐点仿射变换进行配准,而第2组配准使用了单应变换加上逐点仿射变换。最后,在重叠区域,利用最大流最小割算法寻找最优拼接缝,沿着拼接缝评估两组配准的质量,选取最优的那组进行融合拼接。结果自拍图库和公开数据集上的大量测试结果表明,本文算法的配准精度超过95%,透视扭曲比例小于17%。与近期拼接方法相比,本文配准算法精度提高3%,拼接结果中透视扭曲现象减少73%。结论运动最小二乘法可以准确地配准特征点,但可能会扭曲图像中的结构对象。而线约束项则尽量保持结构,阻止扭曲。因此,线约束运动最小二乘法兼顾了图像结构的完整性和匹配特征点的对准精度,基于此配准模型的拼接方法能够有效减少重影和鬼影等人工痕迹,拼接结果真实自然。
Objective Image alignment is a key factor in assessing stitching performance. Image deformation is a critical step of the alignment model for parallax image stitching and directly determines the alignment quality. Accurately aligning all points in an overlapping region of parallax images is difficult. Thus,an alignment strategy that can produce visually satisfying stitching results must be developed. Recent state-of-the-art stitching methods practically combine homography with content-preserving warping. Either homography is first used to pre-align two images and is followed by content-preserving warping to refine alignment,or the mesh deformation is globally optimized by solving an energy function,which is a weighting linear combination of homography and content-preserving warping. Both approaches commonly use homography in the aligning phase and therefore easily produce perspective distortion. At the same time,these approaches possibly misalign the object edges of images with several dominant structural objects. To address these problems,this paper presented a novel stitching method that combines homography,deformation using moving least squares( MLS),and line constraint. The deformation method based on MLS has an interpolation property and can therefore accurately align matching feature points.However,this deformation method may distort structural regions; thus a line constraint item was added to the deformation model to preserve the structure. Method To attain a clear depiction,we considered a two-image stitching as an example.The two input images are called the target and reference images,respectively,and are denoted by T and R. First,feature detection and matching estimation were conducted using SIFT and RANSAC,followed by distance similarity to check the matching accuracy of the feature points. The homography( denoted by H) with the best geometric fit was selected. Then,H was applied to the target image T,and the transformed image was denoted as TH. Afterward,the two group images( T,R) and(TH,R) were aligned using a line constraint MLS. To eliminate perspective distortion in the deformation image,affine transformation was used in MLS. However,a simple affine transformation was insufficient to handle the parallax.Thus,an additional pair of images( TH,R) was processed as a candidate stitching result for the pair of images(T,R).The test experiments revealed that many examples obtained a more natural stitching result when only affine transformation rather than the composite transformation of homography and affine transformation was applied,implying that the alignment between T and R was better than that between THand R. Taking the deformation from the target image T to the reference image R as an example,the line constraint MLS was outlined as follows. First,the four corner points of T were deformed to the coordinate system of R by using matching feature points as control points based on MLS. Then,we deformed the remaining points on the four border lines( top,bottom,left,and right boundaries) of T by using line constraint MLS. Here,the line constraint was constructed by preserving the relative position of each point of a border line,based on which a deformation objective function was developed. Similarly,we handled the internal points of T by using vertical and horizontal grid lines as constraint conditions,and the vertical and horizontal grid lines are consisted of the constraint lines of their intersection point. Finally,the quality of each alignment was evaluated,and the best one was chosen to blend them. In the overlapping regions,the max-flow min-cut algorithm was used to find the best stitching seam-cut of two alignments and assess the alignment quality along the seam-cut. The assessment of the alignment quality mainly considered the color and structural differences between overlapping regions of two images,and the structure was reflected by a gradient. Then,feathering approach was utilized to blend the two images of the best alignment. Result To test our stitching algorithm,23 pairs of pictures,which cover commonly seen natural and man-made scenes,were captured. In addition,we conducted several experiments on publicly published data provided by recent related works. The experimental results demonstrated that the alignment accuracy of our method exceeded 95%,and the ratio of perspective distortion was lower than 17%. Compared with recent state-of-the-art methods,our method's alignment accuracy was higher by 3%,and the ratio of perspective distortion was lower by 73%. Therefore,our method exhibits a better performance in handling image stitching with a large parallax,and the stitching result is authentic and natural. Conclusion This paper presented a hybrid transformation for aligning two images that combines line constraint with MLS. In addition,an alignment quality evaluation rule was introduced by computing the weighted differences of the points along the stitching seam-cut and the remaining points in the overlapping region.As the proposed method can balance alignment accuracy and structure preservation,it can address the misalignment issues easily caused by current stitching approaches for parallax images and effectively reduce stitching artifacts,such as ghosting and distortion.
Objective Image alignment is a key factor in assessing stitching performance. Image deformation is a critical step of the alignment model for parallax image stitching and directly determines the alignment quality. Accurately aligning all points in an overlapping region of parallax images is difficult. Thus,an alignment strategy that can produce visually satisfying stitching results must be developed. Recent state-of-the-art stitching methods practically combine homography with content-preserving warping. Either homography is first used to pre-align two images and is followed by content-preserving warping to refine alignment,or the mesh deformation is globally optimized by solving an energy function,which is a weighting linear combination of homography and content-preserving warping. Both approaches commonly use homography in the aligning phase and therefore easily produce perspective distortion. At the same time,these approaches possibly misalign the object edges of images with several dominant structural objects. To address these problems,this paper presented a novel stitching method that combines homography,deformation using moving least squares( MLS),and line constraint. The deformation method based on MLS has an interpolation property and can therefore accurately align matching feature points.However,this deformation method may distort structural regions; thus a line constraint item was added to the deformation model to preserve the structure. Method To attain a clear depiction,we considered a two-image stitching as an example.The two input images are called the target and reference images,respectively,and are denoted by T and R. First,feature detection and matching estimation were conducted using SIFT and RANSAC,followed by distance similarity to check the matching accuracy of the feature points. The homography( denoted by H) with the best geometric fit was selected. Then,H was applied to the target image T,and the transformed image was denoted as TH. Afterward,the two group images( T,R) and(TH,R) were aligned using a line constraint MLS. To eliminate perspective distortion in the deformation image,affine transformation was used in MLS. However,a simple affine transformation was insufficient to handle the parallax.Thus,an additional pair of images( TH,R) was processed as a candidate stitching result for the pair of images(T,R).The test experiments revealed that many examples obtained a more natural stitching result when only affine transformation rather than the composite transformation of homography and affine transformation was applied,implying that the alignment between T and R was better than that between THand R. Taking the deformation from the target image T to the reference image R as an example,the line constraint MLS was outlined as follows. First,the four corner points of T were deformed to the coordinate system of R by using matching feature points as control points based on MLS. Then,we deformed the remaining points on the four border lines( top,bottom,left,and right boundaries) of T by using line constraint MLS. Here,the line constraint was constructed by preserving the relative position of each point of a border line,based on which a deformation objective function was developed. Similarly,we handled the internal points of T by using vertical and horizontal grid lines as constraint conditions,and the vertical and horizontal grid lines are consisted of the constraint lines of their intersection point. Finally,the quality of each alignment was evaluated,and the best one was chosen to blend them. In the overlapping regions,the max-flow min-cut algorithm was used to find the best stitching seam-cut of two alignments and assess the alignment quality along the seam-cut. The assessment of the alignment quality mainly considered the color and structural differences between overlapping regions of two images,and the structure was reflected by a gradient. Then,feathering approach was utilized to blend the two images of the best alignment. Result To test our stitching algorithm,23 pairs of pictures,which cover commonly seen natural and man-made scenes,were captured. In addition,we conducted several experiments on publicly published data provided by recent related works. The experimental results demonstrated that the alignment accuracy of our method exceeded 95%,and the ratio of perspective distortion was lower than 17%. Compared with recent state-of-the-art methods,our method's alignment accuracy was higher by 3%,and the ratio of perspective distortion was lower by 73%. Therefore,our method exhibits a better performance in handling image stitching with a large parallax,and the stitching result is authentic and natural. Conclusion This paper presented a hybrid transformation for aligning two images that combines line constraint with MLS. In addition,an alignment quality evaluation rule was introduced by computing the weighted differences of the points along the stitching seam-cut and the remaining points in the overlapping region.As the proposed method can balance alignment accuracy and structure preservation,it can address the misalignment issues easily caused by current stitching approaches for parallax images and effectively reduce stitching artifacts,such as ghosting and distortion.
引文
[1] Szeliski R. Image alignment and stitching:a tutorial[J]. Foundations and Trends in Computer Graphics and Vision,2006,2(1):1-104.[DOI:10. 1561/0600000009]
[2] Lowe D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision,2004,60(2):91-110.[DOI:10. 1023/b:visi. 0000029664. 99615.94]
[3] Fischler M A,Bolles R C. Random sample consensus:a paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM,1981,24(6):381-395.[DOI:10. 1145/358669. 358692]
[4] Brown M,Szeliski R,Winder S. Multi-image matching using multi-scale oriented patches[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego,CA:IEEE,2005:510-517.[DOI:10. 1109/cvpr. 2005. 235]
[5] Mc Lauchlan P F,Jaenicke A. Image mosaicing using sequential bundle adjustment[J]. Image and Vision Computing,2002,20(9-10):751-759.[DOI:10. 1016/s0262-8856(02)00064-1]
[6] Pérez P,Gangnet M,Blake A. Poisson image editing[J]. ACM Transactions on Graphics,2003,22(3):313-318.[DOI:10.1145/882262. 882269]
[7] Gu Y,Zhou Y,Ren G,et al. Image stitching by combining optimal seam and multi-resolution fusion[J]. Journal of Image and Graphics,2017,22(6):842-851.[谷雨,周阳,任刚,等.结合最佳缝合线和多分辨率融合的图像拼接[J].中国图象图形学报,2017,22(6):842-851.][DOI:10. 11834/jig.160638]
[8] Brown M,Lowe D G. Automatic panoramic image stitching using invariant features[J]. International Journal of Computer Vision,2007,74(1):59-73.[DOI:10. 1007/s11263-006-0002-3]
[9] Gao J H,Kim S J,Brown M S. Constructing image panoramas using dual-homography warping[C]//Proceedings of CVPR2011. Colorado Springs,CO,USA,USA:IEEE,2011:49-56.[DOI:10. 1109/cvpr. 2011. 5995433]
[10] Gao J H,Li Y,Chin T J,et al. Seam-driven image stitching[C]//Proceedings of EuroGraphics. The Eurographics Association,2013:45-48.[DOI:10. 2312/conf/EG2013/short/045-048]
[11] Cao S X,Jiang J,Zhang G J,et al. Multi-scale image mosaic using features from edge[J]. Journal of Computer Research and Development,2011,48(9):1788-1793.[曹世翔,江洁,张广军,等.边缘特征点的多分辨率图像拼接[J].计算机研究与发展,2011,48(9):1788-1793]
[12] Lin W Y,Liu S Y,Matsushita Y,et al. Smoothly varying affine stitching[C]//Proceedings of CVPR 2011. Colorado Springs,CO,USA,USA:IEEE,2011:345-352.[DOI:10. 1109/cvpr. 2011. 5995314]
[13] Zaragoza J,Chin T J,Brown M S,et al. As-projective-as-possible image stitching with moving DLT[C]//Proceedings of 2013IEEE Conference on Computer Vision and Pattern Recognition.Portland,OR,USA:IEEE,2013:2339-2346.[DOI:10.1109/cvpr. 2013. 303]
[14] Schaefer S,Mc Phail T,Warren J. Image deformation using moving least squares[J]. ACM Transactions on Graphics,2006,25(3):533-540.[DOI:10. 1145/1141911. 1141920]
[15] Zhang F,Liu F. Parallax-tolerant image stitching[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus,OH,USA:IEEE,2014:3262-3269.[DOI:10. 1109/cvpr. 2014. 423]
[16] Lin K M,Jiang N J,Cheong L F,et al. SEAGULL:Seamguided local alignment for parallax-tolerant image stitching[C]//Leibe B,Matas J,Sebe N,et al. Computer Vision-ECCV 2016.ECCV 2016. Cham:Springer,2016:370-385.[DOI:10.1007/978-3-319-46487-9_23]
[17] Chu D D,Li H S. Parallax image stitching based on moving least square method[J]. Computer Applications and Software,2017,34(8):231-235.[楚东东,李海晟.基于移动最小二乘法的视差图像拼接[J].计算机应用与软件,2017,34(8):231-235.][DOI:10. 3969/j. issn. 1000-386x. 2017. 08. 041]
[2] Lowe D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision,2004,60(2):91-110.[DOI:10. 1023/b:visi. 0000029664. 99615.94]
[3] Fischler M A,Bolles R C. Random sample consensus:a paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM,1981,24(6):381-395.[DOI:10. 1145/358669. 358692]
[4] Brown M,Szeliski R,Winder S. Multi-image matching using multi-scale oriented patches[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego,CA:IEEE,2005:510-517.[DOI:10. 1109/cvpr. 2005. 235]
[5] Mc Lauchlan P F,Jaenicke A. Image mosaicing using sequential bundle adjustment[J]. Image and Vision Computing,2002,20(9-10):751-759.[DOI:10. 1016/s0262-8856(02)00064-1]
[6] Pérez P,Gangnet M,Blake A. Poisson image editing[J]. ACM Transactions on Graphics,2003,22(3):313-318.[DOI:10.1145/882262. 882269]
[7] Gu Y,Zhou Y,Ren G,et al. Image stitching by combining optimal seam and multi-resolution fusion[J]. Journal of Image and Graphics,2017,22(6):842-851.[谷雨,周阳,任刚,等.结合最佳缝合线和多分辨率融合的图像拼接[J].中国图象图形学报,2017,22(6):842-851.][DOI:10. 11834/jig.160638]
[8] Brown M,Lowe D G. Automatic panoramic image stitching using invariant features[J]. International Journal of Computer Vision,2007,74(1):59-73.[DOI:10. 1007/s11263-006-0002-3]
[9] Gao J H,Kim S J,Brown M S. Constructing image panoramas using dual-homography warping[C]//Proceedings of CVPR2011. Colorado Springs,CO,USA,USA:IEEE,2011:49-56.[DOI:10. 1109/cvpr. 2011. 5995433]
[10] Gao J H,Li Y,Chin T J,et al. Seam-driven image stitching[C]//Proceedings of EuroGraphics. The Eurographics Association,2013:45-48.[DOI:10. 2312/conf/EG2013/short/045-048]
[11] Cao S X,Jiang J,Zhang G J,et al. Multi-scale image mosaic using features from edge[J]. Journal of Computer Research and Development,2011,48(9):1788-1793.[曹世翔,江洁,张广军,等.边缘特征点的多分辨率图像拼接[J].计算机研究与发展,2011,48(9):1788-1793]
[12] Lin W Y,Liu S Y,Matsushita Y,et al. Smoothly varying affine stitching[C]//Proceedings of CVPR 2011. Colorado Springs,CO,USA,USA:IEEE,2011:345-352.[DOI:10. 1109/cvpr. 2011. 5995314]
[13] Zaragoza J,Chin T J,Brown M S,et al. As-projective-as-possible image stitching with moving DLT[C]//Proceedings of 2013IEEE Conference on Computer Vision and Pattern Recognition.Portland,OR,USA:IEEE,2013:2339-2346.[DOI:10.1109/cvpr. 2013. 303]
[14] Schaefer S,Mc Phail T,Warren J. Image deformation using moving least squares[J]. ACM Transactions on Graphics,2006,25(3):533-540.[DOI:10. 1145/1141911. 1141920]
[15] Zhang F,Liu F. Parallax-tolerant image stitching[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus,OH,USA:IEEE,2014:3262-3269.[DOI:10. 1109/cvpr. 2014. 423]
[16] Lin K M,Jiang N J,Cheong L F,et al. SEAGULL:Seamguided local alignment for parallax-tolerant image stitching[C]//Leibe B,Matas J,Sebe N,et al. Computer Vision-ECCV 2016.ECCV 2016. Cham:Springer,2016:370-385.[DOI:10.1007/978-3-319-46487-9_23]
[17] Chu D D,Li H S. Parallax image stitching based on moving least square method[J]. Computer Applications and Software,2017,34(8):231-235.[楚东东,李海晟.基于移动最小二乘法的视差图像拼接[J].计算机应用与软件,2017,34(8):231-235.][DOI:10. 3969/j. issn. 1000-386x. 2017. 08. 041]
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |