基于自适应互信息的红外/深度双摄像机时空配准
|
摘要
红外、深度摄像机构成的双摄像机系统应用广泛,摄像机间的空间配准和时序配准影响着图像信息的整合。空间配准方面,利用不同材料的不同辐射率设计了标定板,解决了传统标定板无法标定红外、深度摄像机的问题。时序配准方面,利用搜索窗口解决了红外摄像机非一致性校正的问题;采用动态改变搜索窗口大小的方法消除了图像序列累积时间偏差;利用求取互信息的方法得到时序配准的红外、深度图像。通过实验验证了本文方法的有效性。
The dual camera system composed of the infrared camera and the depth camera has a widely application and the accuracy of spatial registration and temporal registration directly affects the integration of image information. However, the traditional calibration board cannot calibrate the infrared camera and the depth camera about spatial registration. In general, different materials may have different radiant rates. In order to deal with the spatial registration of the infrared camera and the depth camera, this paper designs a novel calibration board with different materials to achieve spatial registration. According to different radiant rates, the proposed calibration boards can solve the above shortcomings encountered in the traditional calibration board. In the temporal registration, many problems had to be taken into consideration. Different cameras might have different frame rates. Moreover, the frame rate may also fluctuate as the camera works. The precision difference of internal clock will bring cumulative time deviations to the camera image sequences. Furthermore, in order to maintain the accuracy measure signal of the temperature, the infrared camera will conduct regular non-uniformity corrections,which may yield about 1—2 s the time interruption of the camera. All of those above problems have impact on the temporal registration of cameras. To this end, this paper adopts adaptive search window to obtain the depth images and infrared images such that the effect of non-uniformity corrections from the infrared camera can be solved. By adaptively changing the size of the search window, the impact of the cumulative time deviations can be eliminated. Moreover, by calculating the mutual information between depth images and infrared images, we can obtain the depth images and infrared images that are temporal registration. Finally, the effectiveness of the proposed method is verified by experiments, which can effectively solve the temporal registration problem of different cameras with different frame rates.
The dual camera system composed of the infrared camera and the depth camera has a widely application and the accuracy of spatial registration and temporal registration directly affects the integration of image information. However, the traditional calibration board cannot calibrate the infrared camera and the depth camera about spatial registration. In general, different materials may have different radiant rates. In order to deal with the spatial registration of the infrared camera and the depth camera, this paper designs a novel calibration board with different materials to achieve spatial registration. According to different radiant rates, the proposed calibration boards can solve the above shortcomings encountered in the traditional calibration board. In the temporal registration, many problems had to be taken into consideration. Different cameras might have different frame rates. Moreover, the frame rate may also fluctuate as the camera works. The precision difference of internal clock will bring cumulative time deviations to the camera image sequences. Furthermore, in order to maintain the accuracy measure signal of the temperature, the infrared camera will conduct regular non-uniformity corrections,which may yield about 1—2 s the time interruption of the camera. All of those above problems have impact on the temporal registration of cameras. To this end, this paper adopts adaptive search window to obtain the depth images and infrared images such that the effect of non-uniformity corrections from the infrared camera can be solved. By adaptively changing the size of the search window, the impact of the cumulative time deviations can be eliminated. Moreover, by calculating the mutual information between depth images and infrared images, we can obtain the depth images and infrared images that are temporal registration. Finally, the effectiveness of the proposed method is verified by experiments, which can effectively solve the temporal registration problem of different cameras with different frame rates.
引文
[1]BUTKIEWICZ T.Low-cost coastal mapping using Kinect v2 time-of-flight cameras[C]//2014 Oceans-St.John's.Canada:IEEE,2015:1-9.
[2]GONZALEZ J H,RODRíGUEZ G P,MARTíNEZ S J,et al.Metrological comparison between Kinect I and Kinect II sensors[J].Measurement,2015,70:21-26.
[3]BORRMANN D,NüCHTER A,DJAKULOVI'C M,et al.The project thermal mapper-thermal 3D mapping of indoor environments for saving energy[J].IFAC Symposium on Robot Control,2012,45(22):31-38.
[4]SCHONAUER C,VONACH E,GERSTWEILER G,et al.3D building reconstruction and thermal mapping in fire brigade operations[C]//Augmented Human International Conference.Stuttgart,Germany:IEEE,2013:202-205.
[5]ZHANG Z.Flexible camera calibration by viewing a plane from unknown orientations[C]//International Conference on Computer Vision.Kerkyra,Greece:IEEE,1999:666-673.
[6]LUSSIER J T,THRUN S.Automatic calibration of RGBDand thermal cameras[C]//International Conference on Intelligent Robots and Systems.Chicago,USA:IEEE,2014:451-458.
[7]HAM Y,GOLPARVAR F M.An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery[J].Advanced Engineering Informatics,2013,27(3):395-409.
[8]KIM N,CHOI Y,HWANG S.Geometrical calibration of multispectral calibration[C]//International Conference on Ubiquitous Robots and Ambient Intelligence.Goyang,Korea:IEEE,2015:384-385.
[9]GSCHWANDTNER M,KWITT R,UHL A.Infrared camera calibration for dense depth map construction[C]//Intelligent Vehicles Symposium.Baden-Baden,Germany:IEEE,2011:857-862.
[10]REHDER J,SIEGWART R,FURGALE P.A general approach to spatiotemporal calibration in multisensor systems[J].Transactions on Robotics,2016,32(2):383-398.
[11]HWANG S,CHOI Y,KIM N.Low-cost synchronization for multispectral cameras[C]//International Conference on Ubiquitous Robots and Ambient Intelligence.Goyang,Korea:IEEE,2015:435-436.
[12]VIDAS S,MOGHADAM P,BOSSE M.3D thermal mapping of building interiors using an RGB-D and thermal camera[C]//International Conference on Robotics and Automation.Karlsruhe,Germany:IEEE,2013:2311-2318.
[13]BAAR J V,BEARDSLEY P,POLLEFEYS M.Sensor fusion for depth estimation,including TOF and thermal sensors[C]//International Conference on 3D Imaging,Modeling,Processing,Visualization and Transmission.Zurich,Switzerland:IEEE,2012:472-478.
[14]MILLER S,TEICHMAN A,THRUN S.Unsupervised extrinsic calibration of depth sensors in dynamic scenes[C]//Intelligent Robots and Systems.Tokyo,Japan:IEEE.2013:2695-2702.
[15]LEVINSON J,THRUN S.Automatic online calibration of cameras and lasers[C]//Proceedings of the Robotics:Science and Systems.Berlin,Germany:IEEE,2013:24-28
[16]RANGEL J,SOLDAN S,KROLL A.3D Thermal imaging:Fusion of thermography and depth cameras[C]//International Conference on Quantitative Infrared Thermography.Bordeaux:IEEE,2014:325-334.
[17]HAN S,GU X,GU X.An accurate calibration method of a multi camera system[C]//International Conference on Life System Modeling and Simulation.Nanjing:Springer,2017:491-501.
[2]GONZALEZ J H,RODRíGUEZ G P,MARTíNEZ S J,et al.Metrological comparison between Kinect I and Kinect II sensors[J].Measurement,2015,70:21-26.
[3]BORRMANN D,NüCHTER A,DJAKULOVI'C M,et al.The project thermal mapper-thermal 3D mapping of indoor environments for saving energy[J].IFAC Symposium on Robot Control,2012,45(22):31-38.
[4]SCHONAUER C,VONACH E,GERSTWEILER G,et al.3D building reconstruction and thermal mapping in fire brigade operations[C]//Augmented Human International Conference.Stuttgart,Germany:IEEE,2013:202-205.
[5]ZHANG Z.Flexible camera calibration by viewing a plane from unknown orientations[C]//International Conference on Computer Vision.Kerkyra,Greece:IEEE,1999:666-673.
[6]LUSSIER J T,THRUN S.Automatic calibration of RGBDand thermal cameras[C]//International Conference on Intelligent Robots and Systems.Chicago,USA:IEEE,2014:451-458.
[7]HAM Y,GOLPARVAR F M.An automated vision-based method for rapid 3D energy performance modeling of existing buildings using thermal and digital imagery[J].Advanced Engineering Informatics,2013,27(3):395-409.
[8]KIM N,CHOI Y,HWANG S.Geometrical calibration of multispectral calibration[C]//International Conference on Ubiquitous Robots and Ambient Intelligence.Goyang,Korea:IEEE,2015:384-385.
[9]GSCHWANDTNER M,KWITT R,UHL A.Infrared camera calibration for dense depth map construction[C]//Intelligent Vehicles Symposium.Baden-Baden,Germany:IEEE,2011:857-862.
[10]REHDER J,SIEGWART R,FURGALE P.A general approach to spatiotemporal calibration in multisensor systems[J].Transactions on Robotics,2016,32(2):383-398.
[11]HWANG S,CHOI Y,KIM N.Low-cost synchronization for multispectral cameras[C]//International Conference on Ubiquitous Robots and Ambient Intelligence.Goyang,Korea:IEEE,2015:435-436.
[12]VIDAS S,MOGHADAM P,BOSSE M.3D thermal mapping of building interiors using an RGB-D and thermal camera[C]//International Conference on Robotics and Automation.Karlsruhe,Germany:IEEE,2013:2311-2318.
[13]BAAR J V,BEARDSLEY P,POLLEFEYS M.Sensor fusion for depth estimation,including TOF and thermal sensors[C]//International Conference on 3D Imaging,Modeling,Processing,Visualization and Transmission.Zurich,Switzerland:IEEE,2012:472-478.
[14]MILLER S,TEICHMAN A,THRUN S.Unsupervised extrinsic calibration of depth sensors in dynamic scenes[C]//Intelligent Robots and Systems.Tokyo,Japan:IEEE.2013:2695-2702.
[15]LEVINSON J,THRUN S.Automatic online calibration of cameras and lasers[C]//Proceedings of the Robotics:Science and Systems.Berlin,Germany:IEEE,2013:24-28
[16]RANGEL J,SOLDAN S,KROLL A.3D Thermal imaging:Fusion of thermography and depth cameras[C]//International Conference on Quantitative Infrared Thermography.Bordeaux:IEEE,2014:325-334.
[17]HAN S,GU X,GU X.An accurate calibration method of a multi camera system[C]//International Conference on Life System Modeling and Simulation.Nanjing:Springer,2017:491-501.