基于大数据的无人机云交换平台统计分析技术研究
|
摘要
随着无人机在各行业应用的迅速普及和大规模应用,民航局监管的无人机数量和无人机运行数据呈现海量增长,在给政府和行业监管带来便利的同时,也给传统的数据分析方法带来了极大的挑战。本文阐述了民航局无人机云交换平台的运行大数据的现状,分析了传统无人机运行数据统计分析存在的技术瓶颈,提出了基于大数据进行无人机运行数据统计分析方法,建立了无人机云数据交换平台大数据统计和分析系统的设计框架,并概述了如何基于Apache Spark和Cassandra数据库的大数据分析方法将无人机云交换平台生产的海量数据快速处理、存储、统计和分析的研究思路和方法,以及实现各类无人机运行特征分析的情况。目前基于本文的研究成果不仅可以满足行业主管部门对无人机运行数据统计的需求,还对掌握和了解中国无人机运行特征,探索更合适的无人机管理措施和健全无人机监管体制机制,有非常重要的意义。
After several years of development, light and small unmanned aircraft systems(UASs) have been widely used in various industries both in China and many other countries. However, the UASs have many models, with scattered equipments and no systematic management. Meanwhile, some safety issues exist. This urgently requires the relevant regulatory authorities to regulate, supervise, and maintain safe flight operations by taking their operating rules and characteristics into account. In order to standardize the operation of light and small civil UASs across the country and promote the industry development, the Civil Aviation Administration of China has issued the "Provisions for the Operation of Light and Small Unmanned Aircraft(for Trial Implementation)" advisory circular and the "Specification for Interface Data of Unmanned Aircraft System Cloud System". The UAS cloud data exchange platform was developed in 2016, and the data sharing of multiple UAS cloud systems in China was realized. With this platform the UASs registered in different UAS cloud systems are visible to each other in the same airspace, which improved the flight safety of China's low-altitude airspace. However, with the rapid development of the application of the unmanned aerial vehicle(UAV) industry,the number of UAVs supervised by the Civil Aviation Authority and the data on the operation of the UAVs has increased dramatically, which has also brought great challenges to the traditional data management methods. In this paper, we will describe the current situation of the operation of big data from the UAS cloud data exchange platform in China followed by discussion on the technical bottle necks in the statistical analysis of the operation data of the traditional UAS. Then we will propose a statistical analysis method for the UAS operation data, and establish a framework of statistical analysis of big data from the cloud data exchange platform. In the end, we will outline how to use Apache Spark and Cassandra database to quickly process, store, count, and analyze the massive data generated by the UAS cloud data exchange platform. The research situation of implementing various statistical index algorithms based on the platform is introduced in detail. This research not only improves the efficiency of statistical analysis of UAS operation data, but also provides the operation management rules of China's light and small UASs from multiple dimensions. sWe highlight that the UAS has significant operational characteristics, which are different from general and transportation aviations. This paper provides reference for government and industry decision-making, which has strong practical significance.
After several years of development, light and small unmanned aircraft systems(UASs) have been widely used in various industries both in China and many other countries. However, the UASs have many models, with scattered equipments and no systematic management. Meanwhile, some safety issues exist. This urgently requires the relevant regulatory authorities to regulate, supervise, and maintain safe flight operations by taking their operating rules and characteristics into account. In order to standardize the operation of light and small civil UASs across the country and promote the industry development, the Civil Aviation Administration of China has issued the "Provisions for the Operation of Light and Small Unmanned Aircraft(for Trial Implementation)" advisory circular and the "Specification for Interface Data of Unmanned Aircraft System Cloud System". The UAS cloud data exchange platform was developed in 2016, and the data sharing of multiple UAS cloud systems in China was realized. With this platform the UASs registered in different UAS cloud systems are visible to each other in the same airspace, which improved the flight safety of China's low-altitude airspace. However, with the rapid development of the application of the unmanned aerial vehicle(UAV) industry,the number of UAVs supervised by the Civil Aviation Authority and the data on the operation of the UAVs has increased dramatically, which has also brought great challenges to the traditional data management methods. In this paper, we will describe the current situation of the operation of big data from the UAS cloud data exchange platform in China followed by discussion on the technical bottle necks in the statistical analysis of the operation data of the traditional UAS. Then we will propose a statistical analysis method for the UAS operation data, and establish a framework of statistical analysis of big data from the cloud data exchange platform. In the end, we will outline how to use Apache Spark and Cassandra database to quickly process, store, count, and analyze the massive data generated by the UAS cloud data exchange platform. The research situation of implementing various statistical index algorithms based on the platform is introduced in detail. This research not only improves the efficiency of statistical analysis of UAS operation data, but also provides the operation management rules of China's light and small UASs from multiple dimensions. sWe highlight that the UAS has significant operational characteristics, which are different from general and transportation aviations. This paper provides reference for government and industry decision-making, which has strong practical significance.
引文
[1]王万国,田兵,刘越,等.基于RCNN的无人机巡检图像电力小部件识别研究[J].地球信息科学学报,2017,19(2):256-263.[Wang W G,Tian B,Liu Y,et al.Study on the electrical devices detection in UAV images based on region based convolutional neural networks[J].Journal of Geo-information Science,2017,19(2):256-263.]
[2]Khan M A,Ectors W,Bellemans T,et al.Unmanned aerial vehicle-based traffic analysis:A case study for shockwave identification and flow parameters estimation at signalized intersections[J].Remote Sensing,2018,10(3):458-474.
[3]宋晓阳,黄耀欢,董东林,等.融合数字表面模型的无人机遥感影像城市土地利用分类[J].地球信息科学学报,2018,20(5):703-711.[Song X Y,Huang Y H,Dong D L,et al.Urban land use classification from UAV remote sensing images based on digital surface model[J].Journal of Geo-information Science,2018,20(5):703-711.]
[4]杨海军,黄耀欢.化工污染气体无人机遥感监测[J].地球信息科学学报,2015,17(10):1269-1274.[Yang H J,Huang Y H.Evaluating atmospheric pollution of chemical plant based on unmanned aircraft vehicle(UAV)[J].Journal of Geo-information Science,2015,17(10):1269-1274.]
[5]李德仁,李明.无人机遥感系统的研究进展与应用前景[J].武汉大学学报·信息科学版,2014,39(5):505-513.[Li D R,Li M.Research advance and application prospect of unmanned aerial vehicle remote sensing system[J].Geomatics and Information Science of Wuhan University,2014,39(5):505-513.]
[6]廖小罕,周成虎,苏奋振,等.无人机遥感众创时代[J].地球信息科学学报,2016,18(11):1439-1447.[Liao X H,Zhou C H,Su F Z,et al.The mass innovation era of UAVremote sensing[J].Journal of Geo-information Science,2016,18(11):1439-1447.]
[7]任丽艳,李英成,薛艳丽,等.基于北斗技术的无人机飞行监管系统开发与应用[J].国土资源遥感,2018,30(2):238-242.[Ren L Y,Li Y C,Xue Y L,et al.Development and application of the UAV flying supervisory system based on Beidou technology[J].Remote Sensing for Land&Resources,2018,30(2):238-242]
[8]高国柱.中国民用无人机监管制度研究[J].北京航空航天大学学报(社会科学版),2017,30(5):28-36.[Gao G Z.On China's regulatory regime for civil unmanned aerial vehicle[J].Journal of Beijing University of Aeronautics and Astronautics Social Sciences Edition,2017,30(5):28-36.]
[9]Spousta R,Chan S.Hold the drones:Fostering the development of big data paradigms through regulatory frameworks[J].Journal of Communication and Computer,2015,(12):135-145.
[10]中国民用航空局.AC-91-31.轻小无人机运行管理规定(试行)[S].2015:5.[CAAC.AC-91-31.Provisions for the operation of light and small unmanned aircraft(for trial implementation)[S].2015:5.]
[11]中国民用航空局.MH/T2009-2017.无人机云系统接口数据规范[S].2017:2-6.[CAAC.MH/T2009-2017.Specification for interface data of unmanned aircraft system cloud system[S].2017:2-6.]
[12]孟小峰,慈祥.大数据管理:概念、技术与挑战[J].计算机研究与发展,2013,50(1):146-169.[Meng X F,Ci X.Big data management:Concepts,techniques and challenges[J].Journal of Computer Research and Development,2013,50(1):146-169.]
[13]Big data analytics-a research and innovation agenda for sweden[EB/OL].https://www.vinnova.se/globalassets/mikrosajter/strategiska-innovationsprogram/agendor/analaysav-stordata.pdf.
[14]李广健,化柏林.大数据分析与情报分析关系辨析[J].中国图书馆学报,2014,40(5):14-22.[Li G J,Hua B L.Relationship between big data analysis and intelligence analysis[J].Journal of Library Science in China,2014,40(5):14-22.]
[15]官思发,孟玺,李宗洁,等.大数据分析研究现状、问题与对策[J],情报工程,2015,34(5):98-104.[Guan S F,Meng X,Li Z J,et al.Big data study on the current situation,problems and countermeasures[J].Journal of Intelligence,2015,34(5):98-104.]
[16]孙飞,张霞,唐炉亮,等.基于GPS轨迹大数据的优质客源时空分布研究[J].地球信息科学学报,2015,17(3):329-335.[Sun F,Zhang X,Tang L L,et al.Temporal and spatial distribution of high efficiency passengers based on GPS trajectory big data[J].Journal of Geo-information Science,2015,17(3):329-335.]
[17]洪漪,赵栋祥,赵一鸣.大数据环境下的信息架构与数据模型[J].信息资源管理学报,2018,8(1):29-38.[Hong Y,Zhao D X,Zhao Y M.Information architecture and data modeling in a big data environment[J].Journal of Information Resources Management,2018,8(1):29-38.]
[18]夏俊鸾,等.Spark大数据处理技术[M].北京:电子工业出版社,2015.[Xia J Y,et al.Spark big data processing technology[M].Beijing:Publishing House of Electronics Industry,2015.]
[19]Benchmarking NoSQL databases:Cassandra vs.Mongo DB vs.HBase vs.Couchbase[EB/OL].https://www.datastax.com/nosql-databases/benchmarks-cassandra-vsmongodb-vs-hbase.
[20]任磊,杜一,马帅,等.大数据可视分析综述[J].软件学报,2014,25(9):1909-1936.[Ren L,Du Y,Ma S,et al.Visual analytics towards big data[J].Ruan Jian Xue Bao/Journal of Software,2014,25(9):1909-1936.]
[21]Atufaali S,Chauhan P,Potdar M B.Analysis of big data using geomesa[J].International Journal of Advance Research and Innovative Ideas in Education,2018,4(2):4604-4612.
[22]李雪丽,盛勇,兰小机.基于Spark的并行化出租车轨迹热点区域提取与分析[J].计算机科学与应用,2018,8(9):1482-1489.[Li X L,Sheng Y,Lan X J.Extraction and analysis of hotspot region of parallel taxi trajectory based on spark[J].Computer Science and Application,2018,8(9):1482-1489.]
[23]Kwak J,Park J H,Sung Y.Unmanned aerial vehicle flight point classification algorithm based on symmetric big data[J].Symmetry,2017,9(1):1-19.
[24]王胜开,徐志洁,张健钦,等.逆向热力图的绘制方法[J].地球信息科学学报,2018,20(4):515-522.[Wang S K,Xu Z J,Zhang J Q,et al.A reverse rendering method of Heatmap[J].Journal of Geo-information Science,2018,20(4):515-522.]
[25]陈卓然,黄翀,刘高焕,等.基于出租车GPS数据的居民就医时空特征分析[J].地球信息科学学报,2018,20(8):1111-1122.[Chen Z R,Huang C,Liu G H,et al.Analysis of spatial-temporal characteristics of resident travel for hospitals based on taxi GPS data[J].Journal of Geo-information Science,2018,20(8):1111-1122.]
[2]Khan M A,Ectors W,Bellemans T,et al.Unmanned aerial vehicle-based traffic analysis:A case study for shockwave identification and flow parameters estimation at signalized intersections[J].Remote Sensing,2018,10(3):458-474.
[3]宋晓阳,黄耀欢,董东林,等.融合数字表面模型的无人机遥感影像城市土地利用分类[J].地球信息科学学报,2018,20(5):703-711.[Song X Y,Huang Y H,Dong D L,et al.Urban land use classification from UAV remote sensing images based on digital surface model[J].Journal of Geo-information Science,2018,20(5):703-711.]
[4]杨海军,黄耀欢.化工污染气体无人机遥感监测[J].地球信息科学学报,2015,17(10):1269-1274.[Yang H J,Huang Y H.Evaluating atmospheric pollution of chemical plant based on unmanned aircraft vehicle(UAV)[J].Journal of Geo-information Science,2015,17(10):1269-1274.]
[5]李德仁,李明.无人机遥感系统的研究进展与应用前景[J].武汉大学学报·信息科学版,2014,39(5):505-513.[Li D R,Li M.Research advance and application prospect of unmanned aerial vehicle remote sensing system[J].Geomatics and Information Science of Wuhan University,2014,39(5):505-513.]
[6]廖小罕,周成虎,苏奋振,等.无人机遥感众创时代[J].地球信息科学学报,2016,18(11):1439-1447.[Liao X H,Zhou C H,Su F Z,et al.The mass innovation era of UAVremote sensing[J].Journal of Geo-information Science,2016,18(11):1439-1447.]
[7]任丽艳,李英成,薛艳丽,等.基于北斗技术的无人机飞行监管系统开发与应用[J].国土资源遥感,2018,30(2):238-242.[Ren L Y,Li Y C,Xue Y L,et al.Development and application of the UAV flying supervisory system based on Beidou technology[J].Remote Sensing for Land&Resources,2018,30(2):238-242]
[8]高国柱.中国民用无人机监管制度研究[J].北京航空航天大学学报(社会科学版),2017,30(5):28-36.[Gao G Z.On China's regulatory regime for civil unmanned aerial vehicle[J].Journal of Beijing University of Aeronautics and Astronautics Social Sciences Edition,2017,30(5):28-36.]
[9]Spousta R,Chan S.Hold the drones:Fostering the development of big data paradigms through regulatory frameworks[J].Journal of Communication and Computer,2015,(12):135-145.
[10]中国民用航空局.AC-91-31.轻小无人机运行管理规定(试行)[S].2015:5.[CAAC.AC-91-31.Provisions for the operation of light and small unmanned aircraft(for trial implementation)[S].2015:5.]
[11]中国民用航空局.MH/T2009-2017.无人机云系统接口数据规范[S].2017:2-6.[CAAC.MH/T2009-2017.Specification for interface data of unmanned aircraft system cloud system[S].2017:2-6.]
[12]孟小峰,慈祥.大数据管理:概念、技术与挑战[J].计算机研究与发展,2013,50(1):146-169.[Meng X F,Ci X.Big data management:Concepts,techniques and challenges[J].Journal of Computer Research and Development,2013,50(1):146-169.]
[13]Big data analytics-a research and innovation agenda for sweden[EB/OL].https://www.vinnova.se/globalassets/mikrosajter/strategiska-innovationsprogram/agendor/analaysav-stordata.pdf.
[14]李广健,化柏林.大数据分析与情报分析关系辨析[J].中国图书馆学报,2014,40(5):14-22.[Li G J,Hua B L.Relationship between big data analysis and intelligence analysis[J].Journal of Library Science in China,2014,40(5):14-22.]
[15]官思发,孟玺,李宗洁,等.大数据分析研究现状、问题与对策[J],情报工程,2015,34(5):98-104.[Guan S F,Meng X,Li Z J,et al.Big data study on the current situation,problems and countermeasures[J].Journal of Intelligence,2015,34(5):98-104.]
[16]孙飞,张霞,唐炉亮,等.基于GPS轨迹大数据的优质客源时空分布研究[J].地球信息科学学报,2015,17(3):329-335.[Sun F,Zhang X,Tang L L,et al.Temporal and spatial distribution of high efficiency passengers based on GPS trajectory big data[J].Journal of Geo-information Science,2015,17(3):329-335.]
[17]洪漪,赵栋祥,赵一鸣.大数据环境下的信息架构与数据模型[J].信息资源管理学报,2018,8(1):29-38.[Hong Y,Zhao D X,Zhao Y M.Information architecture and data modeling in a big data environment[J].Journal of Information Resources Management,2018,8(1):29-38.]
[18]夏俊鸾,等.Spark大数据处理技术[M].北京:电子工业出版社,2015.[Xia J Y,et al.Spark big data processing technology[M].Beijing:Publishing House of Electronics Industry,2015.]
[19]Benchmarking NoSQL databases:Cassandra vs.Mongo DB vs.HBase vs.Couchbase[EB/OL].https://www.datastax.com/nosql-databases/benchmarks-cassandra-vsmongodb-vs-hbase.
[20]任磊,杜一,马帅,等.大数据可视分析综述[J].软件学报,2014,25(9):1909-1936.[Ren L,Du Y,Ma S,et al.Visual analytics towards big data[J].Ruan Jian Xue Bao/Journal of Software,2014,25(9):1909-1936.]
[21]Atufaali S,Chauhan P,Potdar M B.Analysis of big data using geomesa[J].International Journal of Advance Research and Innovative Ideas in Education,2018,4(2):4604-4612.
[22]李雪丽,盛勇,兰小机.基于Spark的并行化出租车轨迹热点区域提取与分析[J].计算机科学与应用,2018,8(9):1482-1489.[Li X L,Sheng Y,Lan X J.Extraction and analysis of hotspot region of parallel taxi trajectory based on spark[J].Computer Science and Application,2018,8(9):1482-1489.]
[23]Kwak J,Park J H,Sung Y.Unmanned aerial vehicle flight point classification algorithm based on symmetric big data[J].Symmetry,2017,9(1):1-19.
[24]王胜开,徐志洁,张健钦,等.逆向热力图的绘制方法[J].地球信息科学学报,2018,20(4):515-522.[Wang S K,Xu Z J,Zhang J Q,et al.A reverse rendering method of Heatmap[J].Journal of Geo-information Science,2018,20(4):515-522.]
[25]陈卓然,黄翀,刘高焕,等.基于出租车GPS数据的居民就医时空特征分析[J].地球信息科学学报,2018,20(8):1111-1122.[Chen Z R,Huang C,Liu G H,et al.Analysis of spatial-temporal characteristics of resident travel for hospitals based on taxi GPS data[J].Journal of Geo-information Science,2018,20(8):1111-1122.]
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |