网络化测试系统中并行数据处理架构的实现
|
摘要
网络化测试系统通常由智能仪器组成,采用传统的集中式数据处理结构一方面造成仪器内部计算资源的浪费,同时也对网络带宽带来较大的压力。为解决上述问题,提出了一种利用智能仪器内部计算资源进行并行数据处理的计算架构。提出了一种双层计算资源模型,在顶层使用PBS(portable batch system)作业管理系统,实现在网络化测试系统中的物理资源管理和计算节点分配;在底层使用隔离技术,基于Linux的Cgroups内核特性构建资源容器,实现节点内部的计算资源有效隔离。并行任务基于MPI非虚拟化并行计算平台实现,利用并行FFT算法对该计算模式进行了验证。实验结果表明,该计算架构具有良好的可行性和实用性。作为并行计算在网络化自动测试领域的拓展,具有很好的研究前景及实用价值。
Networked test system is generally composed of intelligent instruments. There is an allpervading problem in the traditional centralized data processing method that the structure results in potential wasting of computing resources on the intelligent instruments. Besides,data processing in centralized architecture brings greater pressure to the network bandwidth. In order to solve the problems above,a computing mode of parallel data processing is proposed,which can make full use of the internal calculation resources in the devices. In particular,a double-layer computing resource model is proposed. At the top level,a job management system called portable batch system( PBS) for cluster is used to manage the physical resources and carry calculation node allocation in net-centric automatic test system. In addition,Linux containers are built to achieve internal computing resource isolation effectively within nodes,where isolation technology is used based on control groups( C-groups) kernel features at the bottom. Parallel tasks are conducted based on message passing interface( MPI),which is a non-virtualized parallel computing platform. This is verified using parallel FFT algorithm. The experimental results show that the parallel data processing architecture is of high feasibility and utility. As an expansion on parallel computing in the field of net-centric automation test,this article has application prospect as well as practicability.
Networked test system is generally composed of intelligent instruments. There is an allpervading problem in the traditional centralized data processing method that the structure results in potential wasting of computing resources on the intelligent instruments. Besides,data processing in centralized architecture brings greater pressure to the network bandwidth. In order to solve the problems above,a computing mode of parallel data processing is proposed,which can make full use of the internal calculation resources in the devices. In particular,a double-layer computing resource model is proposed. At the top level,a job management system called portable batch system( PBS) for cluster is used to manage the physical resources and carry calculation node allocation in net-centric automatic test system. In addition,Linux containers are built to achieve internal computing resource isolation effectively within nodes,where isolation technology is used based on control groups( C-groups) kernel features at the bottom. Parallel tasks are conducted based on message passing interface( MPI),which is a non-virtualized parallel computing platform. This is verified using parallel FFT algorithm. The experimental results show that the parallel data processing architecture is of high feasibility and utility. As an expansion on parallel computing in the field of net-centric automation test,this article has application prospect as well as practicability.
引文
[1]李凤保,古天祥,沈艳.基于Ethernet的网络化测试技术研究[J].仪器仪表学报,2001,22(S2):261-263.LI Feng-bao,GU Tian-xiang,SHEN Yan. The Study ofEthernet-Based Networked Measurement Technology[J]. Journal of Instrument and Instrument,2001,22(S2):261-163.
[2]刘清文.网络化测试系统的实时性研究[D].太原:中北大学,2008.LIU Qing-wen. Research on Real-Time Property of Net-worked Test System[D]. Taiyuan:North University ofChina,2008.
[3]白剑斐,叶绿宽,杨文钧,等.集中式系统的分布式通用查询框架设计[J].计算机工程,2010,36(20):71-73.BAI Jian-fei,YE Lv-kuan,YANG Wen-yun,et al. Designof Distributed General Query Framework in CentralizedSystem[J]. Computer Engineering,2010,36(20):71-73.
[4] FOSTER I,ZHAO Y,RAICU I,et al. Cloud Computingand Grid Computing 360-Degree Compared[J]. GridComputing Environments Workshop Gce,2009,5:1-10.
[5]郭本俊,王鹏,陈高云,等.基于MPI的云计算模型[J].计算机工程,2009,35(24):84-86.GUO Ben-jun,WANG Peng,CHEN Gao-yun,et al.Cloud Computing Model Based on MPI[J]. ComputerEngineering,2009,35(24):84-86.
[6]郭羽成. MPI高性能云计算平台关键技术研究[D].武汉:武汉理工大学,2013.GUO Yu-cheng. Research on Key Technologies of theMPI-Based High-Performance Cloud Computing Platfo-rom[D]. Wuhan:Wuhan University of Technology,2013.
[7] NGUYEN N,BEIN D. Distributed MPI Cluster withDocker Swarm Mode[C]∥Computing and Communica-tion Workshop and Conference. IEEE,2017:1-7.
[8] MARIOTTI M,GERVASI O,VELLA F,et al. Strategiesand Systems Towards Grids and Clouds Integration:ADBMS-Based Solution[J]. Future Generation ComputerSystems,2018,88(11):718-729.
[9] PEINL R,HOLZSCHUHER F,PFITZER F. DockerCluster Management for the Cloud-Survey Results andOwn Solution[M]. Springer-Verlag New York,Inc.2016.
[10]段赫.基于LXC容器资源优化的研究与实现[D].广州:华南理工大学,2016.DUAN He. Research and Implementation of ImprovedTechnologies in Virtualization Technology Based onLinux Container Resource Utilization[D]. Guangzhou:South China University of Technology,2016
[11] STAPLES G. TORQUE Resource Manager[C]∥Pro-ceedings of the 2006 ACM/IEEE Conference on Super-computing. ACM,2006:8.
[12] JACKSON D B,SNELL Q,CLEMENT M J. Core Algo-rithms of the Maui Scheduler[J]. Lecture Notes in Com-puter Science,2001,2221(2221):87-102.
[13] PACHECO P S. Parallel Programming with MPI[M].Morgan Kaufmann,1997.
[14] GROPP W,LUSK E,DOSS N,et al. A high-Perform-ance,Portable Implementation of the MPI Message Pass-ing Interface Standard[J]. Parallel computing,1996,22(6):789-828.
[15] KAN T. Parallel Data Processing System Combining aSIMD Unit with a MIMD Unit and Sharing a CommonBus,Memory,and System Controller:U. S. Patent 5,355,508[P]. 1994-10-11.
[16]陈国良.并行计算:结构·算法·编程[M].北京:高等教育出版社,2011.CHEN Guo-liang. Parallel Computing:Structures,Algo-rithms,and Programming[M]. Beijing:Higher EducationPress,2011.
[2]刘清文.网络化测试系统的实时性研究[D].太原:中北大学,2008.LIU Qing-wen. Research on Real-Time Property of Net-worked Test System[D]. Taiyuan:North University ofChina,2008.
[3]白剑斐,叶绿宽,杨文钧,等.集中式系统的分布式通用查询框架设计[J].计算机工程,2010,36(20):71-73.BAI Jian-fei,YE Lv-kuan,YANG Wen-yun,et al. Designof Distributed General Query Framework in CentralizedSystem[J]. Computer Engineering,2010,36(20):71-73.
[4] FOSTER I,ZHAO Y,RAICU I,et al. Cloud Computingand Grid Computing 360-Degree Compared[J]. GridComputing Environments Workshop Gce,2009,5:1-10.
[5]郭本俊,王鹏,陈高云,等.基于MPI的云计算模型[J].计算机工程,2009,35(24):84-86.GUO Ben-jun,WANG Peng,CHEN Gao-yun,et al.Cloud Computing Model Based on MPI[J]. ComputerEngineering,2009,35(24):84-86.
[6]郭羽成. MPI高性能云计算平台关键技术研究[D].武汉:武汉理工大学,2013.GUO Yu-cheng. Research on Key Technologies of theMPI-Based High-Performance Cloud Computing Platfo-rom[D]. Wuhan:Wuhan University of Technology,2013.
[7] NGUYEN N,BEIN D. Distributed MPI Cluster withDocker Swarm Mode[C]∥Computing and Communica-tion Workshop and Conference. IEEE,2017:1-7.
[8] MARIOTTI M,GERVASI O,VELLA F,et al. Strategiesand Systems Towards Grids and Clouds Integration:ADBMS-Based Solution[J]. Future Generation ComputerSystems,2018,88(11):718-729.
[9] PEINL R,HOLZSCHUHER F,PFITZER F. DockerCluster Management for the Cloud-Survey Results andOwn Solution[M]. Springer-Verlag New York,Inc.2016.
[10]段赫.基于LXC容器资源优化的研究与实现[D].广州:华南理工大学,2016.DUAN He. Research and Implementation of ImprovedTechnologies in Virtualization Technology Based onLinux Container Resource Utilization[D]. Guangzhou:South China University of Technology,2016
[11] STAPLES G. TORQUE Resource Manager[C]∥Pro-ceedings of the 2006 ACM/IEEE Conference on Super-computing. ACM,2006:8.
[12] JACKSON D B,SNELL Q,CLEMENT M J. Core Algo-rithms of the Maui Scheduler[J]. Lecture Notes in Com-puter Science,2001,2221(2221):87-102.
[13] PACHECO P S. Parallel Programming with MPI[M].Morgan Kaufmann,1997.
[14] GROPP W,LUSK E,DOSS N,et al. A high-Perform-ance,Portable Implementation of the MPI Message Pass-ing Interface Standard[J]. Parallel computing,1996,22(6):789-828.
[15] KAN T. Parallel Data Processing System Combining aSIMD Unit with a MIMD Unit and Sharing a CommonBus,Memory,and System Controller:U. S. Patent 5,355,508[P]. 1994-10-11.
[16]陈国良.并行计算:结构·算法·编程[M].北京:高等教育出版社,2011.CHEN Guo-liang. Parallel Computing:Structures,Algo-rithms,and Programming[M]. Beijing:Higher EducationPress,2011.