基于Hadoop的云平台的实现与基准测试
|
摘要
Hadoop是google云计算理论的开源实现,作为软件系统中间件的软件框架,它可以对大量数据进行分布式处理。通过Haddop,用户可以在不了解分布式底层细节的情况下开发分布式程序,充分利用集群的威力进行高速运算和存储。通过使用VMware虚拟机技术实现在单机上配置多个虚拟计算机节点,从而进行集群测试;在虚拟节点上安装Ubuntu操作系统作为Hadoop的操作系统支持;同时,利用Xmanager软件,以及配置局域网中宿主机与虚拟机、远程控制机的网络参数,实现对虚拟节点的远程控制;在已经安装好Ubuntu操作系统的各个虚拟节点上安装Hadoop、java-JDK等软件,并进行相关的参数设置,实现在虚拟机上各个虚拟节点的Hadoop完全分布式平台。最后在Hadoop平台上,使用Hadoop软件自带的基准测试程序包对平台进行4个Hadoop的经典基准测试。同时,每个测试都会加载不同的数据量及负载进行多次实验,通过比较在不同的负载下Hadoop的基准测试结果,测试Hadoop平台的相关性能,并分析负载及数据量的变化对Hadoop平台性能的影响。
Hadoop is an open source implementation of Google cloud computing theory.Hadoop,as a software framework,can process a large amount with distributed processing.By using Hadoop,users can develop a distributed program,and make full use of the cluster to carry out high-speed computing and storage,even without knowing the details of the distributed layer.In this paper,we use VMware virtual machine technology to realize the configuration of multiple virtual computer nodes on a single machine,and then use it to carry out the cluster test.As the Hadoop's operating system,Ubuntu operating system is installed on virtual nodes;meanwhile,with the help of Xmanager software,host computer and virtual machine,as well as referring to network parameters from the remote controller,the remote control of virtual nodes is realized.In the Ubuntu operating system,Hadoop,java-JDK,and other software are installed on each virtual node.The relevant parameter configuration is set up to realize the complete distributed platform of Hadoop in the virtual machine.Finally,the benchmark program package,included with the Hadoop software,do 4 Hadoop benchmark tests were carried out on the platform.In the same time,each test will load a number of different data and load for many experiments.By comparing the results of the benchmark test from different load Hadoop,performance of Hadoop platform was tested,and the impact of load and data on the Hadoop platform also is analyzed.
Hadoop is an open source implementation of Google cloud computing theory.Hadoop,as a software framework,can process a large amount with distributed processing.By using Hadoop,users can develop a distributed program,and make full use of the cluster to carry out high-speed computing and storage,even without knowing the details of the distributed layer.In this paper,we use VMware virtual machine technology to realize the configuration of multiple virtual computer nodes on a single machine,and then use it to carry out the cluster test.As the Hadoop's operating system,Ubuntu operating system is installed on virtual nodes;meanwhile,with the help of Xmanager software,host computer and virtual machine,as well as referring to network parameters from the remote controller,the remote control of virtual nodes is realized.In the Ubuntu operating system,Hadoop,java-JDK,and other software are installed on each virtual node.The relevant parameter configuration is set up to realize the complete distributed platform of Hadoop in the virtual machine.Finally,the benchmark program package,included with the Hadoop software,do 4 Hadoop benchmark tests were carried out on the platform.In the same time,each test will load a number of different data and load for many experiments.By comparing the results of the benchmark test from different load Hadoop,performance of Hadoop platform was tested,and the impact of load and data on the Hadoop platform also is analyzed.
引文
[1]WHITE T.Hadoop权威指南[M].3版.北京:清华大学出版社,2014.
[2]ARUN C M,VAVILAPALLI V K,EADLINE D,et al.Hadoop YARN权威指南[M].北京:机械工业出版社,2015.
[3]HOLMES A.Hadoop硬实战[M].北京:电子工业出版社,2015.
[4]崔文斌,牟少敏,王云诚,等.Hadoop大数据平台的搭建与测试[J].山东农业大学学报(自然科学版),2013,44(4):550-555.
[5]张新玲.Hadoop平台基准性能测试研究[J].软件导刊,2015,14(1):30-32.
[6]张岩,郭松,赵国海.基于Hadoop的云计算试验平台搭建研究[J].沈阳师范大学学报(自然科学版),2013,31(1):85-89.
[7]李三淼,李龙澍.Hadoop中处理小文件的四种方法的性能分析[EB/OL].[2015-11-13].http:∥www.cnki.net/kcms/detail/11.2127.TP.20141230.1656.014.html.
[8]管莹,李佳音.基于Hadoop的实验室数据管理系统的实现[J].电脑编程技巧与维护,2014(4):39-40.
[9]潘慧,朱信忠,赵建民,等.基于Hadoop云测试体系架构的设计[J].计算机工程与科学,2013,35(10):73-78.
[10]徐东.基于Hadoop的云教学资源平台设计与实现[D].北京:北京交通大学,2014.
[11]张洪磊.基于Hadoop的医院数据中心系统设计与实现[D].杭州:浙江大学,2014.
[12]刘源.基于Hadoop的海量数据分析系统设计与实现[D].大连:大连理工大学,2013.
[13]张朋,常静,范福玲.云计算与高等院校教育信息化变革研究[J].科教文汇(上旬刊),2012(9):24-25.
[14]李丹.校园分布式系统集成管理平台的设计与实现[J].电子测试,2013(13):3-4.
[15]潘丹,甘宏.基于异构集群环境下Hadoop作业调度优化的研究[J].科技广场,2015(9):16-19.
[16]尹颖,林庆,林涵阳.HDFS中高效存储小文件的方法[J].计算机工程与设计,2015(2):406-409.
[2]ARUN C M,VAVILAPALLI V K,EADLINE D,et al.Hadoop YARN权威指南[M].北京:机械工业出版社,2015.
[3]HOLMES A.Hadoop硬实战[M].北京:电子工业出版社,2015.
[4]崔文斌,牟少敏,王云诚,等.Hadoop大数据平台的搭建与测试[J].山东农业大学学报(自然科学版),2013,44(4):550-555.
[5]张新玲.Hadoop平台基准性能测试研究[J].软件导刊,2015,14(1):30-32.
[6]张岩,郭松,赵国海.基于Hadoop的云计算试验平台搭建研究[J].沈阳师范大学学报(自然科学版),2013,31(1):85-89.
[7]李三淼,李龙澍.Hadoop中处理小文件的四种方法的性能分析[EB/OL].[2015-11-13].http:∥www.cnki.net/kcms/detail/11.2127.TP.20141230.1656.014.html.
[8]管莹,李佳音.基于Hadoop的实验室数据管理系统的实现[J].电脑编程技巧与维护,2014(4):39-40.
[9]潘慧,朱信忠,赵建民,等.基于Hadoop云测试体系架构的设计[J].计算机工程与科学,2013,35(10):73-78.
[10]徐东.基于Hadoop的云教学资源平台设计与实现[D].北京:北京交通大学,2014.
[11]张洪磊.基于Hadoop的医院数据中心系统设计与实现[D].杭州:浙江大学,2014.
[12]刘源.基于Hadoop的海量数据分析系统设计与实现[D].大连:大连理工大学,2013.
[13]张朋,常静,范福玲.云计算与高等院校教育信息化变革研究[J].科教文汇(上旬刊),2012(9):24-25.
[14]李丹.校园分布式系统集成管理平台的设计与实现[J].电子测试,2013(13):3-4.
[15]潘丹,甘宏.基于异构集群环境下Hadoop作业调度优化的研究[J].科技广场,2015(9):16-19.
[16]尹颖,林庆,林涵阳.HDFS中高效存储小文件的方法[J].计算机工程与设计,2015(2):406-409.