MrHeter: improving MapReduce performance in heterogeneous environments

详细信息    查看全文

• 作者：Xiao Zhang ; Yanjun Wu ; Chen Zhao
• 关键词：MapReduce ; Heterogeneous cluster ; Scheduling ; Performance
• 刊名：Cluster Computing
• 出版年：2016
• 出版时间：December 2016
• 年：2016
• 卷：19
• 期：4
• 页码：1691-1701
• 全文大小：
• 刊物类别：Computer Science
• 刊物主题：Processor Architectures; Operating Systems; Computer Communication Networks;
• 出版者：Springer US
• ISSN：1573-7543
• 卷排序：19

文摘

As GPUs, ARM CPUs and even FPGAs are widely used in modern computing, a data center gradually develops towards the heterogeneous clusters. However, many well-known programming models such as MapReduce are designed for homogeneous clusters and have poor performance in heterogeneous environments. In this paper, we reconsider the problem and make four contributions: (1) We analyse the causes of MapReduce poor performance in heterogeneous clusters, and the most important one is unreasonable task allocation between nodes with different computing ability. (2) Based on this, we propose MrHeter, which separates MapReduce process into map-shuffle stage and reduce stage, then constructs optimization model separately for them and gets different task allocation \(ml_{ij}, mr_{ij}, r_{ij}\) for heterogeneous nodes based on computing ability.(3) In order to make it suitable for dynamic execution, we propose D-MrHeter, which includes monitor and feedback mechanism. (4) Finally, we prove that MrHeter and D-MrHeter can greatly decrease total execution time of MapReduce from 30 to 70 % in heterogeneous cluster comparing with original Hadoop, having better performance especially in the condition of heavy-workload and large-difference between nodes computing ability.