基于RDD非序列化本地存储的Spark存储性能优化
|
摘要
Spark框架被越来越多的企业用作大数据的计算框架,但随着现有服务器的可用内存资源增加,Spark并不能与新环境相匹配。Spark运行在Java虚拟机上,随着堆空间内存被大量使用,Java虚拟机通过回收内存来为新对象提供空间(垃圾回收机制,GC)的时间开销占Spark作业总耗时的比例显著增加,但Spark作业的效率并未随着可用内存的增加而保持一定比例的提升。在使用非堆(本地)内存存储模式后,GC开销问题得以缓解,但缓存数据的序列化开销成为新的矛盾点。文中利用本地存储方式解决GC问题,同时通过减少序列化开销以加快作业速度,提出并修改了Spark的存储结构,改进了RDD的淘汰机制和缓存方式,将去序列化的数据引入到本地内存中,在保持较低的垃圾回收开销的同时,降低了序列化的开销。实验结果表明,与原Spark的堆上存储方式相比,非序列化的本地存储方法在单结点、大内存的服务器上的GC时间缩短到5%～30%,同时,序列化开销显著降低,吞吐量得到提升,作业耗时缩短8%以上。
Spark framework is taken as the computing framework of big data by more and more enterprises.However,with the increasing of available memory resource of current severs,Spark can't match with new environment well.Spark runs on Java Virtual Machine(JVM).As heap space memory is used heavily,the ratio of time cost produced by Java virtual machine to provide space for new objects by reclaiming memory(GC) to total time cost of Spark jobs increases significantly,but the efficiency of Spark jobs doesn't improve with a certain ratio when the available memory increases.After using OffHeap(native) memory storage mode,the cost of serialization/deserialization becomes the new conflict point instead of GC.This paper used the way of native storage to deal with GC problem,and speeded up the job by reducing the overhead of GC.This paper also proposed and modified the storage structure of Spark,and improved the elimination mechanism and the caching way of RDD.The data without serialization are moved into native memory,realizing low garbage collection overhead and avoiding the time spending on serialization.Experimental results demonstrate that the GC cost of modification method on server with single node and large memory is 5% to 30% compared with the storage on heap of Spark.Meanwhile,the overhead of serialization decreases,the throughput increases and the running time of job can be reduced by more than 8%.
Spark framework is taken as the computing framework of big data by more and more enterprises.However,with the increasing of available memory resource of current severs,Spark can't match with new environment well.Spark runs on Java Virtual Machine(JVM).As heap space memory is used heavily,the ratio of time cost produced by Java virtual machine to provide space for new objects by reclaiming memory(GC) to total time cost of Spark jobs increases significantly,but the efficiency of Spark jobs doesn't improve with a certain ratio when the available memory increases.After using OffHeap(native) memory storage mode,the cost of serialization/deserialization becomes the new conflict point instead of GC.This paper used the way of native storage to deal with GC problem,and speeded up the job by reducing the overhead of GC.This paper also proposed and modified the storage structure of Spark,and improved the elimination mechanism and the caching way of RDD.The data without serialization are moved into native memory,realizing low garbage collection overhead and avoiding the time spending on serialization.Experimental results demonstrate that the GC cost of modification method on server with single node and large memory is 5% to 30% compared with the storage on heap of Spark.Meanwhile,the overhead of serialization decreases,the throughput increases and the running time of job can be reduced by more than 8%.
引文
[1] WHITE T,CUTTING D.Hadoop:the definitive guide[J].O’reilly Media Inc Gravenstein Highway North,2012,215(11):1-4.
[2] ZAHARIA M,CHOWDHURY M,FRANKLIN M J,et al.Spark:cluster computing with working sets[C]//Usenix Conference on Hot Topics in Cloud Computing.USENIX Association,2010.
[3] NGUYEN K,WANG K,BU Y,et al.FACADE:A Compiler and Runtime for (Almost) Object-Bounded Big Data Applications[C]//Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems.ACM,2015:675-690.
[4] LU L,SHI X,ZHOU Y,et al.Lifetime-based memory management for distributed data processing systems[J].Proceedings of the Vldb Endowment,2016,9(12):936-947.
[5] FANG L,NGUYEN K,XU G,et al.Interruptible tasks:treating memory pressure as interrupts for highly scalable data-pa-rallel programs[C]//Symposium on Operating Systems Principles.ACM,2015:394-409.
[6] Project Tungsten[EB/OL].http://tinyurl.com/mzw7hew.
[7] BIAN C,YU J,RONG C T,et al.Self-Adaptive Strategy for Cache Management in Spark[J].Acta Electronica Sinica,2017,45(2):278-284.(in Chinese)卞琛,于炯,英昌甜,等.并行计算框架Spark的自适应缓存管理策略[J].电子学报,2017,45(2):278-284.
[8] MENG H T,YU S P,LIU F,et al.Research on Memory Ma- nagement and Cache Replacement Polices in Spark[J].Computer Science,2017,44(6):31-35.(in Chinese)孟红涛,余松平,刘芳,等.Spark内存管理及缓存策略研究[J].计算机科学,2017,44(6):31-35.
[9] NGUYEN K,FANG L,XU G,et al.Yak:a high-performance big-data-friendly garbage collector[C]//Usenix Conference on Operating Systems Design and Implementation.USENIX Association,2016:349-365.
[10] KIM M,LI J,VOLOS H,et al.Sparkle:optimizing spark for large memory machines and analytics[J].arXiv preprint arXiv:2017:656-656.
[11] NGUYEN K,FANG L,NAVASCA C,et al.Skyway:Connec- ting Managed Heaps in Distributed Big Data Systems[C]//International Conference.2018:56-69.
[12] Hotspot[EB/OL].http://openjdk.java.net/groups/hotspot.
[13] MURRAY D G,MCSHERRY F,ISAACS R,et al.Naiad:a timely dataflow system[C]//Twenty-Fourth ACM Symposium on Operating Systems Principles.ACM,2013:439-455.
[14] BORKAR V,CAREY M,GROVER R,et al.Hyracks:A flexible and extensible foundation for data-intensive computing[C]//IEEE,International Conference on Data Engineering.IEEE Computer Society,2011:1151-1162.
[15] Oracle.Java Platform,Standard Edition HotSpotVirtual Ma- chine Garbage Collection Tuning Guide [EB/OL].https://docs.oracle.com/javase/9/gctuning/garbage-collector-implemen-tation.html.
[16] BU Y,BORKAR V,XU G,et al.A bloat-aware design for big data applications[C]//International Symposium on Memory Management.ACM,2013:119-130.
[17] NGUYEN K,FANG L,XU G,et al.Speculative region-based memory management for big data systems[C]//The Workshop on Programming Languages and Operating Systems.ACM,2015:27-32.
[18] YAN L,SHEN R.Java Serialization Technology[J].Yunnan:Journal of Honghe University,2011,9(4):37-39.
[19] LIAO W J,HUANG Y F,BAO C K.Memory optimization of Spark parallel computing framework[J].Computer Engineering &Science,2018,40(4):21-27.
[20] HUANG T H,WANG Y L,WANG Z,et al.Spark I/O Per- formance Optimization Based on Memory and File Sharing Mechanism[J].Computing Engineering,2017,43(3):1-6.
[21] CHAI N,WU Y J,ZHAO W Y.Optimization For Spark Mission Performance Based on Data Characteristics[J].Computer Applications and Software,2018,35(1):52-58.
[22] Scala Collections[EB/OL].https://www.scala-lang.org/docu/files/collections-api/collections.html.
[23] MetricsSystem[EB/OL].https://spark.apache.org/docs/1.2.0/api/java/org/apache/spark/metrics/MetricsSystem.html.
[2] ZAHARIA M,CHOWDHURY M,FRANKLIN M J,et al.Spark:cluster computing with working sets[C]//Usenix Conference on Hot Topics in Cloud Computing.USENIX Association,2010.
[3] NGUYEN K,WANG K,BU Y,et al.FACADE:A Compiler and Runtime for (Almost) Object-Bounded Big Data Applications[C]//Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems.ACM,2015:675-690.
[4] LU L,SHI X,ZHOU Y,et al.Lifetime-based memory management for distributed data processing systems[J].Proceedings of the Vldb Endowment,2016,9(12):936-947.
[5] FANG L,NGUYEN K,XU G,et al.Interruptible tasks:treating memory pressure as interrupts for highly scalable data-pa-rallel programs[C]//Symposium on Operating Systems Principles.ACM,2015:394-409.
[6] Project Tungsten[EB/OL].http://tinyurl.com/mzw7hew.
[7] BIAN C,YU J,RONG C T,et al.Self-Adaptive Strategy for Cache Management in Spark[J].Acta Electronica Sinica,2017,45(2):278-284.(in Chinese)卞琛,于炯,英昌甜,等.并行计算框架Spark的自适应缓存管理策略[J].电子学报,2017,45(2):278-284.
[8] MENG H T,YU S P,LIU F,et al.Research on Memory Ma- nagement and Cache Replacement Polices in Spark[J].Computer Science,2017,44(6):31-35.(in Chinese)孟红涛,余松平,刘芳,等.Spark内存管理及缓存策略研究[J].计算机科学,2017,44(6):31-35.
[9] NGUYEN K,FANG L,XU G,et al.Yak:a high-performance big-data-friendly garbage collector[C]//Usenix Conference on Operating Systems Design and Implementation.USENIX Association,2016:349-365.
[10] KIM M,LI J,VOLOS H,et al.Sparkle:optimizing spark for large memory machines and analytics[J].arXiv preprint arXiv:2017:656-656.
[11] NGUYEN K,FANG L,NAVASCA C,et al.Skyway:Connec- ting Managed Heaps in Distributed Big Data Systems[C]//International Conference.2018:56-69.
[12] Hotspot[EB/OL].http://openjdk.java.net/groups/hotspot.
[13] MURRAY D G,MCSHERRY F,ISAACS R,et al.Naiad:a timely dataflow system[C]//Twenty-Fourth ACM Symposium on Operating Systems Principles.ACM,2013:439-455.
[14] BORKAR V,CAREY M,GROVER R,et al.Hyracks:A flexible and extensible foundation for data-intensive computing[C]//IEEE,International Conference on Data Engineering.IEEE Computer Society,2011:1151-1162.
[15] Oracle.Java Platform,Standard Edition HotSpotVirtual Ma- chine Garbage Collection Tuning Guide [EB/OL].https://docs.oracle.com/javase/9/gctuning/garbage-collector-implemen-tation.html.
[16] BU Y,BORKAR V,XU G,et al.A bloat-aware design for big data applications[C]//International Symposium on Memory Management.ACM,2013:119-130.
[17] NGUYEN K,FANG L,XU G,et al.Speculative region-based memory management for big data systems[C]//The Workshop on Programming Languages and Operating Systems.ACM,2015:27-32.
[18] YAN L,SHEN R.Java Serialization Technology[J].Yunnan:Journal of Honghe University,2011,9(4):37-39.
[19] LIAO W J,HUANG Y F,BAO C K.Memory optimization of Spark parallel computing framework[J].Computer Engineering &Science,2018,40(4):21-27.
[20] HUANG T H,WANG Y L,WANG Z,et al.Spark I/O Per- formance Optimization Based on Memory and File Sharing Mechanism[J].Computing Engineering,2017,43(3):1-6.
[21] CHAI N,WU Y J,ZHAO W Y.Optimization For Spark Mission Performance Based on Data Characteristics[J].Computer Applications and Software,2018,35(1):52-58.
[22] Scala Collections[EB/OL].https://www.scala-lang.org/docu/files/collections-api/collections.html.
[23] MetricsSystem[EB/OL].https://spark.apache.org/docs/1.2.0/api/java/org/apache/spark/metrics/MetricsSystem.html.
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |