三维交错网格有限差分地震波模拟的GPU集群实现
|
摘要
有限差分实现简单、速度快,作为地震波场模拟一种有效数值方法,被广泛用于正演计算密集的波形反演和逆时偏移中.三维地震波正演模拟计算量大,一直以来制约着三维叠前逆时偏移和反演的工业化应用,GPU通用计算技术的产生及其内在的数据并行性有望改变这一现状.本文通过分析三维交错网格有限差分方法在GPU上的实施,利用片内共享存储器实现了三维地震波数值模拟的高效算法,取得了较单核CPU快79x～108x的加速比;通过区域分解技术将单GPU上不能计算的地质体模型沿Z轴方向进行粗粒度分解,采用消息传递接口交换边界数据,运用MPI+CUDA的方式实现了大尺度三维地震波场模拟,并着重分析了影响GPU并行计算效率的一些关键因素.大尺度三维地震波场模拟的加速实现,为促进叠前逆时偏移和波形反演技术的工业化转化提供了可能,因此具有重要的研究意义.
As an efficient numerical method for seismic modeling,finite difference has been widely used in computation-intensive waveform inversion and reverse-time migration.Although simple and fast,it is still hard to alleviate giant computation cost in three-dimensional seismic modeling that greatly restricts the industrial application of pre-stack reverse-time migration and inversion.The GPU general-purpose computation technology that is well known for its inherent data parallelism is expected to break through this limitation and makes large-scale threedimensional reverse-time migration and inversion possible in application.In this paper,we present a method to configure the discrete grids on GPU thread blocks that optimally maps the global memory address on global video memory into inner shared memory to achieve maximum efficiency.The tests applied in different scale models show that we can achieve 79x~108x speedup ratio when compared to traditional single-core CPU.By virtue of domain decomposition,we use coarse-grained algorithm to partition the large-scale geological model along the Z-axis and compute them using hybrid MPI and CUDA codes to realize the large-scale seismic modeling which can hardly be implemented on single GPU.In the experiments,we also analyze some key factors that affect the parallel efficiency of GPU cluster.The methodology we present and the problems we discuss can be used as a basis to promote the industrial application of pre-stack reverse-time migration and inversion that depend heavily on large-scale threedimensional seismic modeling.
As an efficient numerical method for seismic modeling,finite difference has been widely used in computation-intensive waveform inversion and reverse-time migration.Although simple and fast,it is still hard to alleviate giant computation cost in three-dimensional seismic modeling that greatly restricts the industrial application of pre-stack reverse-time migration and inversion.The GPU general-purpose computation technology that is well known for its inherent data parallelism is expected to break through this limitation and makes large-scale threedimensional reverse-time migration and inversion possible in application.In this paper,we present a method to configure the discrete grids on GPU thread blocks that optimally maps the global memory address on global video memory into inner shared memory to achieve maximum efficiency.The tests applied in different scale models show that we can achieve 79x~108x speedup ratio when compared to traditional single-core CPU.By virtue of domain decomposition,we use coarse-grained algorithm to partition the large-scale geological model along the Z-axis and compute them using hybrid MPI and CUDA codes to realize the large-scale seismic modeling which can hardly be implemented on single GPU.In the experiments,we also analyze some key factors that affect the parallel efficiency of GPU cluster.The methodology we present and the problems we discuss can be used as a basis to promote the industrial application of pre-stack reverse-time migration and inversion that depend heavily on large-scale threedimensional seismic modeling.
引文
[1]张金海,王卫民,赵连锋等.傅里叶有限差分法三维波动方程正演模拟.地球物理学报,2007,50(6):1854-1862. Zhang J H,Wang W M,Zhao L F,et al.Modeling 3-D scalar waves using the Fourier finite-difference method. Chinese J.Geophys.(in Chinese),2007,50(6):1854- 1862.
[2]Alterman Z,Karal F C Propagation of elastic waves in layered media by finite difference methods.Bull.Seism.Soc. Am.,1968,58(1):367-398.
[3]Vireux J.P-SV wave propagation in heterogeneous media: velocity stress finite-difference method.Geophysics,1986,51 (4):889-901.;
[4]董良国,马在田,曹景忠.一阶弹性波方程交错网格有限差分解法.地球物理学报,2000,43(3):411-419. Dong L G,Ma Z T,Cao J Z.The staggered-grid high-order difference method of one-order elastic equation.Chinese J. Geophys.(in Chinese),2000,43(3):411-419.
[5]Graves R W.Simulating seismic wave propagation in 3D elastic media using staggered-grid finite differences.Bull. Seis.Soc.Am.,1996,86(4):1091-1106.
[6]Moczo P,Robertsson J,Eisner L.The finite-difference time-domain method for modeling of seismic wave propagation,in Advances in Wave Propagation in Heterogeneous Media,Vol. 48 of Advances in Geophysics,Chapter 8,pp.421～516,eds Wu,R.-S.& Maupin,V.,Elsevier/Academic Press, London,UK.
[7]张美根.各向异性弹性波正反演问题研究[博士论文].北京:中国科学院地质与地球物理研究所,2000. Zhang M G.The researches of forward modeling and inversion of anisotropic elastic wave[Ph.D.thesis](in Chinese).Beijing: Institute of Geology and Geophysics,Chinese Academy of Sciences,2000.
[8]Fornberg B.The pseudospectral method:accurate representation of interfaces in elastic wave calculations. Geophysics,1988,53(5):625-637.
[9]Dault C R,Brail L W,Nowack R L,et al.A comparison of finite-difference and Fourier method calculations of synthetic seismograms.Bull.Seism.Soc.Am.,1989,79(4):1210- 1230.
[10]Correa G J,Spiegelman M,Carbotle S,et al.Centered and staggered Fourier derivatives and Hilbert transforms. Geophysics,2002,67(5):1558-1563.
[11]Bale R A.Modeling 3D anisotropic elastic data using the pseudospectral approach.CREWES Research Report, Calgary,2002.
[12]Komatitsch D,Tromp J.Introduction to the spectral-element method for 3-D seismic wave propagation.Geophys.J.Int., 1999,139(3):806-822.
[13]Tromp J,Komatitsch D,Liu Q.Spectral-element and adjoint methods in seismology.Commun.Comput.Phys.,2004,3 (1):1-32.
[14]王童奎,李瑞华,李小凡等.横向各向同性介质中地震波场谱元数值模拟.地球物理学进展,2007,22(3):778-784. Wang T K,Li R H,Li X F,et al.Numerical spectral element modeling for seismic wave propagation in transversely isotropic medium.Progress in Geophysics(in Chinese),2007,22(3):778-784.
[15]Vireux J,Operto S.An overview of full-waveform inversion in exploration geophysics.Geophysics,2009,74(6):127- 152.
[16]http://developer.download.nvidia.com/compute/cuda/2_3/ docs/CUDA_Getting_Started 2.3 Linux.pdf.
[17]http://www.khronos.org/opencl.
[18]Owens J D,Houston M,Luebke D,et al.GPU Computing. Proceedings of the IEEE,2008,96(5):879-899.
[19]Nguyen H,Zeller C,Hart E,et al.GPU Gems 3.Addison-Wesley Professional,2007.
[20]Cohen J M,Molemaker M J.A fast double precision CFD code using CUDA.Proceedings of Parallel CFD,2009.
[21]Micikevicius P.3D finite difference computation on GPUs using CUDA.GPGPU-2:Proceeding of 2~(nd)Workshop on General Purpose Processing on Graphics Processing Units (New York,USA),ACM,2009,79-84.
[22]Abdelkhalek R,Calandra H,Coulaud O,et al.Fast seismic modeling and reverse time migration on a GPU cluster.The 2009 High Performance Computation & Simulation HPCS'09,2009.
[23]李博,刘国峰,刘洪.地震叠前时间偏移的一种图形处理器提速实现方法.地球物理学报,2009,52(1):245-252. Li B,Liu G F,Liu H.A method of using GPU to accelerate seismic pre-stack time migration.Chinese J.Geophys.(in Chinese),2009,52(1):245-252.
[24]刘国峰,刘洪,李博等.山地地震资料叠前时间偏移方法及其GPU实现.地球物理学报,2009,52(12):3101-3108. Liu G F,Liu H,Ii B,et al.Method of prestack time migration of seismic data of mountainous regions and its GPU implementation.Chinese J.Geophys.(in Chinese),2009, 52(12):3101-3108.
[25]刘红伟,李博,刘洪等.地震叠前逆时偏移高阶有限差分算法及GPU实现.地球物理学报,2010,53(7):1725-1733. Liu H W,Li B,Liu H,et al.The algorithm of high order finite difference pre stack reverse time migration and GPU implementation.Chinese J.Geophys.(in Chinese),2010, 53(7):1725-1733.
[26]Zhang J H,Wang S Q,Yao Z X.Accelerating 3D Fourier migration with graphics processing units.Geophysics,2009, 74(6):129-139.
[27]石颖,刘洪.基于GPU的表面多次波预测技术研究.石油地球物理勘探,2010,45(4):540-544. Shi Y,Liu H.GPU-based surface multiple prediction technique.Oil geophysical prospecting,2010,45(4):540-544.
[28]Michea D,Komatitsch D.Accelerating a three-dimensional finite-difference wave propagation code using GPU graphics cards.Geophys.J.Int.,2010,182(1):389-402.
[29]Komatitsch D,Tromp J.A perfectly matched layer absorbing boundary condition for the second-order seismic wave equation.Geophys.J.Int.,2003,154:146-153.
[30]Collino F,Tsogka C.Application of the PML absorbing layer model to the linear electrodynamic problem in anisotropic heterogeneous media.Geophysics,2001,66(1):294-307.
[31]龙桂华.粘弹性介质中的地震波传播与波形反演研究[博士论文].北京:中国科学院地质与地球物理研究所,2009. Long G H.The study of seismic wave propagation and waveform inversion in viscoelastic media[Ph.D.thesis](in Chinese).Beijing:Institute of Geology and Geophysics, Chinese Academy of Sciences,2009.
[32]Helberg O.Computational aspects of the choice of operator and sampling interval for numerical differentiation in large-scale simulation of wave phenomena.Geophys.Prospect., 1987,35(6):629-655.
[33]龙桂华,李小凡,张美根.基于Shannon奇异核理论的褶积微分算子在地震波场模拟中的应用.地球物理学报,2009,52 (4):1014-1024. Long G H,Li X F,Zhang M G.The application of convolutional differentiator in seismic modeling based on Shannon singular kernel theory.Chinese J.Geophys.(in Chinese),2009,52(4):1014-1024.
[34]张舒,褚艳利,赵开勇等.GPU高性能计算之CUDA.北京:中国水利水电出版社,2009. Zhang S,Zhu Y L,Zhao K Y,et al.GPU high-performance computation:CUDA.Beijing:China WaterPower Press, 2009.
[35]龙桂华,李小凡,赵家福.基于交错网格Fourier伪谱微分矩阵算子的地震波场模拟GPU加速方案.地球物理学报, 2010,53(12):2964-2971. Long G H,Li X F,Zhao J F.Accelerating seismic wave modeling with staggered-grid Fourier pseudo-spectral differentiation matrix operator method on graphics processing units.Chinese J.Geophys.(in Chinese),2010,53(12): 2964-2971.
[36]http://www.nvidia.com/content/PDF/fermi_white_papers/ P.Glaskowsky_NVIDIA's_Fermi-The_First_Complete_GPU Architecture.pdf.
[2]Alterman Z,Karal F C Propagation of elastic waves in layered media by finite difference methods.Bull.Seism.Soc. Am.,1968,58(1):367-398.
[3]Vireux J.P-SV wave propagation in heterogeneous media: velocity stress finite-difference method.Geophysics,1986,51 (4):889-901.;
[4]董良国,马在田,曹景忠.一阶弹性波方程交错网格有限差分解法.地球物理学报,2000,43(3):411-419. Dong L G,Ma Z T,Cao J Z.The staggered-grid high-order difference method of one-order elastic equation.Chinese J. Geophys.(in Chinese),2000,43(3):411-419.
[5]Graves R W.Simulating seismic wave propagation in 3D elastic media using staggered-grid finite differences.Bull. Seis.Soc.Am.,1996,86(4):1091-1106.
[6]Moczo P,Robertsson J,Eisner L.The finite-difference time-domain method for modeling of seismic wave propagation,in Advances in Wave Propagation in Heterogeneous Media,Vol. 48 of Advances in Geophysics,Chapter 8,pp.421～516,eds Wu,R.-S.& Maupin,V.,Elsevier/Academic Press, London,UK.
[7]张美根.各向异性弹性波正反演问题研究[博士论文].北京:中国科学院地质与地球物理研究所,2000. Zhang M G.The researches of forward modeling and inversion of anisotropic elastic wave[Ph.D.thesis](in Chinese).Beijing: Institute of Geology and Geophysics,Chinese Academy of Sciences,2000.
[8]Fornberg B.The pseudospectral method:accurate representation of interfaces in elastic wave calculations. Geophysics,1988,53(5):625-637.
[9]Dault C R,Brail L W,Nowack R L,et al.A comparison of finite-difference and Fourier method calculations of synthetic seismograms.Bull.Seism.Soc.Am.,1989,79(4):1210- 1230.
[10]Correa G J,Spiegelman M,Carbotle S,et al.Centered and staggered Fourier derivatives and Hilbert transforms. Geophysics,2002,67(5):1558-1563.
[11]Bale R A.Modeling 3D anisotropic elastic data using the pseudospectral approach.CREWES Research Report, Calgary,2002.
[12]Komatitsch D,Tromp J.Introduction to the spectral-element method for 3-D seismic wave propagation.Geophys.J.Int., 1999,139(3):806-822.
[13]Tromp J,Komatitsch D,Liu Q.Spectral-element and adjoint methods in seismology.Commun.Comput.Phys.,2004,3 (1):1-32.
[14]王童奎,李瑞华,李小凡等.横向各向同性介质中地震波场谱元数值模拟.地球物理学进展,2007,22(3):778-784. Wang T K,Li R H,Li X F,et al.Numerical spectral element modeling for seismic wave propagation in transversely isotropic medium.Progress in Geophysics(in Chinese),2007,22(3):778-784.
[15]Vireux J,Operto S.An overview of full-waveform inversion in exploration geophysics.Geophysics,2009,74(6):127- 152.
[16]http://developer.download.nvidia.com/compute/cuda/2_3/ docs/CUDA_Getting_Started 2.3 Linux.pdf.
[17]http://www.khronos.org/opencl.
[18]Owens J D,Houston M,Luebke D,et al.GPU Computing. Proceedings of the IEEE,2008,96(5):879-899.
[19]Nguyen H,Zeller C,Hart E,et al.GPU Gems 3.Addison-Wesley Professional,2007.
[20]Cohen J M,Molemaker M J.A fast double precision CFD code using CUDA.Proceedings of Parallel CFD,2009.
[21]Micikevicius P.3D finite difference computation on GPUs using CUDA.GPGPU-2:Proceeding of 2~(nd)Workshop on General Purpose Processing on Graphics Processing Units (New York,USA),ACM,2009,79-84.
[22]Abdelkhalek R,Calandra H,Coulaud O,et al.Fast seismic modeling and reverse time migration on a GPU cluster.The 2009 High Performance Computation & Simulation HPCS'09,2009.
[23]李博,刘国峰,刘洪.地震叠前时间偏移的一种图形处理器提速实现方法.地球物理学报,2009,52(1):245-252. Li B,Liu G F,Liu H.A method of using GPU to accelerate seismic pre-stack time migration.Chinese J.Geophys.(in Chinese),2009,52(1):245-252.
[24]刘国峰,刘洪,李博等.山地地震资料叠前时间偏移方法及其GPU实现.地球物理学报,2009,52(12):3101-3108. Liu G F,Liu H,Ii B,et al.Method of prestack time migration of seismic data of mountainous regions and its GPU implementation.Chinese J.Geophys.(in Chinese),2009, 52(12):3101-3108.
[25]刘红伟,李博,刘洪等.地震叠前逆时偏移高阶有限差分算法及GPU实现.地球物理学报,2010,53(7):1725-1733. Liu H W,Li B,Liu H,et al.The algorithm of high order finite difference pre stack reverse time migration and GPU implementation.Chinese J.Geophys.(in Chinese),2010, 53(7):1725-1733.
[26]Zhang J H,Wang S Q,Yao Z X.Accelerating 3D Fourier migration with graphics processing units.Geophysics,2009, 74(6):129-139.
[27]石颖,刘洪.基于GPU的表面多次波预测技术研究.石油地球物理勘探,2010,45(4):540-544. Shi Y,Liu H.GPU-based surface multiple prediction technique.Oil geophysical prospecting,2010,45(4):540-544.
[28]Michea D,Komatitsch D.Accelerating a three-dimensional finite-difference wave propagation code using GPU graphics cards.Geophys.J.Int.,2010,182(1):389-402.
[29]Komatitsch D,Tromp J.A perfectly matched layer absorbing boundary condition for the second-order seismic wave equation.Geophys.J.Int.,2003,154:146-153.
[30]Collino F,Tsogka C.Application of the PML absorbing layer model to the linear electrodynamic problem in anisotropic heterogeneous media.Geophysics,2001,66(1):294-307.
[31]龙桂华.粘弹性介质中的地震波传播与波形反演研究[博士论文].北京:中国科学院地质与地球物理研究所,2009. Long G H.The study of seismic wave propagation and waveform inversion in viscoelastic media[Ph.D.thesis](in Chinese).Beijing:Institute of Geology and Geophysics, Chinese Academy of Sciences,2009.
[32]Helberg O.Computational aspects of the choice of operator and sampling interval for numerical differentiation in large-scale simulation of wave phenomena.Geophys.Prospect., 1987,35(6):629-655.
[33]龙桂华,李小凡,张美根.基于Shannon奇异核理论的褶积微分算子在地震波场模拟中的应用.地球物理学报,2009,52 (4):1014-1024. Long G H,Li X F,Zhang M G.The application of convolutional differentiator in seismic modeling based on Shannon singular kernel theory.Chinese J.Geophys.(in Chinese),2009,52(4):1014-1024.
[34]张舒,褚艳利,赵开勇等.GPU高性能计算之CUDA.北京:中国水利水电出版社,2009. Zhang S,Zhu Y L,Zhao K Y,et al.GPU high-performance computation:CUDA.Beijing:China WaterPower Press, 2009.
[35]龙桂华,李小凡,赵家福.基于交错网格Fourier伪谱微分矩阵算子的地震波场模拟GPU加速方案.地球物理学报, 2010,53(12):2964-2971. Long G H,Li X F,Zhao J F.Accelerating seismic wave modeling with staggered-grid Fourier pseudo-spectral differentiation matrix operator method on graphics processing units.Chinese J.Geophys.(in Chinese),2010,53(12): 2964-2971.
[36]http://www.nvidia.com/content/PDF/fermi_white_papers/ P.Glaskowsky_NVIDIA's_Fermi-The_First_Complete_GPU Architecture.pdf.
版权所有：© 2023 中国地质图书馆 中国地质调查局地学文献中心