达梦OLAP聚集设计器的研究
|
摘要
商业智能工具迅速发展,多维查询技术日趋成熟,在线联机分析系统查询效率的提升成为急需解决的问题。聚集技术是提升在线联机分析系统性能的关键技术之一。在分析达梦聚集技术的基础上,研究聚集设计器的设计与实现,主要从系统的需求分析和体系设计、关键技术以及具体实现三方面进行研究。
根据系统的需求分析,论证设计方案,设计出聚集设计器的体系结构。聚集设计器的体系结构主要包括元数据管理模块、聚集设计模块以及聚集方案优化模块。达梦多维查询模型元数据信息是聚集设计和聚集方案优化的重要依据,聚集元数据是多维查询模型元数据的一部分。元数据管理模块主要解决两方面的问题,一方面是多维查询模型元数据和聚集元数据格式的规范性问题,另一方面是元数据初始化方法的合理性问题。聚集设计模块主要功能是创建满足聚集技术要求的元素,这些元素包括聚集表、聚集表模式匹配以及聚集表排除。聚集方案优化模块研究聚集优化策略,其功能是从联机查询日志中分析系统执行查询的行为,然后根据优化策略动态生成聚集表,此模块主要包括查询日志分析组件、聚集方案优化组件、聚集物化组件和聚集元数据维护组件。
实验表明,聚集设计器实现了聚集表设计、聚集模式匹配设计、聚集排除方式设计以及聚集方案优化策略,完成了系统的功能需求,提高了达梦在线联机分析系统的整体性能。
With the rapid development of business intelligence tools and the mature of multidimensional query technology, the query efficiency of On-Line Analysis Processing system becomes an urgent problem.The aggregation technology is one of key technologies which can enhance the performance of the OLAP systerm.This paper,based on the analysis about Dameng aggregation technology,researches the design and implementation of the Aggregation Designer.Mainly research the three points about requirement analysis and system design,key technologies and implementation.
According to the requirement analysis,demonstate the design program,and construct a reasonable system structure of Aggregation Designer.The structure mainly includes metadata management, the aggregation design and the aggregation program optimization.The metadata of Dameng Multi-dimensional query system is the basis of the aggregation design and the aggregation optimization.Aggregation metadata is part of it.The Metadata Management module mainly solves two problems of the storage format and the loading method.The Aggregation Design module is to creat elements which meet aggregation technical requirements.The elements include aggregation table,aggregation pattern matching and aggregation excluding tables.The Aggregation Program Optimization module researches the optimization strategy.Its function is to analyze the behavior of system’s queries from online query logs at first,and then dynamically generate aggregation table based on optimization strategy.The module includes analysis component of query log,optimization component,materilize component and aggregation metadata maintenance component.
Experiment shows that the Aggregation Designer has achieved the design of aggregation tables,the design of aggregation pattern matching,the design of aggregation excluding tables and the optimization strategy,and realized the system functional requirements,improved the OLAP system performance.
根据系统的需求分析,论证设计方案,设计出聚集设计器的体系结构。聚集设计器的体系结构主要包括元数据管理模块、聚集设计模块以及聚集方案优化模块。达梦多维查询模型元数据信息是聚集设计和聚集方案优化的重要依据,聚集元数据是多维查询模型元数据的一部分。元数据管理模块主要解决两方面的问题,一方面是多维查询模型元数据和聚集元数据格式的规范性问题,另一方面是元数据初始化方法的合理性问题。聚集设计模块主要功能是创建满足聚集技术要求的元素,这些元素包括聚集表、聚集表模式匹配以及聚集表排除。聚集方案优化模块研究聚集优化策略,其功能是从联机查询日志中分析系统执行查询的行为,然后根据优化策略动态生成聚集表,此模块主要包括查询日志分析组件、聚集方案优化组件、聚集物化组件和聚集元数据维护组件。
实验表明,聚集设计器实现了聚集表设计、聚集模式匹配设计、聚集排除方式设计以及聚集方案优化策略,完成了系统的功能需求,提高了达梦在线联机分析系统的整体性能。
With the rapid development of business intelligence tools and the mature of multidimensional query technology, the query efficiency of On-Line Analysis Processing system becomes an urgent problem.The aggregation technology is one of key technologies which can enhance the performance of the OLAP systerm.This paper,based on the analysis about Dameng aggregation technology,researches the design and implementation of the Aggregation Designer.Mainly research the three points about requirement analysis and system design,key technologies and implementation.
According to the requirement analysis,demonstate the design program,and construct a reasonable system structure of Aggregation Designer.The structure mainly includes metadata management, the aggregation design and the aggregation program optimization.The metadata of Dameng Multi-dimensional query system is the basis of the aggregation design and the aggregation optimization.Aggregation metadata is part of it.The Metadata Management module mainly solves two problems of the storage format and the loading method.The Aggregation Design module is to creat elements which meet aggregation technical requirements.The elements include aggregation table,aggregation pattern matching and aggregation excluding tables.The Aggregation Program Optimization module researches the optimization strategy.Its function is to analyze the behavior of system’s queries from online query logs at first,and then dynamically generate aggregation table based on optimization strategy.The module includes analysis component of query log,optimization component,materilize component and aggregation metadata maintenance component.
Experiment shows that the Aggregation Designer has achieved the design of aggregation tables,the design of aggregation pattern matching,the design of aggregation excluding tables and the optimization strategy,and realized the system functional requirements,improved the OLAP system performance.
引文
[1] Codd E F, Codd S B, Salley C T. Providing OLAP to User-Analysts: An IT Mandate. Technical Report, 1993
[2] Inmon W H. Building the Warehouse (2nd Edition). New York: John Wiley and Sons, 1996. 1~10
[3]张柏礼,孙志挥,周晓云等.静态物化视图的动态Cache优化算法.软件学报, 2006, 17(5): 1213~1221
[4]萨师煊,王珊.数据库概论(第2版).北京:高等教育出版社, 1991. 5~10
[5] Han J, Kamber M.数据挖掘概念与技术.范明,孟小峰.北京:机械工业出版社, 2001. 29~56
[6]邵峰晶,于忠清.数据挖掘原理与算法.北京:中国水利水电出版社, 2003. 31~64
[7] Konstantions M, Stratis K, Nikolaos K. ROLAP Implementations of the Data Cube. ACM Computing Surveys, 2007, 39(4): 12~65
[8] Dehne F, Eavis T, RauChaplin A. Parallel querying of ROLAP cubes in the presence of hierarchies. in: Proceedings of the 8th ACM international workshop-Data warehousing and OLAP, 2005. 234~251
[9]万怀宇,黄厚宽.位图索引及其在数据仓库中的应用研究.铁路计算机应用, 2006, 15(12): 31~33
[10]周丽萍,黄厚宽.关于数据仓库中编码位图索引的研究.铁路计算机应用, 2004, 13(6): 4~6
[11]胡孔法,董逸生,陈崚.数据仓库中一种基于维层次编码的位图索引方法.东南大学学报, 2005, 35(2): 171~177
[12]叶德谦,马勤勇,叶柠.优化MDX查询语句提高OLAP系统性能的研究.微处理机, 2002, 8: 27~29
[13]李翠平,王珊.基于学习的OLAP查询结果缓存技术.计算机研究与发展, 2004, (10): 21~24
[14]冯建华,蒋旭东,孟宪虎.基于分组序号的聚集算法.软件学报, 2003, 14(2): 222~229
[15] Igor Timko, Curtis E Dyreson,Torben Bach Pedersen. Pre-Aggregation with Probability Distributions. USA:DOLAP,2006, 10: 35~42
[16] Gray J, Chaudhuri S, Bosworth A. Data Cube:A Relational Aggregation Operator Generalizing Group-By,Cross-By,Cross-Tab and Sub-Totals.in: Proc. of the 12th Int. Conf. On Data Engg., 1996: 152~159
[17]许建,马强. ROLAP查询优化的研究.计算机与现代化, 2008, (7): 4~7
[18]迟忠先,王红新,于风友.数据仓库中聚集管理与导航策略.小型微型计算机系统, 2002, 26(12): 1456~1461
[19] Harinarayan V, Rajaraman A, Ullman J D. Implementing Data Cubes Efficiently. in: Proc. ACM SIGMOD International Conf. on Management of Data, New York: ACM Press, 1996. 205~227
[20] Zhao Yihong, Deshpande Prasad M. Simultaneous Optimization and Evaluation of Multiple Dimentions Queries. CN-SIGMOD, 1998. 271~282
[21] Panos Kalnisn, Dimitris Papadias. Multi-query optimization for on-line analytical processing. Information Systems, 2003, (28): 457~473
[22] Gupta H. Selection of Views to Materialize in a Data Warehouse. in: Proc. of the 3th International Coference on Data Engineering, 1997. 98~112
[23] Gupta H, Mumick I S. Selection of Views to Materialize Under a Maintenance Cost Constraint. in: Proc. of the 7th International Conference on Database Theory. London: Springer Verlag, 1999. 453~470
[24] Mumick I S. Maintenance of data cubes and summary tables in a warehouse. in: Proc. of SIGMOD, 1997. 100~111
[25] Pedersen T B, Jensen C S, Dyreson C E. Extending practical pre-aggregation in On-Line Analytical Processing. in: Proc. of the 25th International Conference on Very Large Databases, 1999. 663~674
[26] Yang J, Karlapalem K, Qing L. Algorithms for Materialized View Design in Data Warehousing Environment. in: Proc. of the 23rd International Conference on Very Large Databases, San Francisco: Morgan Kaufmann Publishers, 1997. 136~145
[27] Indulska M. Shared Result Identification for Materialized View Selection. in: Proceedings of the Australasian Database Conference. Washington:IEEE Computer Society, 2000. 117~126
[28]邓蓓,刘嘉勇.数据仓库中物化视图的遗传选择算法.信息与电子工程, 2009: 30~40
[29] Zhang C, Yang J. Materialized View Evolution Support in Data Warehouse Environment.Proceedings of the 6th International Conference on Database Systems for Advanced Applications. Washington:IEEE Computer Society, 1999. 247~254
[30] Zhang C, Yao X, Yang J. An Evolutionary Approach to Materialized Views Selection in a Data Warehouse Environment. IEEE Transactions on Systems, Man and Cybernetics, 2001, 31(3): 231~243
[31]张晓辉,袁愿,虞健飞等.数据仓库物化视图选择的混合算法.计算机应用, 2003, 23(7): 92~97
[32]赖明珠,周丽娟.基于AO View Graph的实视图选择算法的研究.哈尔滨理工大学学报, 2006, 11(6): 23~29
[33]王自强,孙霞,张德贤.数据仓库中用于视图选择的增强遗传算法.小型微型计算机系统, 2007, 28(2): 367~371
[34]张柏礼,孙志挥,孙翔.物化视图选择的预处理算法.计算机研究与发展, 2004, 41(10): 1645~1651
[35] Wiley. MDX Solutions with Microsoft SQL Server Analysis Services 2005 and Hyperion Essbase(2nd Edition). Feb 2006
[36] Mondrian 2.2.2 Technical Gudie Developing OLAP Solutions with Mondrian. 2007: 3~23
[37] Shukla A, Deshpande P M, Naughton J F. Storage Estimation for Multidimensional Aggregates in the Presence of Hierarchies. in: Proc. of the 22nd VLDB Conf., Mumbai, India, 1996: 522~531
[38] Amit Shukla, Prasad M Deshpande, Jeffrey F Naughton. Materialized View Selection for Multidimensional Datasets. in: Proc. of the 24th VLDB Conference, New York, USA, 1998. 488~499
[39] Kanda Runapongsa, Thomas P Nadeau, Toby J Teorey. Storage estimation for multidimensional aggregates in OLAP. in: Proc. ACM SIGMOD, 2002. 3~9
[40]王金水,张东站.数据仓库中物化视图的选择与调整[硕士学位论文].厦门:厦门大学图书馆, 2009. 20~30
[41]张柏礼,朱文,吕建华. DAOA_一种动态数据仓库聚集优化算法.现代电子技术, 2008, 273(10): 31~32
[42] Tan Hongxing, Zhou Longxiang. Dynamic selection of materialized views of multidimensional data. Journal of Software, 2002, 13(6): 1090~1096
[43]张东站,黄宗毅,薛永生. NDSMMV——一种多维数据集物化视图动态选择新策略.计算机研究与发展, 2008, 45(5): 1000~1239
[2] Inmon W H. Building the Warehouse (2nd Edition). New York: John Wiley and Sons, 1996. 1~10
[3]张柏礼,孙志挥,周晓云等.静态物化视图的动态Cache优化算法.软件学报, 2006, 17(5): 1213~1221
[4]萨师煊,王珊.数据库概论(第2版).北京:高等教育出版社, 1991. 5~10
[5] Han J, Kamber M.数据挖掘概念与技术.范明,孟小峰.北京:机械工业出版社, 2001. 29~56
[6]邵峰晶,于忠清.数据挖掘原理与算法.北京:中国水利水电出版社, 2003. 31~64
[7] Konstantions M, Stratis K, Nikolaos K. ROLAP Implementations of the Data Cube. ACM Computing Surveys, 2007, 39(4): 12~65
[8] Dehne F, Eavis T, RauChaplin A. Parallel querying of ROLAP cubes in the presence of hierarchies. in: Proceedings of the 8th ACM international workshop-Data warehousing and OLAP, 2005. 234~251
[9]万怀宇,黄厚宽.位图索引及其在数据仓库中的应用研究.铁路计算机应用, 2006, 15(12): 31~33
[10]周丽萍,黄厚宽.关于数据仓库中编码位图索引的研究.铁路计算机应用, 2004, 13(6): 4~6
[11]胡孔法,董逸生,陈崚.数据仓库中一种基于维层次编码的位图索引方法.东南大学学报, 2005, 35(2): 171~177
[12]叶德谦,马勤勇,叶柠.优化MDX查询语句提高OLAP系统性能的研究.微处理机, 2002, 8: 27~29
[13]李翠平,王珊.基于学习的OLAP查询结果缓存技术.计算机研究与发展, 2004, (10): 21~24
[14]冯建华,蒋旭东,孟宪虎.基于分组序号的聚集算法.软件学报, 2003, 14(2): 222~229
[15] Igor Timko, Curtis E Dyreson,Torben Bach Pedersen. Pre-Aggregation with Probability Distributions. USA:DOLAP,2006, 10: 35~42
[16] Gray J, Chaudhuri S, Bosworth A. Data Cube:A Relational Aggregation Operator Generalizing Group-By,Cross-By,Cross-Tab and Sub-Totals.in: Proc. of the 12th Int. Conf. On Data Engg., 1996: 152~159
[17]许建,马强. ROLAP查询优化的研究.计算机与现代化, 2008, (7): 4~7
[18]迟忠先,王红新,于风友.数据仓库中聚集管理与导航策略.小型微型计算机系统, 2002, 26(12): 1456~1461
[19] Harinarayan V, Rajaraman A, Ullman J D. Implementing Data Cubes Efficiently. in: Proc. ACM SIGMOD International Conf. on Management of Data, New York: ACM Press, 1996. 205~227
[20] Zhao Yihong, Deshpande Prasad M. Simultaneous Optimization and Evaluation of Multiple Dimentions Queries. CN-SIGMOD, 1998. 271~282
[21] Panos Kalnisn, Dimitris Papadias. Multi-query optimization for on-line analytical processing. Information Systems, 2003, (28): 457~473
[22] Gupta H. Selection of Views to Materialize in a Data Warehouse. in: Proc. of the 3th International Coference on Data Engineering, 1997. 98~112
[23] Gupta H, Mumick I S. Selection of Views to Materialize Under a Maintenance Cost Constraint. in: Proc. of the 7th International Conference on Database Theory. London: Springer Verlag, 1999. 453~470
[24] Mumick I S. Maintenance of data cubes and summary tables in a warehouse. in: Proc. of SIGMOD, 1997. 100~111
[25] Pedersen T B, Jensen C S, Dyreson C E. Extending practical pre-aggregation in On-Line Analytical Processing. in: Proc. of the 25th International Conference on Very Large Databases, 1999. 663~674
[26] Yang J, Karlapalem K, Qing L. Algorithms for Materialized View Design in Data Warehousing Environment. in: Proc. of the 23rd International Conference on Very Large Databases, San Francisco: Morgan Kaufmann Publishers, 1997. 136~145
[27] Indulska M. Shared Result Identification for Materialized View Selection. in: Proceedings of the Australasian Database Conference. Washington:IEEE Computer Society, 2000. 117~126
[28]邓蓓,刘嘉勇.数据仓库中物化视图的遗传选择算法.信息与电子工程, 2009: 30~40
[29] Zhang C, Yang J. Materialized View Evolution Support in Data Warehouse Environment.Proceedings of the 6th International Conference on Database Systems for Advanced Applications. Washington:IEEE Computer Society, 1999. 247~254
[30] Zhang C, Yao X, Yang J. An Evolutionary Approach to Materialized Views Selection in a Data Warehouse Environment. IEEE Transactions on Systems, Man and Cybernetics, 2001, 31(3): 231~243
[31]张晓辉,袁愿,虞健飞等.数据仓库物化视图选择的混合算法.计算机应用, 2003, 23(7): 92~97
[32]赖明珠,周丽娟.基于AO View Graph的实视图选择算法的研究.哈尔滨理工大学学报, 2006, 11(6): 23~29
[33]王自强,孙霞,张德贤.数据仓库中用于视图选择的增强遗传算法.小型微型计算机系统, 2007, 28(2): 367~371
[34]张柏礼,孙志挥,孙翔.物化视图选择的预处理算法.计算机研究与发展, 2004, 41(10): 1645~1651
[35] Wiley. MDX Solutions with Microsoft SQL Server Analysis Services 2005 and Hyperion Essbase(2nd Edition). Feb 2006
[36] Mondrian 2.2.2 Technical Gudie Developing OLAP Solutions with Mondrian. 2007: 3~23
[37] Shukla A, Deshpande P M, Naughton J F. Storage Estimation for Multidimensional Aggregates in the Presence of Hierarchies. in: Proc. of the 22nd VLDB Conf., Mumbai, India, 1996: 522~531
[38] Amit Shukla, Prasad M Deshpande, Jeffrey F Naughton. Materialized View Selection for Multidimensional Datasets. in: Proc. of the 24th VLDB Conference, New York, USA, 1998. 488~499
[39] Kanda Runapongsa, Thomas P Nadeau, Toby J Teorey. Storage estimation for multidimensional aggregates in OLAP. in: Proc. ACM SIGMOD, 2002. 3~9
[40]王金水,张东站.数据仓库中物化视图的选择与调整[硕士学位论文].厦门:厦门大学图书馆, 2009. 20~30
[41]张柏礼,朱文,吕建华. DAOA_一种动态数据仓库聚集优化算法.现代电子技术, 2008, 273(10): 31~32
[42] Tan Hongxing, Zhou Longxiang. Dynamic selection of materialized views of multidimensional data. Journal of Software, 2002, 13(6): 1090~1096
[43]张东站,黄宗毅,薛永生. NDSMMV——一种多维数据集物化视图动态选择新策略.计算机研究与发展, 2008, 45(5): 1000~1239
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |