数据仓库元数据集成系统的设计与实现
|
摘要
随着信息技术的不断发展,各个企业内部和企业之间通过开发各种各样的管理信息系统已经逐渐实现了业务的信息化。然而这些系统无法互相连通进行信息交流,导致了“信息孤岛”的现象,业务间的关联使得企业内部在各个环节需要进行数据交换,因此有效解决数据集成问题具有重要的现实意义,而元数据中包含的数据结构和语义说明信息是数据集成的重要信息,因此元数据集成是数据集成关键技术。
在分析了元数据集成的功能需求以及元数据集成系统的设计目标的基础上,设计了一个元数据集成系统的总体结构,并分析了内部各个模块的功能及系统流程,整个系统包括抽取模块和元数据装载存储模块。详细介绍了元数据集成系统中抽取和装载存储两个模块的实现。在抽取模块,结合公共仓库元模型CWM关系型包中的各个对象,给出了抽取的流程和抽取算法,介绍了基于CWM关系型包的元数据抽取技术。在装载和存储模块的实现部分,给出了模块内部的详细结构,包括:将抽取出元数据信息写到XML文档、存储元数据到存储库以及组装形成XML文档三个部分。给出了元数据的存储策略;分析了存储模型,包括保存到XML和存储元数据的数据库模型,在此基础上分别给出了存储元数据和取出装载元数据的算法。
实验部分主要是验证元数据抽取和存储性能的实验,实验表明,通过抽取和存储算法能够很好的提取和存储元数据,性能较好。
With the rapid development of information and network technologies, more and more management information systems have been constructed among enterprises and in the departments of enterprises. However, these systems are not often interoperable and also are not able to communicate with one another to exchange information, which has resulted in the phenomenon of isolated information island, so-called“information island”, the interconnections among businesses require enterprises to solve the problem of data exchange. Therefore, it’s meaningful to solve the problem of data integration effectively. In recent years, the wide application of metadata has brought hope for enterprise data integration. The information on data structure and meaning contained in metadata can be very helpful for data integration. Therefore, metadata integration has been an important method for data integration.
This paper has introduced the characteristics of metadata、requirements of metadata integration、design principles and aims of metadata intergration, and also the paper has introduced common warehouse metamodel(CWM), on introducing recent several popular solutions of metadata integration, it has presented a solution of metadata integration and analyzed the function of every part and system flow. This paper has laid emphasis on the extraction and storage part of the metadata integration system, introduced the implementation of these two parts. The aim of extraction is to extract specific metadata from relational databases for the preparation of load and storage. In the extraction part, with the knowledge of CWM this paper presents the range and objective of extraction, also the flow of extraction and the implementation algorithm. In the module of load and storage, firstly it gives out the structure of the module including keeping metadata in XML file storing in database and extracting metadata to build an XML file. Storage stratege is presented, storage model including keeping metadata in XML and storing metadata in database are introduced, and the algorithms are introduced.
Experiment mainly shows that the performance of metadata extraction and storage, it shows that with the rules of extraction and storage algorithm the system could fulfill the job well, with good performance.
在分析了元数据集成的功能需求以及元数据集成系统的设计目标的基础上,设计了一个元数据集成系统的总体结构,并分析了内部各个模块的功能及系统流程,整个系统包括抽取模块和元数据装载存储模块。详细介绍了元数据集成系统中抽取和装载存储两个模块的实现。在抽取模块,结合公共仓库元模型CWM关系型包中的各个对象,给出了抽取的流程和抽取算法,介绍了基于CWM关系型包的元数据抽取技术。在装载和存储模块的实现部分,给出了模块内部的详细结构,包括:将抽取出元数据信息写到XML文档、存储元数据到存储库以及组装形成XML文档三个部分。给出了元数据的存储策略;分析了存储模型,包括保存到XML和存储元数据的数据库模型,在此基础上分别给出了存储元数据和取出装载元数据的算法。
实验部分主要是验证元数据抽取和存储性能的实验,实验表明,通过抽取和存储算法能够很好的提取和存储元数据,性能较好。
With the rapid development of information and network technologies, more and more management information systems have been constructed among enterprises and in the departments of enterprises. However, these systems are not often interoperable and also are not able to communicate with one another to exchange information, which has resulted in the phenomenon of isolated information island, so-called“information island”, the interconnections among businesses require enterprises to solve the problem of data exchange. Therefore, it’s meaningful to solve the problem of data integration effectively. In recent years, the wide application of metadata has brought hope for enterprise data integration. The information on data structure and meaning contained in metadata can be very helpful for data integration. Therefore, metadata integration has been an important method for data integration.
This paper has introduced the characteristics of metadata、requirements of metadata integration、design principles and aims of metadata intergration, and also the paper has introduced common warehouse metamodel(CWM), on introducing recent several popular solutions of metadata integration, it has presented a solution of metadata integration and analyzed the function of every part and system flow. This paper has laid emphasis on the extraction and storage part of the metadata integration system, introduced the implementation of these two parts. The aim of extraction is to extract specific metadata from relational databases for the preparation of load and storage. In the extraction part, with the knowledge of CWM this paper presents the range and objective of extraction, also the flow of extraction and the implementation algorithm. In the module of load and storage, firstly it gives out the structure of the module including keeping metadata in XML file storing in database and extracting metadata to build an XML file. Storage stratege is presented, storage model including keeping metadata in XML and storing metadata in database are introduced, and the algorithms are introduced.
Experiment mainly shows that the performance of metadata extraction and storage, it shows that with the rules of extraction and storage algorithm the system could fulfill the job well, with good performance.
引文
[1]谢泽添.基于CWM的商业银行元数据仓库的研究与应用: [硕士学位论文].厦门:厦门大学. 2008
[2] David S Linthicum. Enterprise Application Integration, Addison-Wesley Pub, USA, 1999
[3]丁长松.基于CWM的企业元数据集成研究: [硕士学位论文].长沙:国防科技大学. 2006
[4] A Halevy. Data Integration: A Status Report. Invited talk on the German Database Conference (BTW), Leipzig, Germany, 2003
[5]曹蓟光,王申康.元数据管理策略的比较研究.计算机应用, 2001, (21): 3~5
[6]余宇荧.基于CWM的企业元数据集成研究: [硕士学位论文].长沙:国防科技大学. 2006
[7] A Brandt, Ethan L Miller, Darrell D, E Long, Lan Xue. Efficient Metadata Management in Large Distributed Storage Systems. In Proceedings of the 20th IEEE/1lth NASA Goddard Conference on Mass Storage Systems and Technologies, 2003, 290~298
[8]徐慧.元数据集成系统研究及应用: [硕士学位论文].江苏:江苏大学, 2005
[9] Prothman B. Meta data, IEEE Volume: 19, Issue: 1, Feb.2 March 2000. 20~23
[10] Tannenbaum A. Metadata Solutions: Using metamodels, repositories, xml, and enterprise portals to generate information on demand. Addison Wesley Professional, Boston, MA, 2001
[11] Stohr T Muller R, Rahm E. An Integrative and Uniform Model for Metadata Management in Data Warehousing Environments, Proceedings of BIS2000, Poznan, Poland
[12]金晶.基于元数据的商业银行员工绩效考核体系的研究与应用: [硕士学位论文].内蒙古:内蒙古大学, 2009
[13]王强,刘东波,王建新.数据仓库元数据标准研究.计算机工程, 2002, (12): 123~125
[14]聂茹,张虹.数据仓库元数据管理模式的分析与比较.计算机应用研究, 2005, (2): 57~61
[15]麻广伟.基于CWM的元数据集成的研究和应用: [硕士学位论文].湖南:中南大学, 2009
[16] Vaduva A, Dittrich KR. Metadata Management for data Warehousing. International Symposium on Database Engineering & Applications. 2001: 129~135
[17] John Poole, Dan Chang, Douglas Tolbert, David Mellor. Common Warehouse Metamodel, John Wiley & Sons, Inc. 2002
[18]施洋.数据仓库元数据集成与转换工具的设计与实现: [硕士学位论文].北京:北京交通大学. 2008
[19] Open Information Model XML Encoding, Meta Data Coalition [EB/OL]. Version1.0. 1999
[20] OMG. Common Warehouse Metamodel Specification version1.1. 2003
[21] OMG. Unified Modeling Language Specification version1.3. 1999
[22]李珊珊,宁洪等.通用数据仓库元数据模型的研究.计算机工程与科学, 2004, (5): 52~53
[23] OMG. Meta Object Facility Specification version1.4. 2000
[24]向浩翔.基于CWM的企业元数据集成环境——元数据存储和检索机制的研究: [硕士学位论文].长沙:国防科技大学. 2006
[25] OMG. XML Metadata Interchange Specification version1.1. 2000
[26]朱晓春.基于CWM的关系型数据库建模工具的研究: [硕士学位论文].安徽:合肥工业大学. 2005
[27]叶国权.支持数据集成的元数据仓库管理与维护工具的设计与实现: [硕士学位论文].湖南:国防科技大学. 2010
[28]王灵芝,张肖霞,段焰.数据仓库元数据管理研究.福建电脑, 2006, (7): 53~54
[29]于千城.商业智能系统中元数据的提取.电脑知识与技术, 2007, (22): 1115~1117
[30]李胜利,李昌清,袁平鹏.基于Web的电子期刊元数据信息提取方法.华中科技大学学报(自然科学版), 2007, (12): 13~15
[31]王洪滨,刘大昕.元数据提取综述.黑龙江大学自然科学学报, 2009, (26): 141~143
[32]王素丽,牛建强.基于XML的元数据管理框架研究.计算机工程与设计, 2008, (12): 3008~3010
[33]任辉. XML数据到关系数据映射的研究: [硕士学位论文].安徽:安徽理工大学.2006
[34]萨师煊,王珊.数据库系统概论(第三版).北京.高等教育出版社. 2002. 8~10
[35]戴超凡等.数据仓库中的元数据管理.计算机工程与科学, 2003, 25(4): 54~57
[36] David Macro著.张铭,李钦等译.元数据仓储的构建与管理.北京.机械工业出版社. 2004.5
[37]吴晓渊.基于CWM的企业数据集成研究: [硕士学位论文].湖南:国防科技大学. 2005
[38]曹迪.基于CWM的企业元数据集成研究——元数据转换及模型集成工具: [硕士学位论文].湖南:国防科技大学. 2006
[39]戴超凡,陈文伟,邓苏等.数据仓库中元数据技术研究.计算机工程与应用, 2001, (14): 85~87
[40]廖瞬,王李刚等.构造数据仓库系统的元数据.计算机工程与应用, 2001, (16): 94~96
[41]任辉. XML数据到关系数据映射的研究: [硕士学位论文].安徽:安徽理工大学. 2006
[2] David S Linthicum. Enterprise Application Integration, Addison-Wesley Pub, USA, 1999
[3]丁长松.基于CWM的企业元数据集成研究: [硕士学位论文].长沙:国防科技大学. 2006
[4] A Halevy. Data Integration: A Status Report. Invited talk on the German Database Conference (BTW), Leipzig, Germany, 2003
[5]曹蓟光,王申康.元数据管理策略的比较研究.计算机应用, 2001, (21): 3~5
[6]余宇荧.基于CWM的企业元数据集成研究: [硕士学位论文].长沙:国防科技大学. 2006
[7] A Brandt, Ethan L Miller, Darrell D, E Long, Lan Xue. Efficient Metadata Management in Large Distributed Storage Systems. In Proceedings of the 20th IEEE/1lth NASA Goddard Conference on Mass Storage Systems and Technologies, 2003, 290~298
[8]徐慧.元数据集成系统研究及应用: [硕士学位论文].江苏:江苏大学, 2005
[9] Prothman B. Meta data, IEEE Volume: 19, Issue: 1, Feb.2 March 2000. 20~23
[10] Tannenbaum A. Metadata Solutions: Using metamodels, repositories, xml, and enterprise portals to generate information on demand. Addison Wesley Professional, Boston, MA, 2001
[11] Stohr T Muller R, Rahm E. An Integrative and Uniform Model for Metadata Management in Data Warehousing Environments, Proceedings of BIS2000, Poznan, Poland
[12]金晶.基于元数据的商业银行员工绩效考核体系的研究与应用: [硕士学位论文].内蒙古:内蒙古大学, 2009
[13]王强,刘东波,王建新.数据仓库元数据标准研究.计算机工程, 2002, (12): 123~125
[14]聂茹,张虹.数据仓库元数据管理模式的分析与比较.计算机应用研究, 2005, (2): 57~61
[15]麻广伟.基于CWM的元数据集成的研究和应用: [硕士学位论文].湖南:中南大学, 2009
[16] Vaduva A, Dittrich KR. Metadata Management for data Warehousing. International Symposium on Database Engineering & Applications. 2001: 129~135
[17] John Poole, Dan Chang, Douglas Tolbert, David Mellor. Common Warehouse Metamodel, John Wiley & Sons, Inc. 2002
[18]施洋.数据仓库元数据集成与转换工具的设计与实现: [硕士学位论文].北京:北京交通大学. 2008
[19] Open Information Model XML Encoding, Meta Data Coalition [EB/OL]. Version1.0. 1999
[20] OMG. Common Warehouse Metamodel Specification version1.1. 2003
[21] OMG. Unified Modeling Language Specification version1.3. 1999
[22]李珊珊,宁洪等.通用数据仓库元数据模型的研究.计算机工程与科学, 2004, (5): 52~53
[23] OMG. Meta Object Facility Specification version1.4. 2000
[24]向浩翔.基于CWM的企业元数据集成环境——元数据存储和检索机制的研究: [硕士学位论文].长沙:国防科技大学. 2006
[25] OMG. XML Metadata Interchange Specification version1.1. 2000
[26]朱晓春.基于CWM的关系型数据库建模工具的研究: [硕士学位论文].安徽:合肥工业大学. 2005
[27]叶国权.支持数据集成的元数据仓库管理与维护工具的设计与实现: [硕士学位论文].湖南:国防科技大学. 2010
[28]王灵芝,张肖霞,段焰.数据仓库元数据管理研究.福建电脑, 2006, (7): 53~54
[29]于千城.商业智能系统中元数据的提取.电脑知识与技术, 2007, (22): 1115~1117
[30]李胜利,李昌清,袁平鹏.基于Web的电子期刊元数据信息提取方法.华中科技大学学报(自然科学版), 2007, (12): 13~15
[31]王洪滨,刘大昕.元数据提取综述.黑龙江大学自然科学学报, 2009, (26): 141~143
[32]王素丽,牛建强.基于XML的元数据管理框架研究.计算机工程与设计, 2008, (12): 3008~3010
[33]任辉. XML数据到关系数据映射的研究: [硕士学位论文].安徽:安徽理工大学.2006
[34]萨师煊,王珊.数据库系统概论(第三版).北京.高等教育出版社. 2002. 8~10
[35]戴超凡等.数据仓库中的元数据管理.计算机工程与科学, 2003, 25(4): 54~57
[36] David Macro著.张铭,李钦等译.元数据仓储的构建与管理.北京.机械工业出版社. 2004.5
[37]吴晓渊.基于CWM的企业数据集成研究: [硕士学位论文].湖南:国防科技大学. 2005
[38]曹迪.基于CWM的企业元数据集成研究——元数据转换及模型集成工具: [硕士学位论文].湖南:国防科技大学. 2006
[39]戴超凡,陈文伟,邓苏等.数据仓库中元数据技术研究.计算机工程与应用, 2001, (14): 85~87
[40]廖瞬,王李刚等.构造数据仓库系统的元数据.计算机工程与应用, 2001, (16): 94~96
[41]任辉. XML数据到关系数据映射的研究: [硕士学位论文].安徽:安徽理工大学. 2006