互联网环境下电子商务数据仓库技术的研究
摘要
在当今信息飞速发展的时代，各个企业之间的竞争愈演愈烈，只有能够快速响应和预测用户需求并不断进行技术创新的企业才有可能立于不败之地。一方面为了更好的预测用户需求，了解企业情况，数据仓库(Data Warehousing)技术与在线分析处理(On-line Analytical Processing)技术的应用得到了企业的认同和重视；另一方面通过Web方式可以充分地共享应用和信息，利用Web技术进行原有业务增值已成为信息技术的新的趋势。结合了Web技术和数据仓库技术两大热门技术的基于Web的数据仓库系统，具有其独特的优点和潜在的应用需求，将会得到越来越广泛的应用。
针对这一新的领域，论文研究互联网环境下电子商务数据仓库系统构架，目的是在于利用FISChina已有的大量数据，建立一个可以为领域专家提供数据分析服务的互联网环境下的数据仓库系统。论文通过对现实项目的研究，深入讨论了以下方面：
1、通过对现有系统地现状分析，设计出FISChina保险数据仓库系统的结构。对FISChina保险数据仓库的设计进行了详细的分析，包括数据仓库的概念模型、逻辑模型、物理模型设计和元数据管理等。
2、针对数据仓库的设计，完成了数据仓库的数据提取、转换和加载(Extraction，Transformation and Loading，简写为ETL)的设计，提出FISChina数据维护和载入模块的软件构架及业务功能。
3、针对FISChina的另一个重要组成部分——联机分析处理模块，进行详细的论述。
通过对以上各个方面的理论研究，并结合对流行的系统开发工具DELPHI、开发语言JAVA和微软的SQL Server数据库的使用，论文较好的实现了研究的目的，实现了FISChina的数据仓库系统。
Nowadays, with the rapid growth of information, the competition between enterprises is becoming more and more vehement, and only those who spare no efforts in technology innovation and forecast and respond to the demand tendency of the customers quickly and properly may win the battle. As is so often pointed out, Data Warehousing and On-line Analytical Processing (OLAP) technology are of great importance in the analysis of huge amount of data. Technology and information are now shared fully through web, and adding increment to traditional business has now become the trend of the development of information technology. The application of Data Warehousing based on the Web is brought forward because as much information and technology application as possible are needed in the Data Warehouse System (DWS) design. Data Warehouse System based on the Web combines two kinds of hot technologies today, i.e. Web technology and Data Warehousing technology, and therefore, it has its own particular merits and potential appli
cation demands, therefore, more and more enterprises are making great efforts in the application of these technologies.
Focusing on this new field, this paper is to study the construction of DWS in the Internet environment. The paper mainly discussed the main concepts, important characteristics and key techniques during the process of the development of DWS and OLAP, and provides concrete steps in implementing the design and construction of DWS and the application of OLAP based on DWS in the Web environment. Including the design of conceptual module, logical module and physical module; metadata management and indexing technology of DWS, the design and implementation ETL module of DWS, and multi-dimensional analysis and management based on the Web.
针对这一新的领域，论文研究互联网环境下电子商务数据仓库系统构架，目的是在于利用FISChina已有的大量数据，建立一个可以为领域专家提供数据分析服务的互联网环境下的数据仓库系统。论文通过对现实项目的研究，深入讨论了以下方面：
1、通过对现有系统地现状分析，设计出FISChina保险数据仓库系统的结构。对FISChina保险数据仓库的设计进行了详细的分析，包括数据仓库的概念模型、逻辑模型、物理模型设计和元数据管理等。
2、针对数据仓库的设计，完成了数据仓库的数据提取、转换和加载(Extraction，Transformation and Loading，简写为ETL)的设计，提出FISChina数据维护和载入模块的软件构架及业务功能。
3、针对FISChina的另一个重要组成部分——联机分析处理模块，进行详细的论述。
通过对以上各个方面的理论研究，并结合对流行的系统开发工具DELPHI、开发语言JAVA和微软的SQL Server数据库的使用，论文较好的实现了研究的目的，实现了FISChina的数据仓库系统。
Nowadays, with the rapid growth of information, the competition between enterprises is becoming more and more vehement, and only those who spare no efforts in technology innovation and forecast and respond to the demand tendency of the customers quickly and properly may win the battle. As is so often pointed out, Data Warehousing and On-line Analytical Processing (OLAP) technology are of great importance in the analysis of huge amount of data. Technology and information are now shared fully through web, and adding increment to traditional business has now become the trend of the development of information technology. The application of Data Warehousing based on the Web is brought forward because as much information and technology application as possible are needed in the Data Warehouse System (DWS) design. Data Warehouse System based on the Web combines two kinds of hot technologies today, i.e. Web technology and Data Warehousing technology, and therefore, it has its own particular merits and potential appli
cation demands, therefore, more and more enterprises are making great efforts in the application of these technologies.
Focusing on this new field, this paper is to study the construction of DWS in the Internet environment. The paper mainly discussed the main concepts, important characteristics and key techniques during the process of the development of DWS and OLAP, and provides concrete steps in implementing the design and construction of DWS and the application of OLAP based on DWS in the Web environment. Including the design of conceptual module, logical module and physical module; metadata management and indexing technology of DWS, the design and implementation ETL module of DWS, and multi-dimensional analysis and management based on the Web.
引文
[1] 王珊等．数据仓库技术与联机分析处理．北京：科学出版社，1998．6
[2] William A.Giovinazzo著．面向对象数据仓库设计．北京：人民邮电出版社，2000．12
[3] Harjinder S.GILL等著，王仲谋，刘书丹译．数据仓库-客户服务器计算指南．北京：清华大学出版社，1997．10
[4] 李子木，莫倩，周兴铭．数据库技术的研究现状及未来方向．计算机科学，1998，4
[5] 许向东，张全寿．数据仓库与数据发掘的应用．计算机系统应用．1998．4
[6] 练慧萍，杨明福．数据仓库技术及其在电子商务中的应用．微型电脑应用．1998．5
[7] Jiawei Han,Micheline Kamber．数据挖掘概念与技术
[8] 马宏鹏，赵新，李明，卢桂章．数据仓库原形系统设计．计算机工程与应用．2000，11
[9] 盂志青，段智华．商业销售分析数据仓库的数据可视化的一种设计．计算机工程与应用．2000，10
[10] 唐林燕．数据仓库查询与智能查询程序的实现．计算机工程与应用．2000，8
[11] 卓明强，黄兴文．数据仓库在保险业中的开发与应用，四川通信技术．2001，5
[12] 廖磷，王立刚，刘文煌．构造数据仓库系统的元数据．计算机工程与应用．2001，16
[13] W.H.Inmon.Building the data warehouse,机械工业出版社，2000，
[14] W.H.Inmon,Ken Rudin,Christopher K.Buss,Ryan Sousa著．Data Warehouse Performance.电子工业出版社．2000，5
[15] Joyce BisChoff Ted Alexandor．数据仓库技术．电子工业出版社．1998
[16] 廖磷，王立刚，刘文煌．构造数据仓库系统的元数据．计算机工程与应用．2001，16
[17] 马光志，罗治平．数据仓库、联机分析处理和联机分析开采研究．计算机应用研究．1999，16
[18] 冯晓君，李莹，蔡炯等著．数据库与工程应用．化学工业出版社．2002，1
[19] Jiawei Han,Micheline Kamber．数据挖掘概念与技术．机械工业出版社．2001，8
[20] M.SPenik,O.Sledge.SQL Server 7.0系统管理指南．清华大学出版社．1999
[21] Ahraham SilheDchatz,Henry F.Korth．数据库系统概论．机械工业出版社，2000．
[22] Bruce Eckel著．Java编程思想．机械工业出版社．2000
[23] 飞思科技产品研发中心编著．JSP应用开发详解．电子工业出版社．2002，1
[24] Ryan K.Stephens,Ronald R.Plew著．数据库设计．机械：工业出版社．2001，9
[25] Harjinder S.Gill等．数据仓库-客户服务器计算指南．北京：消华大学出版社，1997
[26] 许向东，张全寿．数据仓库与数据发掘的应用．计算机系统应用，1998；(4)
[27] J.Bischoff,TedAlexander，成栋等著．数据仓库技术．电子工业出版社，1998
[28] 罗运模，等．SQL Server2000数据仓库应用与开发．北京：人民邮电出版社，2001，7
[29] Paul J,Perrone,et al．著，张志伟等译．J2EE构造企业系统—专家组解决方案．北京：清华大学出版社，2001
[30] 李秀，廖磷，刘文煌．基于Web的数据仓库系统的研究．计算机工程，2001，11
[31] 熊忠阳，张玉芳，吴中福．三层结构中的数据库访问技术．计算机科学，2000，27
[32] 黄若波，左春，孙玉芳．基于Web环境下的OLAP技术的研究和实现．计算机工程，2000，10
[33] 胡泳，张志洁，陈福民．WEB应用服务器体系结构研究及其应用．计算机系统应用，1999，9
[34] Lei-da chen,Mark N,Frolick. Web—based data warehousing-fundamentals, challenges and solution. Information systems Management,2000,17(2):83-86
[35] Eike A, Rundensteiner, et al. Maintaining data warehouse over changing information sources. Communtications of the ACM,2000,43(6):57—62.
[36] Data warehousing: usage, architecture, and research issues. Taha, Yousy; Helal, Abdelsalam; Ahmed, Khalil M. Microcomputer Applications 1997,16(2).70-80
[37] Architecture and quality in data warehouses: an extended repository approach. Jarke, Matthias; Jeusfeld, Manfred A.; Quix, Christoph;Vassiliadis,Panos.Information Systems. 1999,24(3).229-253
[38] Making multiple views self-maintainable in a data warehouse Liang, Weifa; Li, Hui; Wang, Hui; Orlowska, Maria E. Data and Knowledge Engineering. 1999,30(2).121-134
[39] Understanding data quality in a data warehouse. Shanks, Graeme; Darke, Peta. Australian Computer Journal.-1998,30(4).122-128
[40] Data Warehousing on Automatic. Aganagic, Edin. Intelligent Enterprise -1999,2(3).-40-40
[41] Java-Charging the Warehouse.Taylor, Art.Database programming & design.-1997, 10(10).58-65
[42] Q. Chen, U. Dayal and M. Hsu, "A distributed OLAP infrastructure for e-commerce," Proc. of IFCIS International Conference on Cooperative Information Systems, p. 209-220, 1999.
[43] S. Chaudhuri and U. Dayal, "An overview of data warehouse and OLAP technology," ACM SIGMOD Record, pp. 65-74, 1997
[44] Incremental Design of a Data Warehouse. DIMITRI THEODORATOS, TIMOS SELLIS. Journal of Intelligent Information Systems.-2000,15(1).7-27
[45] Management issues in data warehousing: insights from the Housing and Development Board. James Ang, Thompson S.H. Teo. Decision Support Systems.2000,29(1).11-20
[46] Management policies and the diffusion of data warehouse: a case study using system dynamics-based decision support system. Mohammed Quaddus, Arunee Intrapairot. Decision Support Systems.2001,31(2).223-240
[47] Chaudhuri S, Dayal U. An overview of data warehousing and OLAP technology. ACM SIGMON Record, 1997, 26:65-74.
[48] Harlnarayan V, Rajaraman A, Ullman J D.Implementing data cubes efficiently. ACM SIGMON Record, 1996, (7): 205-216.
[49] Stohr. T. Muller. R.and Rahm. E. An Integrative and Uniform Model for Metadata Management in Data Warehousing Environments.
[50] Marco. David. Building and managing the Meta Data Repository a full life cycle Guide WILEY 2000
[51] http://www.olapreport.org/fasmi.html
[52] Joseph M.Firestone,Ph.d.,Data Warehouses and Data Marts:A Dynamic View, Executive Information System Inc. White Paper 3,March 27,1997;(3)
[53] Jeusfeld, M.A.; Quix, C.; Jarke, M.Design and Analysis of Quality Information forData Warehouses. In Proc. 17th International Conference on Conceptual Modeling(ER'98), Singapore, Nov 16-19, 1998.
[54] Object Management Group(OMG).Common Warehouse MetaModel Specification(1.0), JAN,2000 http://www.cwmforum.org/
[55] Daniel. Dr. Chang. T. CWM Enablement Showcase: Warehouse Metadata Interchange Made Easy Using CWM http://www.cwmforum.org/.
[2] William A.Giovinazzo著．面向对象数据仓库设计．北京：人民邮电出版社，2000．12
[3] Harjinder S.GILL等著，王仲谋，刘书丹译．数据仓库-客户服务器计算指南．北京：清华大学出版社，1997．10
[4] 李子木，莫倩，周兴铭．数据库技术的研究现状及未来方向．计算机科学，1998，4
[5] 许向东，张全寿．数据仓库与数据发掘的应用．计算机系统应用．1998．4
[6] 练慧萍，杨明福．数据仓库技术及其在电子商务中的应用．微型电脑应用．1998．5
[7] Jiawei Han,Micheline Kamber．数据挖掘概念与技术
[8] 马宏鹏，赵新，李明，卢桂章．数据仓库原形系统设计．计算机工程与应用．2000，11
[9] 盂志青，段智华．商业销售分析数据仓库的数据可视化的一种设计．计算机工程与应用．2000，10
[10] 唐林燕．数据仓库查询与智能查询程序的实现．计算机工程与应用．2000，8
[11] 卓明强，黄兴文．数据仓库在保险业中的开发与应用，四川通信技术．2001，5
[12] 廖磷，王立刚，刘文煌．构造数据仓库系统的元数据．计算机工程与应用．2001，16
[13] W.H.Inmon.Building the data warehouse,机械工业出版社，2000，
[14] W.H.Inmon,Ken Rudin,Christopher K.Buss,Ryan Sousa著．Data Warehouse Performance.电子工业出版社．2000，5
[15] Joyce BisChoff Ted Alexandor．数据仓库技术．电子工业出版社．1998
[16] 廖磷，王立刚，刘文煌．构造数据仓库系统的元数据．计算机工程与应用．2001，16
[17] 马光志，罗治平．数据仓库、联机分析处理和联机分析开采研究．计算机应用研究．1999，16
[18] 冯晓君，李莹，蔡炯等著．数据库与工程应用．化学工业出版社．2002，1
[19] Jiawei Han,Micheline Kamber．数据挖掘概念与技术．机械工业出版社．2001，8
[20] M.SPenik,O.Sledge.SQL Server 7.0系统管理指南．清华大学出版社．1999
[21] Ahraham SilheDchatz,Henry F.Korth．数据库系统概论．机械工业出版社，2000．
[22] Bruce Eckel著．Java编程思想．机械工业出版社．2000
[23] 飞思科技产品研发中心编著．JSP应用开发详解．电子工业出版社．2002，1
[24] Ryan K.Stephens,Ronald R.Plew著．数据库设计．机械：工业出版社．2001，9
[25] Harjinder S.Gill等．数据仓库-客户服务器计算指南．北京：消华大学出版社，1997
[26] 许向东，张全寿．数据仓库与数据发掘的应用．计算机系统应用，1998；(4)
[27] J.Bischoff,TedAlexander，成栋等著．数据仓库技术．电子工业出版社，1998
[28] 罗运模，等．SQL Server2000数据仓库应用与开发．北京：人民邮电出版社，2001，7
[29] Paul J,Perrone,et al．著，张志伟等译．J2EE构造企业系统—专家组解决方案．北京：清华大学出版社，2001
[30] 李秀，廖磷，刘文煌．基于Web的数据仓库系统的研究．计算机工程，2001，11
[31] 熊忠阳，张玉芳，吴中福．三层结构中的数据库访问技术．计算机科学，2000，27
[32] 黄若波，左春，孙玉芳．基于Web环境下的OLAP技术的研究和实现．计算机工程，2000，10
[33] 胡泳，张志洁，陈福民．WEB应用服务器体系结构研究及其应用．计算机系统应用，1999，9
[34] Lei-da chen,Mark N,Frolick. Web—based data warehousing-fundamentals, challenges and solution. Information systems Management,2000,17(2):83-86
[35] Eike A, Rundensteiner, et al. Maintaining data warehouse over changing information sources. Communtications of the ACM,2000,43(6):57—62.
[36] Data warehousing: usage, architecture, and research issues. Taha, Yousy; Helal, Abdelsalam; Ahmed, Khalil M. Microcomputer Applications 1997,16(2).70-80
[37] Architecture and quality in data warehouses: an extended repository approach. Jarke, Matthias; Jeusfeld, Manfred A.; Quix, Christoph;Vassiliadis,Panos.Information Systems. 1999,24(3).229-253
[38] Making multiple views self-maintainable in a data warehouse Liang, Weifa; Li, Hui; Wang, Hui; Orlowska, Maria E. Data and Knowledge Engineering. 1999,30(2).121-134
[39] Understanding data quality in a data warehouse. Shanks, Graeme; Darke, Peta. Australian Computer Journal.-1998,30(4).122-128
[40] Data Warehousing on Automatic. Aganagic, Edin. Intelligent Enterprise -1999,2(3).-40-40
[41] Java-Charging the Warehouse.Taylor, Art.Database programming & design.-1997, 10(10).58-65
[42] Q. Chen, U. Dayal and M. Hsu, "A distributed OLAP infrastructure for e-commerce," Proc. of IFCIS International Conference on Cooperative Information Systems, p. 209-220, 1999.
[43] S. Chaudhuri and U. Dayal, "An overview of data warehouse and OLAP technology," ACM SIGMOD Record, pp. 65-74, 1997
[44] Incremental Design of a Data Warehouse. DIMITRI THEODORATOS, TIMOS SELLIS. Journal of Intelligent Information Systems.-2000,15(1).7-27
[45] Management issues in data warehousing: insights from the Housing and Development Board. James Ang, Thompson S.H. Teo. Decision Support Systems.2000,29(1).11-20
[46] Management policies and the diffusion of data warehouse: a case study using system dynamics-based decision support system. Mohammed Quaddus, Arunee Intrapairot. Decision Support Systems.2001,31(2).223-240
[47] Chaudhuri S, Dayal U. An overview of data warehousing and OLAP technology. ACM SIGMON Record, 1997, 26:65-74.
[48] Harlnarayan V, Rajaraman A, Ullman J D.Implementing data cubes efficiently. ACM SIGMON Record, 1996, (7): 205-216.
[49] Stohr. T. Muller. R.and Rahm. E. An Integrative and Uniform Model for Metadata Management in Data Warehousing Environments.
[50] Marco. David. Building and managing the Meta Data Repository a full life cycle Guide WILEY 2000
[51] http://www.olapreport.org/fasmi.html
[52] Joseph M.Firestone,Ph.d.,Data Warehouses and Data Marts:A Dynamic View, Executive Information System Inc. White Paper 3,March 27,1997;(3)
[53] Jeusfeld, M.A.; Quix, C.; Jarke, M.Design and Analysis of Quality Information forData Warehouses. In Proc. 17th International Conference on Conceptual Modeling(ER'98), Singapore, Nov 16-19, 1998.
[54] Object Management Group(OMG).Common Warehouse MetaModel Specification(1.0), JAN,2000 http://www.cwmforum.org/
[55] Daniel. Dr. Chang. T. CWM Enablement Showcase: Warehouse Metadata Interchange Made Easy Using CWM http://www.cwmforum.org/.