畜产品溯源系统中异构数据整合问题研究
|
摘要
目前,畜产品安全已成为制约畜牧业发展和畜产品国际贸易的重要因素,受到世界各国政府和广大消费者的密切关注。为了有效预防牲畜口蹄疫、疯牛病、二垩英、禽流感等事件的发生,我国畜牧业信息化建设必须充分利用已有的大量宝贵的数据资源,建立具有我国自主知识产权的畜产品溯源系统,为各级决策管理者提供及时、准确、科学的辅助决策依据。
在畜牧业的信息化建设过程中,不同部门通常在不同地域使用不同的技术开发了不同的应用系统,如养殖场系统、屠宰场系统、销售系统等。这些系统都是相互独立的应用系统,就像一个个“信息孤岛”,系统在数据的描述方式、存储方式、访问方式上各不相同,差异性很大。要建立统一的畜产品溯源系统,必须对这些孤立的系统进行整合,即对畜产品溯源相关的异构数据进行整合。
异构数据整合是畜产品溯源系统建设过程中需要解决的主要问题。在已经存在的多个异构的、分布的数据库之间实现信息共享,首先需要在这些数据库系统基础上建立一个异构数据整合平台,用户才能实现对多个异构数据库系统的统一、透明的访问。
本文主要进行了两个方面的研究工作。一方面,为了提高模式匹配的自动化程度,本文设计并实现了基于属性空间的模式匹配算法。该算法将“属性空间”的概念引入到模式匹配中,文中详细介绍了其建模过程、算法设计和算法实现,并通过实验证明了该算法的可行性和有效性。
另一方面,以畜产品溯源系统中的异构数据整合问题为研究对象,本文设计了一个基于中间件的异构数据整合框架,即在数据源层与应用层中间添加一个中间件层,提供应用层对异构数据源的透明访问。其中,中间件层是异构数据整合的核心层,它向下协调各数据库系统,向上为访问集成数据的应用者提供统一的访问接口。中间件层由包装器和中介器两部分组成,本文结合畜产品溯源系统,着重对中介器和包装器的设计与实现进行了研究。初步实验表明,该平台实现了畜产品溯源系统中的异构数据整合工作,对实际工程有一定的指导作用。
At present, the safety of livestock products, which is concerned by different governments and consumers, has restricted the development of stockbreeding and the international trade of livestock products. For the sake of preventing foot-and–mouth disease, mad cow disease, Dioxin and fowl typhoid efficiently, we must take advantage of the existing data resources to establish the livestock products traceability system, which can provide timely, exact and scientific assistant decision-making evidence for managers.
During the process of informationization of stockbreeding, different units usually made use of different technology to develop different application systems like farm system, butcher system and market system. These systems are isolated application systems, which are like some“information isolated islands”. These systems are different in description method, storage method and access method. In order to establish a uniform livestock products traceability system, we must integrate these isolated systems. In other words, we must integrate the heterogeneous data which is related to livestock products traceability.
Heterogeneous data integration is the main problem in the livestock products traceability system. In order to share the information of multiple heterogeneous distributed databases, firstly, we need to establish a heterogeneous data integration system based on these database systems. In this way, the users can access these heterogeneous database systems comformably and transparently.
This paper carried out two aspects of research work. On the one hand, in order to increase the automatic degree of schema matching, this paper designed and implemented a attribute space-based schema matching algorithm. This algorithm imported the concept of“attribute space”into the schema matching. We introduced the modeling process, designment and implementation of this algorithm and validated the feasibility and validity of this algorithm.
On the other hand, based on the problem of heterogeneous data integration in the livestock products traceability system, we designed a heterogeneous data integration framwork based on mediation. We added a mediation layer between the data source layer and application layer. Medation layer is the core layer of heterogeneous data integraion, which harmonized the database systems and provided a uniform interface for the users who access the integraed data. The mediation layer is composed of mediator and wrappers. We introduced the designment and implementation of mediator and wrappers particularly. The initial experiment indicated that heterogeneous data integration system has implemented the heterogeneous data integration work in the livestock products traceability system and can supervise the actual project.
在畜牧业的信息化建设过程中,不同部门通常在不同地域使用不同的技术开发了不同的应用系统,如养殖场系统、屠宰场系统、销售系统等。这些系统都是相互独立的应用系统,就像一个个“信息孤岛”,系统在数据的描述方式、存储方式、访问方式上各不相同,差异性很大。要建立统一的畜产品溯源系统,必须对这些孤立的系统进行整合,即对畜产品溯源相关的异构数据进行整合。
异构数据整合是畜产品溯源系统建设过程中需要解决的主要问题。在已经存在的多个异构的、分布的数据库之间实现信息共享,首先需要在这些数据库系统基础上建立一个异构数据整合平台,用户才能实现对多个异构数据库系统的统一、透明的访问。
本文主要进行了两个方面的研究工作。一方面,为了提高模式匹配的自动化程度,本文设计并实现了基于属性空间的模式匹配算法。该算法将“属性空间”的概念引入到模式匹配中,文中详细介绍了其建模过程、算法设计和算法实现,并通过实验证明了该算法的可行性和有效性。
另一方面,以畜产品溯源系统中的异构数据整合问题为研究对象,本文设计了一个基于中间件的异构数据整合框架,即在数据源层与应用层中间添加一个中间件层,提供应用层对异构数据源的透明访问。其中,中间件层是异构数据整合的核心层,它向下协调各数据库系统,向上为访问集成数据的应用者提供统一的访问接口。中间件层由包装器和中介器两部分组成,本文结合畜产品溯源系统,着重对中介器和包装器的设计与实现进行了研究。初步实验表明,该平台实现了畜产品溯源系统中的异构数据整合工作,对实际工程有一定的指导作用。
At present, the safety of livestock products, which is concerned by different governments and consumers, has restricted the development of stockbreeding and the international trade of livestock products. For the sake of preventing foot-and–mouth disease, mad cow disease, Dioxin and fowl typhoid efficiently, we must take advantage of the existing data resources to establish the livestock products traceability system, which can provide timely, exact and scientific assistant decision-making evidence for managers.
During the process of informationization of stockbreeding, different units usually made use of different technology to develop different application systems like farm system, butcher system and market system. These systems are isolated application systems, which are like some“information isolated islands”. These systems are different in description method, storage method and access method. In order to establish a uniform livestock products traceability system, we must integrate these isolated systems. In other words, we must integrate the heterogeneous data which is related to livestock products traceability.
Heterogeneous data integration is the main problem in the livestock products traceability system. In order to share the information of multiple heterogeneous distributed databases, firstly, we need to establish a heterogeneous data integration system based on these database systems. In this way, the users can access these heterogeneous database systems comformably and transparently.
This paper carried out two aspects of research work. On the one hand, in order to increase the automatic degree of schema matching, this paper designed and implemented a attribute space-based schema matching algorithm. This algorithm imported the concept of“attribute space”into the schema matching. We introduced the modeling process, designment and implementation of this algorithm and validated the feasibility and validity of this algorithm.
On the other hand, based on the problem of heterogeneous data integration in the livestock products traceability system, we designed a heterogeneous data integration framwork based on mediation. We added a mediation layer between the data source layer and application layer. Medation layer is the core layer of heterogeneous data integraion, which harmonized the database systems and provided a uniform interface for the users who access the integraed data. The mediation layer is composed of mediator and wrappers. We introduced the designment and implementation of mediator and wrappers particularly. The initial experiment indicated that heterogeneous data integration system has implemented the heterogeneous data integration work in the livestock products traceability system and can supervise the actual project.
引文
1刘学兵,郭娜.绿色的梦想与实践——关于畜产品安全的思考[J].畜牧市场.2009,01:11-13.
2 A.Mousavi, M. Sarhadi, A. lenk, S. Fawcett. Tracking and traceability in the meat processing industry: a solution[J]. British-Food-Journal, 2002, Vol.104:7-19.
3 Lucy Atherton. SeamlessUK-building bridges between information islands[J]. New Library World. 2002, Vol.103:467-473.
4 Dogas A, Dengi C, Kilic E, Ozhan G, Ozcan F, Nural S, Evrendilek C, Halici U, Arpinar B, Koksal P, Kesim N, Mancuhan S. METU Interoperable Database System[J]. ACM SIGMOD Record. September 1995, 24(3): 56-61.
5 Halevy A. Data integration: a status report[Z]. Invited talk on the German Database Conference(BTW), Leipzing, Germany, February, 2003.
6 Hazem T. EL-Khatib, Howard Williams, David H. Matwick, et al. Using a Distributed Approach to Retrieve and Integrate Information from Heterogeneous Distributed Databases [J]. The Computer Journal, 2002, Vol.45, No.4:381-394.
7 Mukesh Mohania, Manish Bhide. New Trends in Information Integration[C]. Proceedings of the 2nd international conference on Ubiquitous information management and communication. Jan. 2008: 74-81.
8 Florescu D, Levy A, Mendelzon A. Database Techniques for the World-Wide Web: A Survey[J]. ACM SIGMOD Record, 1998; Vo.l27, No 3:59-74.
9 Juha Muilu, Leena Peltonen, Jan-Eric Litton. The federated database– a basis for biobank-based post-genome studies, integrating phenome and genome data from 600000 twin pairs in Europe[J]. Eur J Hum Genet. 2007, 15(7):718-723.
10 Ashraf Aboulnaga, Kareem El Gebaly.μBE: User Guided Source Selection and Schema Mediation for Internet Scale Data Integraton[C]. Proceedings of IEEE
23rd International Conference on Data Engineering. 2007: 186-195.
11 Rene de Koster, Tho Le-Duc, Kees Jan Roodbergen. Design and control of warehouse order picking: A literature review. European Journal of Operational Research. 2007, 182(2): 481-501.
12 Rafi Ahmed, Joseph Albert, Weimin Du, et al. An Overview of Pegasus[C]. In: Hans-Jorg Schek, Amit P. Sheth, Bogdan D. Czejdo(Eds.): Proceeding of International Workshop on Research Issues in Data Engineering: Interoperability in Multidatabase Systems. Vienna, Austria, 1993: 273-277.
13 Albert D’Andrea, Phil Janus. UniSQL’s next-generation object-relational database management system[J]. ACM SIGMOD Record, September, 1996. Vol.25, No.3: 70-76.
14 Evaggerlia Pitoura, Omran Bukhres, Ahmed Elmagarmid. Object Orientation in Multidatabase Systems[J]. ACM Computing Surveys. June 1995. Vol. 27,No.2:141-195.
15 Chen Li, Ramana Yerneni, Vasilis Vassalos, Hector Garcia-Molina. Capability Based Mediation in TSIMMIS[C]. Proceedings of the 1998 ACM SIGMOD international conference on Management of data. June. 1998: 564-566.
16王宁,王能斌.异构数据源集成系统查询分解和优化的实现[J],软件学报,2000, 11(2):222-228.
17石祥滨,张斌等.SCOPE/CIMS系统中模式集成的形式化基础[J].计算机学报,1998,21(11):1015-1021.
18 Jian Tang, Using dummy reads to maintain consistency in heterogeneous database systems[C], Proc. Third Workshop on Future Trends of Distributed Computing Systems, 1992. 312-317.
19 Maurizio Lenzerini. Data Integration: A Theoretical Perspective[J]. In: Proc. Of the ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, 2002. 233-246.
20张学旺,汪林林,马中峰.数字化校园综合应用软件平台的关键技术[J].计算机工程. 2007.12, 33(23): 267-269.
21 Hass, L M Millert, B newsmonger, Wimmers. Transforming Heterogeneous Data With Database Middleware: Beyond Integration[J]. IEEE Tech. Bull. Data Engineering Vol.22, No.1,1999: 31-36.
22 Bray T, Paoli J, Sperberg-McQueenC. Extensible Markup Language(XML)1.0[Z].W3C Recommendation. http://www.w3.org/TR/REC-xml.
23顾天竺,沈洁等.基于XML的异构数据集成模式的研究[J].计算机应用研究. 2007, 24(4): 94-96.
24熊忠阳,任芳等.基于XML描述的数据挖掘结果的存储方法[J].计算机工程与设计. 2006. 27(20): 3874-3877.
25周园春,李淼等.中间件技术综述[J].计算机工程与应用. 2002. (15): 80-82.
26 Sheth A, Kashyap V. So far(schematically) yet so near(semantically)[A]. Proceedings of the IFIP TC2/WG2.6 conference on semantics of interoperable database systems[C]. Australia: North Holland, 1992.
27强保华.异构数据库语义集成技术研究[D].博士学位论文:重庆大学, 2004.
28 Navathe S, Peter B. Integrating user views in database design[J]. Computers, 1986, Vol.19, No.1: 50-62.
29 Li W S, Clifton C. Using field specifications to determine attribute equivalence in heterogeneous databases[A]. Third international workshop on research issues on data engineering: interoperability in multidatabase systems[C]. Vienna, Austria, 1993:174-177.
30 J. Kang, J. Naughton. On Schema Matching with Opaque Column Names and Data Values[C]. In Proceeding of SIGMOD 2003, San Diego, California, USA, 2003:9-12.
31 A. Doan, J. Madhavan, P. Domingos, et al. Learning to Map between Ontologies on the Semantic Web[C]. In Proceeding of the Eleventh International World Wide Web Conference(WWW2002), Honolulu, Hawaii, USA, 2002: 662-673.
32程乾生.属性集和属性综合评价系统[J].系统工程理论与实践, 1997年9月,第9期: 1-9.
33 http://gceclub.sun.com.cn/Java_Docs/html/zh_CN/api/constant-values.html#java.sql.Types.BIT.
34 Bunke H, Jiang X Y, Abegglen K, et al. On the weighted Mean of a Pair of Strings[J]. Pattern Analysis & Applications, 2002, 5(5): 23-30.
35赖欣.基于相关反馈和综合特征的图像内容检索系统研究和实现[D].硕士学位论文.2008年4月.电子科技大学.
36常庭懋,韩中庚.用“匈牙利算法”求解一类最优化问题[J].信息工程大学学报.2004,5(1): 60-62.
37王鹏,伊鹏等.匈牙利算法在输入排队调度仿真中的应用研究[J].计算机应用.2003年7月, 23(7): 4-6.
38 You Li, Dong-Bo Liu, Wei-Ming Zhang. Schema Matching Using Neural Network[C]. Proceedings of the 2005 IEEE/WIC/ACM International Conferenceon Web Intelligence, Compiegne, France, 2005: 743-746.
39 Erhard Rahm, Philip A. Bernstein. A Survey of Approaches to Automatic Schema Matching[J]. The VLDB Jounal, 2001, 10(4): 334-350.
40 AMIT P. SHETH, JAMES A. LARSON. Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases[J]. ACM Computing Surveys(CSUR). 1990. 22(3): 183-236.
41 Denise Draper, Alon Y.Halevy, Daniel S. Weld. The Nimble XML Data Integration System[C], Proc. of Int. Conf. On Data Engineering(ICDE). 2001: 155-160.
42 Yannis Papakonstantinou, Hector Garcia–Molina, Jeffrey Ullman. Medmaker: A Mediation System Based on Declarative Specifications[Z]. Data Engineering. Proceedings of the Twelfth International Conference, 1996: 132-141.
43 Risch T, Josifovski V, Katchaounov T. Functional data integration in a distributed mediator system. In: Gray P, Kerschberg L, King P, Poulovassilis A(eds.): Functional approach to data management—modeling, analyzing and integrating heterogeneous data. ISBN 3-540-00375-4, Springer. 2003:211-238.
44王秀娟,异构环境信息集成[D],硕士学位论文,中北大学,2007年4月.
45梁莹,基于视图的数据库安全模型的研究[J],中山大学学报,2005年.25(3):134-137.
46汤国华,叶丹等.基于设计模式的通用数据库视图生成方法[J].计算机辅助工程. 2008,17(1): 72-76.
2 A.Mousavi, M. Sarhadi, A. lenk, S. Fawcett. Tracking and traceability in the meat processing industry: a solution[J]. British-Food-Journal, 2002, Vol.104:7-19.
3 Lucy Atherton. SeamlessUK-building bridges between information islands[J]. New Library World. 2002, Vol.103:467-473.
4 Dogas A, Dengi C, Kilic E, Ozhan G, Ozcan F, Nural S, Evrendilek C, Halici U, Arpinar B, Koksal P, Kesim N, Mancuhan S. METU Interoperable Database System[J]. ACM SIGMOD Record. September 1995, 24(3): 56-61.
5 Halevy A. Data integration: a status report[Z]. Invited talk on the German Database Conference(BTW), Leipzing, Germany, February, 2003.
6 Hazem T. EL-Khatib, Howard Williams, David H. Matwick, et al. Using a Distributed Approach to Retrieve and Integrate Information from Heterogeneous Distributed Databases [J]. The Computer Journal, 2002, Vol.45, No.4:381-394.
7 Mukesh Mohania, Manish Bhide. New Trends in Information Integration[C]. Proceedings of the 2nd international conference on Ubiquitous information management and communication. Jan. 2008: 74-81.
8 Florescu D, Levy A, Mendelzon A. Database Techniques for the World-Wide Web: A Survey[J]. ACM SIGMOD Record, 1998; Vo.l27, No 3:59-74.
9 Juha Muilu, Leena Peltonen, Jan-Eric Litton. The federated database– a basis for biobank-based post-genome studies, integrating phenome and genome data from 600000 twin pairs in Europe[J]. Eur J Hum Genet. 2007, 15(7):718-723.
10 Ashraf Aboulnaga, Kareem El Gebaly.μBE: User Guided Source Selection and Schema Mediation for Internet Scale Data Integraton[C]. Proceedings of IEEE
23rd International Conference on Data Engineering. 2007: 186-195.
11 Rene de Koster, Tho Le-Duc, Kees Jan Roodbergen. Design and control of warehouse order picking: A literature review. European Journal of Operational Research. 2007, 182(2): 481-501.
12 Rafi Ahmed, Joseph Albert, Weimin Du, et al. An Overview of Pegasus[C]. In: Hans-Jorg Schek, Amit P. Sheth, Bogdan D. Czejdo(Eds.): Proceeding of International Workshop on Research Issues in Data Engineering: Interoperability in Multidatabase Systems. Vienna, Austria, 1993: 273-277.
13 Albert D’Andrea, Phil Janus. UniSQL’s next-generation object-relational database management system[J]. ACM SIGMOD Record, September, 1996. Vol.25, No.3: 70-76.
14 Evaggerlia Pitoura, Omran Bukhres, Ahmed Elmagarmid. Object Orientation in Multidatabase Systems[J]. ACM Computing Surveys. June 1995. Vol. 27,No.2:141-195.
15 Chen Li, Ramana Yerneni, Vasilis Vassalos, Hector Garcia-Molina. Capability Based Mediation in TSIMMIS[C]. Proceedings of the 1998 ACM SIGMOD international conference on Management of data. June. 1998: 564-566.
16王宁,王能斌.异构数据源集成系统查询分解和优化的实现[J],软件学报,2000, 11(2):222-228.
17石祥滨,张斌等.SCOPE/CIMS系统中模式集成的形式化基础[J].计算机学报,1998,21(11):1015-1021.
18 Jian Tang, Using dummy reads to maintain consistency in heterogeneous database systems[C], Proc. Third Workshop on Future Trends of Distributed Computing Systems, 1992. 312-317.
19 Maurizio Lenzerini. Data Integration: A Theoretical Perspective[J]. In: Proc. Of the ACM SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, 2002. 233-246.
20张学旺,汪林林,马中峰.数字化校园综合应用软件平台的关键技术[J].计算机工程. 2007.12, 33(23): 267-269.
21 Hass, L M Millert, B newsmonger, Wimmers. Transforming Heterogeneous Data With Database Middleware: Beyond Integration[J]. IEEE Tech. Bull. Data Engineering Vol.22, No.1,1999: 31-36.
22 Bray T, Paoli J, Sperberg-McQueenC. Extensible Markup Language(XML)1.0[Z].W3C Recommendation. http://www.w3.org/TR/REC-xml.
23顾天竺,沈洁等.基于XML的异构数据集成模式的研究[J].计算机应用研究. 2007, 24(4): 94-96.
24熊忠阳,任芳等.基于XML描述的数据挖掘结果的存储方法[J].计算机工程与设计. 2006. 27(20): 3874-3877.
25周园春,李淼等.中间件技术综述[J].计算机工程与应用. 2002. (15): 80-82.
26 Sheth A, Kashyap V. So far(schematically) yet so near(semantically)[A]. Proceedings of the IFIP TC2/WG2.6 conference on semantics of interoperable database systems[C]. Australia: North Holland, 1992.
27强保华.异构数据库语义集成技术研究[D].博士学位论文:重庆大学, 2004.
28 Navathe S, Peter B. Integrating user views in database design[J]. Computers, 1986, Vol.19, No.1: 50-62.
29 Li W S, Clifton C. Using field specifications to determine attribute equivalence in heterogeneous databases[A]. Third international workshop on research issues on data engineering: interoperability in multidatabase systems[C]. Vienna, Austria, 1993:174-177.
30 J. Kang, J. Naughton. On Schema Matching with Opaque Column Names and Data Values[C]. In Proceeding of SIGMOD 2003, San Diego, California, USA, 2003:9-12.
31 A. Doan, J. Madhavan, P. Domingos, et al. Learning to Map between Ontologies on the Semantic Web[C]. In Proceeding of the Eleventh International World Wide Web Conference(WWW2002), Honolulu, Hawaii, USA, 2002: 662-673.
32程乾生.属性集和属性综合评价系统[J].系统工程理论与实践, 1997年9月,第9期: 1-9.
33 http://gceclub.sun.com.cn/Java_Docs/html/zh_CN/api/constant-values.html#java.sql.Types.BIT.
34 Bunke H, Jiang X Y, Abegglen K, et al. On the weighted Mean of a Pair of Strings[J]. Pattern Analysis & Applications, 2002, 5(5): 23-30.
35赖欣.基于相关反馈和综合特征的图像内容检索系统研究和实现[D].硕士学位论文.2008年4月.电子科技大学.
36常庭懋,韩中庚.用“匈牙利算法”求解一类最优化问题[J].信息工程大学学报.2004,5(1): 60-62.
37王鹏,伊鹏等.匈牙利算法在输入排队调度仿真中的应用研究[J].计算机应用.2003年7月, 23(7): 4-6.
38 You Li, Dong-Bo Liu, Wei-Ming Zhang. Schema Matching Using Neural Network[C]. Proceedings of the 2005 IEEE/WIC/ACM International Conferenceon Web Intelligence, Compiegne, France, 2005: 743-746.
39 Erhard Rahm, Philip A. Bernstein. A Survey of Approaches to Automatic Schema Matching[J]. The VLDB Jounal, 2001, 10(4): 334-350.
40 AMIT P. SHETH, JAMES A. LARSON. Federated Database Systems for Managing Distributed, Heterogeneous, and Autonomous Databases[J]. ACM Computing Surveys(CSUR). 1990. 22(3): 183-236.
41 Denise Draper, Alon Y.Halevy, Daniel S. Weld. The Nimble XML Data Integration System[C], Proc. of Int. Conf. On Data Engineering(ICDE). 2001: 155-160.
42 Yannis Papakonstantinou, Hector Garcia–Molina, Jeffrey Ullman. Medmaker: A Mediation System Based on Declarative Specifications[Z]. Data Engineering. Proceedings of the Twelfth International Conference, 1996: 132-141.
43 Risch T, Josifovski V, Katchaounov T. Functional data integration in a distributed mediator system. In: Gray P, Kerschberg L, King P, Poulovassilis A(eds.): Functional approach to data management—modeling, analyzing and integrating heterogeneous data. ISBN 3-540-00375-4, Springer. 2003:211-238.
44王秀娟,异构环境信息集成[D],硕士学位论文,中北大学,2007年4月.
45梁莹,基于视图的数据库安全模型的研究[J],中山大学学报,2005年.25(3):134-137.
46汤国华,叶丹等.基于设计模式的通用数据库视图生成方法[J].计算机辅助工程. 2008,17(1): 72-76.