径向基神经网络和支持向量机的参数优化方法研究及应用
摘要
本文主要研究基于主成分分析(Principal Component Analysis，PCA)的径向基(Radial basis Function，RBF)神经网络和支持向量机(Support Vector Machine，SVM)的参数优化方法及其在建筑工程投资估算中的应用。
RBF神经网络是一种基于正则化理论的三层前馈网络，它有很好的泛化能力并且不会陷入局部极小，已证明它能以任意精度逼近任一连续函数。RBF神经网络的关键在于径向基函数中心和宽度的选取。
支持向量机是基于统计学习理论的一种新的模式识别技术，它通过Mercer核函数将原始空间中的非线性问题转化为另一个高维空间中的线性问题，在这个变换的高维空间求最优或广义最优分类面。参数选择是影响支持向量机实用性能的重要因素。
针对RBF神经网络和支持向量机中径向基函数宽度(参数)难以合理确定的难题，本文在分析了以上两种网络中径向基函数宽度确定方法的基础上提出了一种在样本各分量相互独立条件下的参数优化方法，该方法是将样本各分量的方差这一先验信息引入到径向基函数宽度的确定方法中以达到参数优化目的。由于主成分分析可以消除原始样本分量间的相关性，从而为参数优化带来了方便；同时主成分分析通过降维处理可以减少网络的输入维数，从而降低网络结构的复杂性，对RBF神经网络来说可尽可能地避免“维数灾”问题。
本文最后将基于主成分分析的RBF神经网络和支持向量机的估算方法用于建筑工程投资估算问题。实验结果表明，本文提出改进后方法的估算精度比改进前方法有了明显的提高。
This paper pays main attention to a parameter optimization method of Radial Basis Function (RBF) neural network and Support Vector Machine (SVM) based on Principal Component Analysis (PCA) for solving complex estimation problems of investment in the engineering field.
RBF neural network based on regularization theory is a feedforward network with three layers.It has strong generalization ability and no problem of the local minimum.It has been proved that RBF neural network can approximate randomly nonlinear function under the condition of the randomly given approximation precision.The key of designing RBF neural network is to determine centers and width of radial basis function.
SVM based on statistical learning theory is a new pattern recognition technology.It uses Mercer kernels to construct nonlinear decision functions by training a classifier to perform a linear separation in some high-dimensional space which is nonlinearly related to input space. Parameter selection is an important issue to make SVM practically useful.
For solving the problem of determining parameter of RBF neural network and SVM rationally, a new method based on analysis of determining parameter of two networks is proposed under the condition of no correlation among the vectors .It applies variance of each vector to optimize the parameter.Because PCA can be applied to eliminate the correlation among the inputs,it is convenient that the parameter is optimized.At the same time, PCA can reduce the dimension of inputs and decrease the complexity of network in order to avoid the problem of Curse of Dimensionability of RBF neural network possibly.
Finally, the presented methods are used to estimate project cost in estimation of investment. The experimental results show that the presented methods are able to effectively improve the accuracy of estimation.
RBF神经网络是一种基于正则化理论的三层前馈网络，它有很好的泛化能力并且不会陷入局部极小，已证明它能以任意精度逼近任一连续函数。RBF神经网络的关键在于径向基函数中心和宽度的选取。
支持向量机是基于统计学习理论的一种新的模式识别技术，它通过Mercer核函数将原始空间中的非线性问题转化为另一个高维空间中的线性问题，在这个变换的高维空间求最优或广义最优分类面。参数选择是影响支持向量机实用性能的重要因素。
针对RBF神经网络和支持向量机中径向基函数宽度(参数)难以合理确定的难题，本文在分析了以上两种网络中径向基函数宽度确定方法的基础上提出了一种在样本各分量相互独立条件下的参数优化方法，该方法是将样本各分量的方差这一先验信息引入到径向基函数宽度的确定方法中以达到参数优化目的。由于主成分分析可以消除原始样本分量间的相关性，从而为参数优化带来了方便；同时主成分分析通过降维处理可以减少网络的输入维数，从而降低网络结构的复杂性，对RBF神经网络来说可尽可能地避免“维数灾”问题。
本文最后将基于主成分分析的RBF神经网络和支持向量机的估算方法用于建筑工程投资估算问题。实验结果表明，本文提出改进后方法的估算精度比改进前方法有了明显的提高。
This paper pays main attention to a parameter optimization method of Radial Basis Function (RBF) neural network and Support Vector Machine (SVM) based on Principal Component Analysis (PCA) for solving complex estimation problems of investment in the engineering field.
RBF neural network based on regularization theory is a feedforward network with three layers.It has strong generalization ability and no problem of the local minimum.It has been proved that RBF neural network can approximate randomly nonlinear function under the condition of the randomly given approximation precision.The key of designing RBF neural network is to determine centers and width of radial basis function.
SVM based on statistical learning theory is a new pattern recognition technology.It uses Mercer kernels to construct nonlinear decision functions by training a classifier to perform a linear separation in some high-dimensional space which is nonlinearly related to input space. Parameter selection is an important issue to make SVM practically useful.
For solving the problem of determining parameter of RBF neural network and SVM rationally, a new method based on analysis of determining parameter of two networks is proposed under the condition of no correlation among the vectors .It applies variance of each vector to optimize the parameter.Because PCA can be applied to eliminate the correlation among the inputs,it is convenient that the parameter is optimized.At the same time, PCA can reduce the dimension of inputs and decrease the complexity of network in order to avoid the problem of Curse of Dimensionability of RBF neural network possibly.
Finally, the presented methods are used to estimate project cost in estimation of investment. The experimental results show that the presented methods are able to effectively improve the accuracy of estimation.
引文
[1] 范金城，梅长林．数据分析．北京：清华大学出版社，1999
[2] Chan JCL, Jiu En Shi. Application of projection pursuit principal component analysis method to climate studies.International Journal of climatology, 1997, 17(1):103～113
[3] Fung T, Ledrew E. Application of principal component analysis to change detection.Photogrammetric Engineering and remote Sensing, 1987, 53(12): 1649～1658
[4] 盼庆仲．主成分分析及与TOPSIS法用于医院候诊室卫生评比的对比分析．数理医药学杂志，1999，2(12)
[5] 朱顺泉，徐国祥．我国上市公司财务状况的主成分分析研究．重庆工商大学学报，2003，(4)：48～50
[6] 张清贵．人工神经网络导论．北京：中国水利水电出版社，2004
[7] A.K.Jain, and J.Mao. Neural Networks and Patem recognition.In Computational Intelligence: Imitating Life.IEEE Press, Piscataway, NJ. 1994
[8] D.Psaltis, A .Sideris, A. A.Yamamura.A multilayed neural network controller. IEEE Control Systems.1998, 8(3): 17～21
[9] Friedhelm Schwenker, Hans A. Kestler, GuEnther Palm.Three learning phases for radial-basis-functionn etworks. NeuralN etworks, 2001(14): 439～458
[10] Simon Haykin. Neural networks: A Comprehensive Foundation.Upper Saddle River, NJ: Prentice Hall, 1999
[11] 张学工 译．统计学习理论的本质．北京：清华大学出版社，2000
[12] Vapnik V N, Chervonenkis A Ja. On the uniform convergence of relative frequencies of events to their probabilities. Doklady Akademii Nauk USSR, 1968
[13] Blumer A, Ehrenfeucht A, Haussler D etal. Learnability and the Vapnik-Chervonenkis dimension. Journal of the ACM. 1989, 36(4): 929～965
[14] Vapnik V N, Chervonenkis A Ja. Necessary and sufficient conditions for the uniform convergence of the means to their expectation.Probability Theory and Its Applications. 1981, 26:532～553
[15] Cortes C, Vapnik V N. Support vector networks.Machine Learning. 1995, 20: 273～297
[16] 周伟达，张莉，焦李成．支撑矢量机推广能力分析．电子学报，2001，29(5)：590～594
[17] SchelkopfB. Support Vector Learning. R Oldenbourg Verlag, Munich, 1997
[18] Burges CJC. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 1998, 2(2): 121～167
[19] 马云潜，张学工．支持向量机函数拟合在分形插值中的应用．清华大学学报(自然科学版)，2000，40(3)：76～78，103
[20] Weston J, Gammerman A, Stitson M O etal. Density estimation using support vector machines. Technical report, Royal Holloway College, Report number CSD-TR-97-23, 1997
[21] 高学，金连文，尹俊勋等．一种基于支持向量机的手写汉字识别方法．电子学报．2002，30(5)：651～654
[22] Maragoudakis M, Kavallieratou E, Fakotakis N. Incorporating conditional independence assumption with support vector machines to enhance handwritten character segmentation performance.In Proceedings. 16th International Conference on Pattern Recognition, 2002, 3: 11～15
[23] Maragoudakis M, Kavallieratou E, Fakotakis N. Improving handwritten character segmentation by incorporating Bayesian knowledge with support vector machines.In Proceedings. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2002, 4: 13～17
[24] Li J M, Zhang B, Lin F Z. Nonlinear speech model based on support vector machine and wavelet transform.In Proceedings. 15th IEEE International Conference on Tools with Artificial Intelligence, 2003: 3～5
[25] Gordan M, Kotropoulos C, Pitas I. Application of support vector machines classifiers to visual speech recognition.In Proceedings. 2002 International Conference on Image Processing, 2002, 3:24～28
[26] Chakrabartty S, Singh G, Cauwenberghs G.Hybrid support vector machine/hidden Markov model approach for continuous speech recognition.In Proceedings. The 43rd IEEE Midwest Symposium on Circuits and Systems, 2000, 2:8～11
[27] 祝海龙，屈梁生，张海军．基于小波变换和支持向量机的人脸检测系统．西安交通大学学报，2002，36(9)：947～950
[28] Tefas A, Kotropoulos C, Pitas I. Using support vector machines to enhance the performance of elastic graph matching for fi'ontal face authentication. IEEE Transactions on Pattern Analysis and Machine Intelligence.2001, 23(7): 735～746
[29] Joachims T. Text categorization with support vector machines. Technical report, LS Ⅷ Number 23 , University of Dortmund, 1997
[30] Muller K R, Smola A, R_tsch G etal. Predicting time series with support vector machines.In Proceedings, International Conference on Artificial Neural Networks, 1997: 999～105
[31] Mukherjee S, Osuna E, Girosi E Nonlinear prediction of chaotic time series using a support vector machine. In Proceeding of the IEEE Workshop on Neural Networks for Signal Processing 7, 1997: 511～519
[32] Powell M J D. Radial Basis Function for multivariable interpolation, Review IMA Conf. on Algorithms for the Approximation of Functions and Data. RMCS Shrivanham, 1985
[33] D. S. Broomhead, D . Lowe. Multivarialble function interpolation and adaptive network. Complex Systems, 1988(2)
[34] J.Moody, C.Darken. Fast L earning in Networks of L ocally-tuned Processing units.Neural Computation, 1989(1): 281～294
[35] Jackon I R H. Convergence properties of radial basis function. Constr Approx, 1988(4): 243～264
[36] J.Park, Etal. Universal Approximations Using RBF Networks. Neural Computation, 1991(3): 246～257
[37] Powell M J D. The Theory of RBF Approximation in 1990, Numerical Analysis Report.University of Cambrige, 1990
[38] D.R.Hush, and B.G. Hone.Progress in supervised neural networks. IEEE signal Processing Magazine, 1993, 1(1): 8～37
[39] 王旭东,邵惠鹤．RBF神经网络理论及其在控制中的应用．信息与控制,1997,26(4)：272～284
[40] S.Chen, C.F.N. Cowan and EM.Grant.Orthogonal Least Squares Learning Algorithm for Radial Basis Function Networks. IEEE Trans, Neural Networks, 1991, 2(2): 302～309
[41] Q.M.Zhu and S.A.Billings. Fast orthogonal identification of nonlinear Stochastic models and radial basis function neural networks. Int. J. Contr, 1996, 64(5): 871～886
[42] Kaufman L. Solving the qp problem for support vector training. In Proceedings of the 1997 NIPS Workshop on Support Vector Machines.1998: 237～246
[43] Ancona N, Cicirelli G, Distante A. Complexity reduction and parameter selection in support vector machines. IJCNN'02, Proceedings of the 2002 International Joint Conference on Neural Networks, 2002, 3: 2375～2380
[44] Jordaan E M, Smits G F. Estimation of the regularization parameter for support vector regression.IJCNN'02, Proceedings of the 2002 International Joint Conference on Neural Networks, 2002, 3:2192～2197
[45] Ito K, Nakano R. Optimizing support vector regression hyperparameters based on cross-validation. 2003 Proceedings of the International Joint Conference on Neural Networks, 2003, 3:2077～2082
[46] Wahba G, Lin Y, Zhang H. Margin-like quantities and generalized approximate cross validation for support vector machines. Proceedings of the 1999 IEEE Signal Processing Society Workshop on Neural Networks for Signal Processing Ⅸ, 1999: 12～20
[47] Hotelling, H. Analysis of a Complex of Statistical Variables and Principal Components. Journal of Educational Psychology, 1933(24): 417～441, 498～520
[48] 飞思科技产品研发中心．神经网络理论与MATLAB7实现．北京：电子工业出版社，2005
[49] 李柞泳，邓新民．BP网络的过拟合现象满足的测不准关系式．红外与毫米波学报，2000，19(2)：142～144
[50] V.Vapnik.An overview of statistical learning theory.IEEE Trans Neural Networks, 1999, 10(5): 998～999
[51] 李映，焦李成．统一框架下软计算方法和支持向量机的机理研究．模式识别与人工智能，2002，15(2)：173～178
[52] 张学工．关于统计学习理论与支持向量机．自动化学报，2000，16(1)：33～43
[53] 郑应亨，吴德夫．探悉项目投资有效控制的途径．重庆建筑大学学报，2002，24(3)：65～68
[54] 周勇．对公路基建中“三超”现象的浅析．公路，2002，7：47～48
[55] 尹贻林，王振强．加入WTO对我国工程造价管理的影响及对策研究．数量经济技术经济研究，2001，8：26～29
[56] 马俐．加入WTO对工程造价管理的影响及对策．概预算与工程造价，2001，5：118～119
[57] Nguyen Duy Long, Stephen Ogunlana, Truong Quang, Ka Chi Lain. Large construction projects in developing countries. International Journal of Project Management, 2004,22:553～561
[58] Sadi A. Assaf, Abdulaziz A. Bubshait, Sulaiman Atiyah, Mohammed Al-Shahri. The management of construction company overhead costs.International Journal of Project Management, 2001, 19:295～303
[59] 刘钟莹．工程估价．南京：东南大学出版社，2000
[60] 王祯显．土木建筑工程管理中地模糊数学方法．长沙：湖南大学出版社，1989
[2] Chan JCL, Jiu En Shi. Application of projection pursuit principal component analysis method to climate studies.International Journal of climatology, 1997, 17(1):103～113
[3] Fung T, Ledrew E. Application of principal component analysis to change detection.Photogrammetric Engineering and remote Sensing, 1987, 53(12): 1649～1658
[4] 盼庆仲．主成分分析及与TOPSIS法用于医院候诊室卫生评比的对比分析．数理医药学杂志，1999，2(12)
[5] 朱顺泉，徐国祥．我国上市公司财务状况的主成分分析研究．重庆工商大学学报，2003，(4)：48～50
[6] 张清贵．人工神经网络导论．北京：中国水利水电出版社，2004
[7] A.K.Jain, and J.Mao. Neural Networks and Patem recognition.In Computational Intelligence: Imitating Life.IEEE Press, Piscataway, NJ. 1994
[8] D.Psaltis, A .Sideris, A. A.Yamamura.A multilayed neural network controller. IEEE Control Systems.1998, 8(3): 17～21
[9] Friedhelm Schwenker, Hans A. Kestler, GuEnther Palm.Three learning phases for radial-basis-functionn etworks. NeuralN etworks, 2001(14): 439～458
[10] Simon Haykin. Neural networks: A Comprehensive Foundation.Upper Saddle River, NJ: Prentice Hall, 1999
[11] 张学工 译．统计学习理论的本质．北京：清华大学出版社，2000
[12] Vapnik V N, Chervonenkis A Ja. On the uniform convergence of relative frequencies of events to their probabilities. Doklady Akademii Nauk USSR, 1968
[13] Blumer A, Ehrenfeucht A, Haussler D etal. Learnability and the Vapnik-Chervonenkis dimension. Journal of the ACM. 1989, 36(4): 929～965
[14] Vapnik V N, Chervonenkis A Ja. Necessary and sufficient conditions for the uniform convergence of the means to their expectation.Probability Theory and Its Applications. 1981, 26:532～553
[15] Cortes C, Vapnik V N. Support vector networks.Machine Learning. 1995, 20: 273～297
[16] 周伟达，张莉，焦李成．支撑矢量机推广能力分析．电子学报，2001，29(5)：590～594
[17] SchelkopfB. Support Vector Learning. R Oldenbourg Verlag, Munich, 1997
[18] Burges CJC. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 1998, 2(2): 121～167
[19] 马云潜，张学工．支持向量机函数拟合在分形插值中的应用．清华大学学报(自然科学版)，2000，40(3)：76～78，103
[20] Weston J, Gammerman A, Stitson M O etal. Density estimation using support vector machines. Technical report, Royal Holloway College, Report number CSD-TR-97-23, 1997
[21] 高学，金连文，尹俊勋等．一种基于支持向量机的手写汉字识别方法．电子学报．2002，30(5)：651～654
[22] Maragoudakis M, Kavallieratou E, Fakotakis N. Incorporating conditional independence assumption with support vector machines to enhance handwritten character segmentation performance.In Proceedings. 16th International Conference on Pattern Recognition, 2002, 3: 11～15
[23] Maragoudakis M, Kavallieratou E, Fakotakis N. Improving handwritten character segmentation by incorporating Bayesian knowledge with support vector machines.In Proceedings. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2002, 4: 13～17
[24] Li J M, Zhang B, Lin F Z. Nonlinear speech model based on support vector machine and wavelet transform.In Proceedings. 15th IEEE International Conference on Tools with Artificial Intelligence, 2003: 3～5
[25] Gordan M, Kotropoulos C, Pitas I. Application of support vector machines classifiers to visual speech recognition.In Proceedings. 2002 International Conference on Image Processing, 2002, 3:24～28
[26] Chakrabartty S, Singh G, Cauwenberghs G.Hybrid support vector machine/hidden Markov model approach for continuous speech recognition.In Proceedings. The 43rd IEEE Midwest Symposium on Circuits and Systems, 2000, 2:8～11
[27] 祝海龙，屈梁生，张海军．基于小波变换和支持向量机的人脸检测系统．西安交通大学学报，2002，36(9)：947～950
[28] Tefas A, Kotropoulos C, Pitas I. Using support vector machines to enhance the performance of elastic graph matching for fi'ontal face authentication. IEEE Transactions on Pattern Analysis and Machine Intelligence.2001, 23(7): 735～746
[29] Joachims T. Text categorization with support vector machines. Technical report, LS Ⅷ Number 23 , University of Dortmund, 1997
[30] Muller K R, Smola A, R_tsch G etal. Predicting time series with support vector machines.In Proceedings, International Conference on Artificial Neural Networks, 1997: 999～105
[31] Mukherjee S, Osuna E, Girosi E Nonlinear prediction of chaotic time series using a support vector machine. In Proceeding of the IEEE Workshop on Neural Networks for Signal Processing 7, 1997: 511～519
[32] Powell M J D. Radial Basis Function for multivariable interpolation, Review IMA Conf. on Algorithms for the Approximation of Functions and Data. RMCS Shrivanham, 1985
[33] D. S. Broomhead, D . Lowe. Multivarialble function interpolation and adaptive network. Complex Systems, 1988(2)
[34] J.Moody, C.Darken. Fast L earning in Networks of L ocally-tuned Processing units.Neural Computation, 1989(1): 281～294
[35] Jackon I R H. Convergence properties of radial basis function. Constr Approx, 1988(4): 243～264
[36] J.Park, Etal. Universal Approximations Using RBF Networks. Neural Computation, 1991(3): 246～257
[37] Powell M J D. The Theory of RBF Approximation in 1990, Numerical Analysis Report.University of Cambrige, 1990
[38] D.R.Hush, and B.G. Hone.Progress in supervised neural networks. IEEE signal Processing Magazine, 1993, 1(1): 8～37
[39] 王旭东,邵惠鹤．RBF神经网络理论及其在控制中的应用．信息与控制,1997,26(4)：272～284
[40] S.Chen, C.F.N. Cowan and EM.Grant.Orthogonal Least Squares Learning Algorithm for Radial Basis Function Networks. IEEE Trans, Neural Networks, 1991, 2(2): 302～309
[41] Q.M.Zhu and S.A.Billings. Fast orthogonal identification of nonlinear Stochastic models and radial basis function neural networks. Int. J. Contr, 1996, 64(5): 871～886
[42] Kaufman L. Solving the qp problem for support vector training. In Proceedings of the 1997 NIPS Workshop on Support Vector Machines.1998: 237～246
[43] Ancona N, Cicirelli G, Distante A. Complexity reduction and parameter selection in support vector machines. IJCNN'02, Proceedings of the 2002 International Joint Conference on Neural Networks, 2002, 3: 2375～2380
[44] Jordaan E M, Smits G F. Estimation of the regularization parameter for support vector regression.IJCNN'02, Proceedings of the 2002 International Joint Conference on Neural Networks, 2002, 3:2192～2197
[45] Ito K, Nakano R. Optimizing support vector regression hyperparameters based on cross-validation. 2003 Proceedings of the International Joint Conference on Neural Networks, 2003, 3:2077～2082
[46] Wahba G, Lin Y, Zhang H. Margin-like quantities and generalized approximate cross validation for support vector machines. Proceedings of the 1999 IEEE Signal Processing Society Workshop on Neural Networks for Signal Processing Ⅸ, 1999: 12～20
[47] Hotelling, H. Analysis of a Complex of Statistical Variables and Principal Components. Journal of Educational Psychology, 1933(24): 417～441, 498～520
[48] 飞思科技产品研发中心．神经网络理论与MATLAB7实现．北京：电子工业出版社，2005
[49] 李柞泳，邓新民．BP网络的过拟合现象满足的测不准关系式．红外与毫米波学报，2000，19(2)：142～144
[50] V.Vapnik.An overview of statistical learning theory.IEEE Trans Neural Networks, 1999, 10(5): 998～999
[51] 李映，焦李成．统一框架下软计算方法和支持向量机的机理研究．模式识别与人工智能，2002，15(2)：173～178
[52] 张学工．关于统计学习理论与支持向量机．自动化学报，2000，16(1)：33～43
[53] 郑应亨，吴德夫．探悉项目投资有效控制的途径．重庆建筑大学学报，2002，24(3)：65～68
[54] 周勇．对公路基建中“三超”现象的浅析．公路，2002，7：47～48
[55] 尹贻林，王振强．加入WTO对我国工程造价管理的影响及对策研究．数量经济技术经济研究，2001，8：26～29
[56] 马俐．加入WTO对工程造价管理的影响及对策．概预算与工程造价，2001，5：118～119
[57] Nguyen Duy Long, Stephen Ogunlana, Truong Quang, Ka Chi Lain. Large construction projects in developing countries. International Journal of Project Management, 2004,22:553～561
[58] Sadi A. Assaf, Abdulaziz A. Bubshait, Sulaiman Atiyah, Mohammed Al-Shahri. The management of construction company overhead costs.International Journal of Project Management, 2001, 19:295～303
[59] 刘钟莹．工程估价．南京：东南大学出版社，2000
[60] 王祯显．土木建筑工程管理中地模糊数学方法．长沙：湖南大学出版社，1989
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |