方差正则化的分类模型选择准则
|
摘要
在传统的机器学习中,模型选择常常是直接基于某个性能度量指标的估计本身进行,没有考虑估计的方差,但是这样的忽略极有可能导致错误模型的选择。于是考虑在分类模型选择研究中添加方差的信息的方法,以提高所选模型的泛化能力,即将泛化误差性能度量指标的组块3×2交叉验证估计的方差估计作为正则化项添加到传统模型选择准则中,提出了一种新的方差正则化的分类模型选择准则。模拟和真实数据实验验证了在分类模型选择问题中,提出的模型选择准则相比传统方法选到正确分类模型的概率更大,验证了方差在模型选择中的重要性以及提出的模型选择准则的有效性。进一步,理论上证明了在二分类问题的模型选择中,该模型选择准则具有选择的一致性。
In traditional machine learning, model selection is always directly performed based on the estimation of one performance measure index, without considering the variance of the estimation. However, this neglection may probably lead to the selection of a wrong model. Therefore, a method of adding the information of variance into the study of classification model selection is considered in order to improve the generalization ability of the selected model, that is, the variance estimation of the block 3×2 cross-validation estimation of the generalization error is added as a regularization term into the traditional model selection criterion, and a new variance-regularized classification model selection criterion is proposed. The simulated and real data experiments show that the proposed model selection criterion has a higher probability to select the correct classification model in the classification model selection problem compared to the traditional methods. The importance of variance in model selection and the effectiveness of the proposed model selection criteria are also validated. Furthermore, the consistency in selection of the proposed criterions is theoretically proven in the model selection task of two-class classification problem.
In traditional machine learning, model selection is always directly performed based on the estimation of one performance measure index, without considering the variance of the estimation. However, this neglection may probably lead to the selection of a wrong model. Therefore, a method of adding the information of variance into the study of classification model selection is considered in order to improve the generalization ability of the selected model, that is, the variance estimation of the block 3×2 cross-validation estimation of the generalization error is added as a regularization term into the traditional model selection criterion, and a new variance-regularized classification model selection criterion is proposed. The simulated and real data experiments show that the proposed model selection criterion has a higher probability to select the correct classification model in the classification model selection problem compared to the traditional methods. The importance of variance in model selection and the effectiveness of the proposed model selection criteria are also validated. Furthermore, the consistency in selection of the proposed criterions is theoretically proven in the model selection task of two-class classification problem.
引文
[1] Akaike H. Information theory and an extension of the maximum likelihood principle[C]//Proceedings of the 2nd International Symposium on Information Theory, Petrov,1973. Berlin, Heidelberg:Springer, 1973:267-281.
[2] Schwarz G. Estimating the dimension of a model[J]. The Annals of Statistics, 1978, 6(2):461-464.
[3] Tibshirani R. Regression shrinkage and selection via the LASSO[J]. Journal of the Royal Statistical Society, 1996, 58(1):267-288.
[4] Yuan M, Lin Y. Model selection and estimation in regression with grouped variables[J]. Journal of the Royal Statistical Society, 2006, 68(1):49-67.
[5] Zou H. The adaptive lasso and its oracle properties[J].Publications of the American Statistical Association, 2006,101(476):1418-1429.
[6] Fan J Q, Li R Z. Variable selection via nonconcave penalized likelihood and its oracle properties[J]. Journal of the American Statistical Association, 2001, 96(456):1348-1360.
[7] Hwang K, Lee K, Park S. Variable selection methods for multiclass classification using signomial function[J]. Journal of the Operational Research Society, 2017, 68(9):1-14.
[8] Wang Y, Wang R B, Jia H C, et al. Blocked 3×2 crossvalidated t-test for comparing supervised classification learning algorithms[J]. Neural Computation, 2014, 26(1):208-235.
[9] van Reenen M, Reinecke C J, Westerhuis J A, et al. Variable selection for binary classification using error rate p-values applied to metabolomics data[J]. BMC Bioinformatics,2016, 17(1):33.
[10] Lever J, Krzywinski M, Altman N. Points of significance:model selection and overfitting[J]. Nature Methods, 2016, 13(9):703-704.
[11] Shao J. Linear model selection by cross-validation[J].Journal of the American Statistical Association, 1993, 88(422):486-494.
[12] Stone M. Cross-validatory choice and assessment of statistical predictions[J]. Journal of the Royal Statistical Society, 1974,36(2):111-147.
[13] Bengio Y, Grandvalet Y. No unbiased estimator of the variance of k-fold cross-validation[J]. Journal of Machine Learning Research, 2004, 5:1089-1105.
[14] Markatou M, Tian H, Biswas S, et al. Analysis of variance of cross-validation estimators of the generalization error[J].Journal of Machine Learning Research, 2005, 6(1):1127-1168.
[15] Grandvalet Y, Bengio Y. Hypothesis testing for cross validation[D]. Montreal:University of Montreal, 2006.
[16] Rodríguez J D, Perez A P, Lozano J A. Sensitivity analysis of k-fold cross validation in prediction error estimation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(3):569-575.
[17] Nadeau C, Bengio Y. Inference for the generalization error[J]. Machine Learning, 2003, 52(3):239-281.
[18] Alpaydin E. Combined 5×2 cv F-test for comparing supervised classification learning algorithms[J]. Neural Computation, 1999, 11(8):1885-1892.
[19] Yang L, Wang Y. Survey for various cross-validation estimators of generalization error[J]. Application Research of Computers, 2015, 32(5):1287-1290.
[20] Li J H, Wang R B, Wang W L, et al. Automatic labeling of semantic roles on Chinese FrameN et[J]. Journal of Software,2010, 21(4):597-611.
[21] Li J H, Hu J Y, Wang Y. Blocked 3×2 cross-validated estimator of the generalization error—simulation comparative study based on biological data[J]. Journal of Biomathematics,2014, 29(4):700-710.
[22] Fortmannroe S. Understanding the bias-variance tradeoff[EB/OL].(2012-06). http://scott.fortmann-roe.com/docs/BiasVariance. html.
[23] Arlot S, Celisse A. A survey of cross-validation procedures for model selection[J]. Statistics Surveys, 2010, 4:40-79.
[24] Newman D J, Hettich S, Blake C L, et al. UCI repository of machine learning databases[EB/OL].(1998)[2018-04-11].http://www.ics.uci.edu/~mlearn/MLRepository.html.
[25] Liu Y Q. Research on model selection based on block 3×2cross-validation t-test[D]. Taiyuan:Shanxi University,2015.
[19]杨柳,王钰.泛化误差的各种交叉验证估计方法综述[J].计算机应用研究, 2015, 32(5):1287-1290.
[20]李济洪,王瑞波,王蔚林,等.汉语框架语义角色的自动标注[J].软件学报, 2010, 21(4):597-611.
[21]李济洪,胡军艳,王钰.预测误差的组块3×2交叉验证估计——基于生物数据的模拟比较研究[J].生物数学学报,2014, 29(4):700-710.
[25]刘焱青.基于组块3×2交叉验证t检验的模型选择研究[D].太原:山西大学, 2015.
[2] Schwarz G. Estimating the dimension of a model[J]. The Annals of Statistics, 1978, 6(2):461-464.
[3] Tibshirani R. Regression shrinkage and selection via the LASSO[J]. Journal of the Royal Statistical Society, 1996, 58(1):267-288.
[4] Yuan M, Lin Y. Model selection and estimation in regression with grouped variables[J]. Journal of the Royal Statistical Society, 2006, 68(1):49-67.
[5] Zou H. The adaptive lasso and its oracle properties[J].Publications of the American Statistical Association, 2006,101(476):1418-1429.
[6] Fan J Q, Li R Z. Variable selection via nonconcave penalized likelihood and its oracle properties[J]. Journal of the American Statistical Association, 2001, 96(456):1348-1360.
[7] Hwang K, Lee K, Park S. Variable selection methods for multiclass classification using signomial function[J]. Journal of the Operational Research Society, 2017, 68(9):1-14.
[8] Wang Y, Wang R B, Jia H C, et al. Blocked 3×2 crossvalidated t-test for comparing supervised classification learning algorithms[J]. Neural Computation, 2014, 26(1):208-235.
[9] van Reenen M, Reinecke C J, Westerhuis J A, et al. Variable selection for binary classification using error rate p-values applied to metabolomics data[J]. BMC Bioinformatics,2016, 17(1):33.
[10] Lever J, Krzywinski M, Altman N. Points of significance:model selection and overfitting[J]. Nature Methods, 2016, 13(9):703-704.
[11] Shao J. Linear model selection by cross-validation[J].Journal of the American Statistical Association, 1993, 88(422):486-494.
[12] Stone M. Cross-validatory choice and assessment of statistical predictions[J]. Journal of the Royal Statistical Society, 1974,36(2):111-147.
[13] Bengio Y, Grandvalet Y. No unbiased estimator of the variance of k-fold cross-validation[J]. Journal of Machine Learning Research, 2004, 5:1089-1105.
[14] Markatou M, Tian H, Biswas S, et al. Analysis of variance of cross-validation estimators of the generalization error[J].Journal of Machine Learning Research, 2005, 6(1):1127-1168.
[15] Grandvalet Y, Bengio Y. Hypothesis testing for cross validation[D]. Montreal:University of Montreal, 2006.
[16] Rodríguez J D, Perez A P, Lozano J A. Sensitivity analysis of k-fold cross validation in prediction error estimation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(3):569-575.
[17] Nadeau C, Bengio Y. Inference for the generalization error[J]. Machine Learning, 2003, 52(3):239-281.
[18] Alpaydin E. Combined 5×2 cv F-test for comparing supervised classification learning algorithms[J]. Neural Computation, 1999, 11(8):1885-1892.
[19] Yang L, Wang Y. Survey for various cross-validation estimators of generalization error[J]. Application Research of Computers, 2015, 32(5):1287-1290.
[20] Li J H, Wang R B, Wang W L, et al. Automatic labeling of semantic roles on Chinese FrameN et[J]. Journal of Software,2010, 21(4):597-611.
[21] Li J H, Hu J Y, Wang Y. Blocked 3×2 cross-validated estimator of the generalization error—simulation comparative study based on biological data[J]. Journal of Biomathematics,2014, 29(4):700-710.
[22] Fortmannroe S. Understanding the bias-variance tradeoff[EB/OL].(2012-06). http://scott.fortmann-roe.com/docs/BiasVariance. html.
[23] Arlot S, Celisse A. A survey of cross-validation procedures for model selection[J]. Statistics Surveys, 2010, 4:40-79.
[24] Newman D J, Hettich S, Blake C L, et al. UCI repository of machine learning databases[EB/OL].(1998)[2018-04-11].http://www.ics.uci.edu/~mlearn/MLRepository.html.
[25] Liu Y Q. Research on model selection based on block 3×2cross-validation t-test[D]. Taiyuan:Shanxi University,2015.
[19]杨柳,王钰.泛化误差的各种交叉验证估计方法综述[J].计算机应用研究, 2015, 32(5):1287-1290.
[20]李济洪,王瑞波,王蔚林,等.汉语框架语义角色的自动标注[J].软件学报, 2010, 21(4):597-611.
[21]李济洪,胡军艳,王钰.预测误差的组块3×2交叉验证估计——基于生物数据的模拟比较研究[J].生物数学学报,2014, 29(4):700-710.
[25]刘焱青.基于组块3×2交叉验证t检验的模型选择研究[D].太原:山西大学, 2015.
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |