Feature Selection Algorithm Based on LightGBM

doi:10.12068/j.issn.1005-3026.2021.12.003

Abstract

Abstract: In order to solve the shortcomings of the following two types of feature selection algorithms， filtering and wrapping based on evolutionary learning， a new wrapping feature selection algorithm LGBFS(LightGBM feature selection) was proposed. First， LightGBM was introduced to construct an gradient boosting tree model for the original features and measure the importance of features; then the proposed LR sequential forward search strategy LRSFFS was combined to select features; finally， the proposed algorithm was compared with nine algorithms in 21 standard datasets. The results show that 16 of the 21 standard data sets of LGBFS have achieved the best classification accuracy， and 18 standard data sets have achieved the best dimensionality reduction rate and the best CPU running time. In addition， time complexity analysis and significance test were carried out. The test shows that LGBFS is significantly different from the six comparison algorithms， and it also shows that LGBFS can balance the calculation efficiency and classification accuracy of feature subsets.

Key words: feature selection; LightGBM; boosting tree; wrapped method; sequential search

CLC Number:

TP18

LI Zhan-shan， YAO Xin， LIU Zhao-geng， ZHANG Jia-chen. Feature Selection Algorithm Based on LightGBM[J]. Journal of Northeastern University(Natural Science), 2021, 42(12): 1688-1695.

References

[1]Guyon I，Elisseeff A.An introduction to variable and feature selection［J］.Journal of Machine Learning Research，2003，3:1157-1182.
[2]Liu H，Motoda H，Setiono R，et al.Feature selection:an ever evolving frontier in data mining［J］.Journal of Machine Learning Research，2010，10:4-13.
[3]Kohavi R，John G H.Wrappers for feature subset selection［J］.Artificial Intelligence，1997，97(1/2):273-324.
[4]Dash M，Liu H.Feature selection for classification［J］.Intelligent Data Analysis，1997，1(3):131-156.
[5]Ke G L，Meng Q，Finley T，et al.LightGBM:a highly efficient gradient boosting decision tree［C］//Conference on Neural Information Processing Systems.Long Beach，2017:569-577.
[6]Guyon I，Elisseeff A.An introduction to variable and feature selection［J］.Journal of Machine Learning Research，2003，3(6):1157-1182.
[7]Hall M A.Feature selection for discrete and numeric class machine learning［C］// Proceedings of the Seventeenth International Conference on Machine Learning.Stanford:Morgan Kaufmann，2000:359-366.
[8]Guyon I，Elisseeff A.An introduction to variable and feature selection［J］.Journal of Machine Learning Research，2003，3(6):1157-1182.
[9]Alweshah M，Khalaileh S A，Gupta B B，et al.The monarch butterfly optimization algorithm for solving feature selection problems［EB/OL］.［2021-04-05］.https://doi.org/10.1007/s00521-020-05210-0.
[10]Taradeh M，Mafarja M，Heidari A，et al.An evolutionary gravitational search-based feature selection［J］.Information Sciences，2019，49(7):219-239.
[11]Arora S，Anand P.Binary butterfly optimization approaches for feature selection［J］.Expert Systems with Applications，2019，116:147-160.
[12]Zhang Y，Song X，Gong D.A return-cost-based binary firefly algorithm for feature selection［J］.Information Sciences， 2017，418/419:561-574.
[13]Emary E，Zawbaa H M，Hassanien A E.Binary grey wolf optimization approaches for feature selection［J］.Neurocomputing，2016，172:371-381.
[14]Emary E，Zawbaa H M，Hassanien A E.Binary ant lion approaches for feature selection［J］.Neurocomputing，2016，213:54-65.
[15]Xue B，Zhang M，Browne W N，et al.A survey on evolutionary computation approaches to feature selection［J］，IEEE Transactions on Evolutionary Computation，2016，20(4):606-626.
[16]Bennasar M，Hicks Y，Setchi R.Feature selection using joint mutual information maximisation［J］.Expert Systems with Applications，2015，42(22):8520-8532.
[17]Wilcoxon F.Individual comparisons by ranking methods［J］.Biometrics Bulletin，1945，1(6):80-83.