Relief-based feature selection: Introduction and review

Agre, 2016, A weighted feature selection method for instance-based classification, 14 Aha, 1991, Instance-based learning algorithms, Mach. Learn., 6, 37, 10.1007/BF00153759 H. Almuallim, T.G. Dietterich, Learning with many irrelevant features, in: AAAI, vol. 91, 1991, pp. 547–552. A. Arauzo-Azofra, J.M. Benitez, J.L. Castro, A feature set measure based on relief, in: Proceedings of the Fifth International Conference on Recent Advances in Soft Computing, 2004, pp. 104–109. L.A. Belanche, F.F. González, Review and evaluation of feature selection algorithms in synthetic problems, 2011. Available from: arXiv preprint <arXiv:1101.2320>. Beretta, 2011, Implementing relieff filters to extract meaningful features from genetic lifetime datasets, J. Biomed. Inform., 44, 361, 10.1016/j.jbi.2010.12.003 Bins, 2000, Feature selection of huge feature sets in the context of computer vision, vol. 156 Bins, 2001, Feature selection from huge feature sets, vol. 2, 159 J. Bins, B.A. Draper, Evaluating feature relevance: Reducing bias in relief, in: JCIS, 2002, pp. 757–760. Blessie, 2011, Relief-disc: an extended relief algorithm using discretization approach for continuous features, 161 Blum, 1997, Selection of relevant features and examples in machine learning, Artif. Intell., 97, 245, 10.1016/S0004-3702(97)00063-5 Bolón-Canedo, 2013, A review of feature selection methods on synthetic data, Knowl. Inform. Syst., 34, 483, 10.1007/s10115-012-0487-8 P.S. Bradley, O.L. Mangasarian, Feature selection via concave minimization and support vector machines, in: ICML, vol. 98, 1998, pp. 82–90. Breiman, 1984 Cai, 2012, Feature weighting by relief based on local hyperplane approximation, 335 J.P. Callan, T. Fawcett, E.L. Rissland, Cabot: an adaptive approach to case-based search, in: IJCAI, vol. 12, 1991, pp. 803–808. Challita, 2015, New technique for feature selection: combination between elastic net and relief, 262 Chandrashekar, 2014, A survey on feature selection methods, Comput. Electr. Eng., 40, 16, 10.1016/j.compeleceng.2013.11.024 Chang, 2010, Generalized iterative relief for supervised distance metric learning, Pattern Recogn., 43, 2971, 10.1016/j.patcog.2010.02.024 Chen, 2016, Xgboost: a scalable tree boosting system, 785 Chikhi, 2009, Reliefmss: a variation on a feature ranking relieff algorithm, Int. J. Bus. Intell. Data Min., 4, 375, 10.1504/IJBIDM.2009.029085 Cortizo, 2006, Multi Criteria Wrapper Improvements to Naive Bayes Learning, 419 Dash, 1997, Feature selection for classification, Intell. Data Anal., 1, 131, 10.3233/IDA-1997-1302 Dash, 2011, Relief-c: efficient feature selection for clustering over noisy data, 869 Dash, 2007, extrarelief: improving relief by efficient selection of instances, Lect. Notes Comput. Sci., 4830, 305, 10.1007/978-3-540-76928-6_32 De Mántaras, 1991, A distance-based attribute selection measure for decision tree induction, Mach. Learn., 6, 81, 10.1023/A:1022694001379 Demšar, 2010, Algorithms for subsetting attribute values with relief, Mach. Learn., 78, 421, 10.1007/s10994-009-5164-0 Demšar, 2013, Orange: data mining toolbox in python, J. Mach. Learn. Res., 14, 2349 Dessı̀, 2013, A comparative analysis of biomarker selection techniques, BioMed Res. Int., 2013 Dorani, 2018, Feature selection for detecting gene-gene interactions in genome-wide association studies, 33 Draper, 2003, Iterative relief, vol. 6, 62 Eiras-Franco, 2016, Multithreaded and spark parallelization of feature selection filters, J. Comput. Sci., 17, 609, 10.1016/j.jocs.2016.07.002 Eppstein, 2008, Very large scale relieff for genome-wide association analysis, 112 Eppstein, 2007, Genomic mining for complex disease traits with “random chemistry”, Genet. Program Evolvable Mach., 8, 395, 10.1007/s10710-007-9039-5 Flórez-López, 2002, Reviewing relief and its extensions: a new approach for estimating attributes considering high-correlated features, 605 Gore, 2016, Feature selection using cooperative game theory and relief algorithm, 401 Granizo-Mackenzie, 2013, Multiple threshold spatially uniform relieff for the genetic analysis of complex human diseases, 1 Greene, 2010, The informative extremes: using both nearest and farthest individuals can improve relief algorithms in the domain of human genetics, 182 Greene, 2009, Spatially uniform relieff (surf) for computationally-efficient filtering of gene-gene interactions, BioData Min., 2, 5, 10.1186/1756-0381-2-5 I. Guyon, H.-M. Bitter, Z. Ahmed, M. Brown, J. Heller, Multivariate non-linear feature selection with kernel multiplicative updates and gram-schmidt relief, in: BISC Flint-CIBI 2003 Workshop. Berkeley, 2003, pp. 1–11. Guyon, 2005, Multivariate non-linear feature selection with kernel methods, 313 Guyon, 2003, An introduction to variable and feature selection, J. Mach. Learn. Res., 3, 1157 Hall, 2009, The weka data mining software: an update, ACM SIGKDD Explor. Newslett., 11, 10, 10.1145/1656274.1656278 Holland, 1992 Hong, 1997, Use of contextual information for feature ranking and discretization, IEEE Trans. Knowl. Data Eng., 9, 718, 10.1109/69.634751 Hunt, 1966 Ihaka, 1996, R: a language for data analysis and graphics, J. Comput. Graph. Stat., 5, 299 Inza, 2000, Feature subset selection by bayesian network-based optimization, Artif. Intell., 123, 157, 10.1016/S0004-3702(00)00052-7 Jedrzejowicz, 2018, Imbalanced data classification using mapreduce and relief, J. Inform. Telecommun., 2, 217, 10.1080/24751839.2018.1440454 Jin, 2006, Machine learning techniques and chi-square feature selection for cancer classification using sage gene expression profiles, 106 Jović, 2015, A review of feature selection methods with applications, 1200 K. Kira, L.A. Rendell, The feature selection problem: traditional methods and a new algorithm, in: AAAI, vol. 2, 1992a, pp. 129–134. K. Kira, L.A. Rendell, A practical approach to feature selection, in: Proceedings of the Ninth International Workshop on Machine Learning, 1992b, pp. 249–256. Kittler, 1978, Feature set search algorithms, Pattern Recogn. Signal Process., 10.1007/978-94-009-9941-1_3 Kohavi, 1997, Wrappers for feature subset selection, Artif. Intell., 97, 273, 10.1016/S0004-3702(97)00043-X Koller, 1996 Kong, 2012, Multi-label relieff and f-statistic feature selections for image annotation, 2352 Kononenko, 1994, Estimating attributes: analysis and extensions of relief, 171 I. Kononenko, On biases in estimating multi-valued attributes, in: Ijcai, vol. 95, 1995, pp. 1034–1040. Kononenko, 1996, Relieff for estimation and discretization of attributes in classification, regression, and ilp problems, Artif. Intell.: Methodol., Syst., Appl., 31 Kononenko, 2008, Non-myopic feature quality evaluation with (r) relieff, Comput. Methods Feature Select., 169 Kononenko, 1997, Overcoming the myopia of inductive learning algorithms with relieff, Appl. Intell., 7, 39, 10.1023/A:1008280620621 P. Langley, Selection of relevant features in machine learning, in: Proceedings of the AAAI Fall Symposium on Relevance, vol. 184, 1994, pp. 245–271. Le, 2017, Differential privacy-based evaporative cooling feature selection and classification with relief-f and random forests, Bioinformatics, 33, 2906, 10.1093/bioinformatics/btx298 K.-Y. Lee, P. Liu, K.-S. Leung, M.-H. Wong, Very large scale relieff algorithm on gpu for genome-wide association study, in: Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA). The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp), 2015, p. 78. Li, 2014 Liu, 2015, Feature selection of medical data sets based on rs-relieff, 1 Martínez, 2001, Pca versus lda, IEEE Trans. Pattern Anal. Mach. Intell., 23, 228, 10.1109/34.908974 McKinney, 2009, Capturing the spectrum of interaction effects in genetic association studies by simulated evaporative cooling network analysis, PLoS Genet., 5, e1000432, 10.1371/journal.pgen.1000432 McKinney, 2007, Evaporative cooling feature selection for genotypic data involving interactions, Bioinformatics, 23, 2113, 10.1093/bioinformatics/btm317 McKinney, 2013, Reliefseq: a gene-wise adaptive-k nearest-neighbor feature selection tool for finding gene-gene interactions and main effects in mrna-seq gene expression data, PloS One, 8, e81527, 10.1371/journal.pone.0081527 Menze, 2009, A comparison of random forest and its gini importance with standard chemometric methods for the feature selection and classification of spectral data, BMC Bioinform., 10, 213, 10.1186/1471-2105-10-213 Mhamdi, 2013, A new algorithm relief hybrid (hrelief) for biological motifs selection, 1 Michalski, 1983, A theory and methodology of inductive learning, Artif. Intell., 20, 111, 10.1016/0004-3702(83)90016-4 Mlambo, 2016, A survey and comparative study of filter and wrapper feature selection techniques, Int. J. Eng. Sci. (IJES), 5, 57 Moore, 2015 Moore, 2007, Tuning relieff for genome-wide genetic analysis, 166 Narendra, 1977, A branch and bound algorithm for feature subset selection, IEEE Trans. Comput., 9, 917, 10.1109/TC.1977.1674939 Ni, 2012 R.S. Olson, W. La Cava, P. Orzechowski, R.J. Urbanowicz, J.H. Moore, Pmlb: a large benchmark suite for machine learning evaluation and comparison, 2017. Available from: arXiv preprint <arXiv:1703.00512>. Park, 2007, Extended relief algorithms in instance-based feature filtering, 123 Pedregosa, 2011, Scikit-learn: machine learning in Python, J. Mach. Learn. Res., 12, 2825 Pupo, 2013, Relieff-ml: an extension of relieff algorithm to multi-label learning, 528 Qamar, 2012, Relief algorithm and similarity learning for k-nn, Int. J. Comput. Inform. Syst. Ind. Manage. Appl. (IJCISIM), 4, 445 Quinlan, 1993, vol. 38 Reyes, 2015, Scalable extensions of the relieff algorithm for weighting and selecting features on the multi-label learning context, Neurocomputing, 161, 168, 10.1016/j.neucom.2015.02.045 Ritchie, 2001, Multifactor-dimensionality reduction reveals high-order interactions among estrogen-metabolism genes in sporadic breast cancer, Am. J. Hum. Genet., 69, 138, 10.1086/321276 Robnik-Šikonja, 2003, Experiments with cost-sensitive feature evaluation, 325 M. Robnik-Šikonja, I. Kononenko, An adaptation of relief for attribute estimation in regression, in: Machine Learning: Proceedings of the Fourteenth International Conference (ICML’97), 1997, pp. 296–304. Robnik-Šikonja, 2001, Comprehensible interpretation of relief’s estimates, 433 Robnik-Šikonja, 2003, Theoretical and empirical analysis of relieff and rrelieff, Mach. Learn., 53, 23, 10.1023/A:1025667309714 Robnik-Sikonja, 2012 Rokach, 2010, Ensemble-based classifiers, Artif. Intell. Rev., 33, 1, 10.1007/s10462-009-9124-7 Saeys, 2008, Robust feature selection using ensemble feature selection techniques, Mach. Learn. Knowl. Discov. Databases, 313, 10.1007/978-3-540-87481-2_21 Saeys, 2007, A review of feature selection techniques in bioinformatics, Bioinformatics, 23, 2507, 10.1093/bioinformatics/btm344 Slavkov, 2013, Extending relieff for hierarchical multi-label classification, Mach. Learn., 4, 13 Smyth, 1992, An information theoretic approach to rule induction from databases, IEEE Trans. Knowl. Data Eng., 4, 301, 10.1109/69.149926 Song, 2013, A fast clustering-based feature subset selection algorithm for high-dimensional data, IEEE Trans. Knowl. Data Eng., 25, 1, 10.1109/TKDE.2011.181 Spolaôr, 2012, Filter approach feature selection methods to support multi-label learning based on relieff and information gain, 72 Spolaôr, 2013, Relieff for multi-label feature selection, 6 Stokes, 2012, Application of a spatially-weighted relief algorithm for ranking genetic predictors of disease, BioData Min., 5, 20, 10.1186/1756-0381-5-20 Sun, 2007, Iterative relief for feature weighting: algorithms, theories, and applications, IEEE Trans. Pattern Anal. Mach. Intell., 29, 10.1109/TPAMI.2007.1093 Sun, 2006, Iterative relief for feature weighting, 913 Sun, 2010, Local-learning-based feature selection for high-dimensional data analysis, IEEE Trans. Pattern Anal. Mach. Intell., 32, 1610, 10.1109/TPAMI.2009.190 Sun, 2008, A relief based feature extraction algorithm, 188 R.S. Sutton, C.J. Matheus, Learning polynomial functions by feature construction, in: ML, 1991, pp. 208–212. Tang, 2014, Feature selection for classification: a review, Data Classif.: Algor. Appl., 37 Tibshirani, 1996, Regression shrinkage and selection via the lasso, J. Roy. Stat. Soc. Ser. B (Methodological), 267 Todorov, 2016, 95 Urbanowicz, 2014, An extended michigan-style learning classifier system for flexible supervised learning, classification, and data mining, 211 Urbanowicz, 2012, Using expert knowledge to guide covering and mutation in a michigan style learning classifier system to detect epistasis and heterogeneity, 266 Urbanowicz, 2015, Exstracs 2.0: description and evaluation of a scalable learning classifier system, Evol. Intell., 8, 89, 10.1007/s12065-015-0128-8 Urbanowicz, 2018, Benchmarking relief-based feature selection methods for bioinformatics data mining, J. Biomed. Inform., 10.1016/j.jbi.2018.07.015 Van Laarhoven, 1987, Simulated annealing, 7 Verma, 2018, Collective feature selection to identify crucial epistatic variants, BioData Min., 11, 5, 10.1186/s13040-018-0168-6 Wolpert, 1997, No free lunch theorems for optimization, IEEE Trans. Evol. Comput., 1, 67, 10.1109/4235.585893 Yang, 2006, Orthogonal relief algorithm for feature selection, 227 Ye, 2008, Multi-relief: a method to recognize specificity determining residues from multiple sequence alignments using a machine-learning approach for feature weighting, Bioinformatics, 24, 18, 10.1093/bioinformatics/btm537 Yu, 2004, Efficient feature selection via analysis of relevance and redundancy, J. Mach. Learn. Res., 5, 1205 Zafra, 2010, Feature selection is the relieff for multiple instance learning, 525 Zafra, 2012, Relieff-mi: an extension of relieff to multiple instance learning, Neurocomputing, 75, 210, 10.1016/j.neucom.2011.03.052 Zeng, 2013, Feature selection based on relieff and pca for underwater sound classification, 442 Zhao, 2009, Searching for interacting features in subset selection, Intell. Data Anal., 13, 207, 10.3233/IDA-2009-0364 Zhou, 2014, Stable feature selection with ensembles of multi-relieff, 742 Zou, 2005, Regularization and variable selection via the elastic net, J. Roy. Stat. Soc.: Ser. B (Statistical Methodology), 67, 301, 10.1111/j.1467-9868.2005.00503.x