JZUS - Journal of Zhejiang University SCIENCE

Frontiers of Information Technology & Electronic Engineering 2019 Vol.20 No.4 P.538-553

Unsupervised feature selection via joint local learning and group sparse regression

Author(s): Yue Wu, Can Wang, Yue-qing Zhang, Jia-jun Bu
Affiliation(s): Zhejiang Provincial Key Laboratory of Service Robot, College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China; more
Corresponding email(s): wy1988@zju.edu.cn, wcan@zju.edu.cn, 704787221@qq.com, bjj@zju.edu.cn
Key Words: Unsupervised, Local learning, Group sparse regression, Feature selection

Share this article to： More <<< Previous Article \|Next Article >>>

Abstract: feature selection has attracted a great deal of interest over the past decades. By selecting meaningful feature subsets, the performance of learning algorithms can be effectively improved. Because label information is expensive to obtain, unsupervised feature selection methods are more widely used than the supervised ones. The key to unsupervised feature selection is to find features that effectively reflect the underlying data distribution. However, due to the inevitable redundancies and noise in a dataset, the intrinsic data distribution is not best revealed when using all features. To address this issue, we propose a novel unsupervised feature selection algorithm via joint local learning and group sparse regression (JLLGSR). JLLGSR incorporates local learning based clustering with group sparsity regularized regression in a single formulation, and seeks features that respect both the manifold structure and group sparse structure in the data space. An iterative optimization method is developed in which the weights finally converge on the important features and the selected features are able to improve the clustering results. Experiments on multiple real-world datasets (images, voices, and web pages) demonstrate the effectiveness of JLLGSR.

联合局部学习和组稀疏回归的无监督特征选择

摘要：近十年，特征选择备受关注。通过挑选特征子集，可有效提升学习算法效率。由于难以获取标签信息，无监督特征选择算法相较于有监督特征选择算法应用更为广泛，其关键在于找出更能反映数据分布的特征集合。由于数据集中存在冗余和噪声，使用全部特征并不能很好展现数据的真实分布。为解决这一问题，本文提出联合局部学习和组稀疏回归的无监督特征选择算法。将基于局部学习聚类方法与组稀疏回归算法有机整合，选出有效反映数据流形分布同时保持组稀疏结构的特征。通过迭代算法，回归系数汇聚到重要特征上，选出能得到更优聚类效果的特征。对多个实际数据集（图像、声音和网页）的实验证明了该算法的有效性。

关键词：无监督；局部学习；组稀疏回归；特征选择

Reference

[1]Belkin M, Niyogi P, 2001. Laplacian eigenmaps and spectral techniques for embedding and clustering. 14^th Int Conf on Neural Information Processing Systems: Natural and Synthetic, p.585-591.

[2]Bellman RE, 1961. Adaptive Control Processes: a Guided Tour. Princeton University Press, Princeton, NJ.

[3]Cai D, Zhang C, He X, 2010. Unsupervised feature selection for multi-cluster data. 16^th Int Conf on Knowledge Discovery and Data Mining, p.333-342.

[4]Chang XJ, Nie FP, Yang Y, et al., 2016. Convex sparse PCA for unsupervised feature learning. ACM Trans Knowl Dis Data, 11(1):3.

[5]Cheung Y, Zeng H, 2009. Local kernel regression score for selecting features of high-dimensional data. IEEE Trans Knowl Data Eng, 21(12):1798-1802.

[6]Doquire G, Verleysen M, 2013. Mutual information-based feature selection for multilabel classification. Neurocomputing, 122:148-155.

[7]Du L, Shen YD, 2015. Unsupervised feature selection with adaptive structure learning. 21^st Int Conf on Knowledge Discovery and Data Mining, p.209-218.

[8]Fanty M, Cole R, 1990. Spoken letter recognition. Conf on Advances in Neural Information Processing Systems, p.220-226.

[9]Georghiades AS, Belhumeur PN, Kriegman DJ, 2001. From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans Patt Anal Mach Intell, 23(6):643-660.

[10]Guyon I, Elisseeff A, 2003. An introduction to variable and feature selection. J Mach Learn Res, 3:1157-1182.

[11]Guyon I, Weston J, Barnhill S, et al., 2002. Gene selection for cancer classification using support vector machines. Mach Learn, 46(1-3):389-422.

[12]Han YH, Wu F, Tian Q, et al., 2012. Image annotation by input-output structural grouping sparsity. IEEE Trans Image Proc, 21(6):3066-3079.

[13]Han YH, Yang Y, Yan Y, et al., 2015. Semisupervised feature selection via spline regression for video semantic recognition. IEEE Trans Neur Netw Learn Syst, 26(2):252- 264.

[14]He X, Niyogi P, 2004. Locality preserving projections. Conf on Advances in Neural Information Processing Systems, p.153-160.

[15]He X, Cai D, Niyogi P, 2005. Laplacian score for feature selection. Conf on Advances in Neural Information Processing Systems, p.507-514.

[16]Hou CP, Nie FP, Li XL, et al., 2014. Joint embedding learning and sparse regression: a framework for unsupervised feature selection. IEEE Trans Cybern, 44(6):793-804.

[17]Hull JJ, 1994. A database for handwritten text recognition research. IEEE Trans Patt Anal Mach Intell, 16(5):550-554.

[18]Jiang Y, Ren JT, 2011. Eigenvalue sensitive feature selection. 28^th Int Conf on Machine Learning, p.89-96.

[19]Jolliffe IT, 2002. Principal Component Analysis (2^nd Ed.). Springer, New York.

[20]Krizhevsky A, 2009. Learning Multiple Layers of Features from Tiny Images. Science Department, University of Toronto, Tech, Toronto.

[21]Kuhn HW, 1955. The Hungarian method for the assignment problem. Nav Res Log Q, 2(1-2):83-97.

[22]Lee KC, Ho J, Kriegman DJ, 2005. Acquiring linear subspaces for face recognition under variable lighting. IEEE Trans Patt Anal Mach Intell, 27(5):684-698.

[23]Luo MN, Nie FP, Chang XJ, et al., 2018. Adaptive unsupervised feature selection with structure regularization. IEEE Trans Neur Netw Learn Syst, 29(4):944-956.

[24]Munkres J, 1957. Algorithms for the assignment and transportation problems. J Soc Ind Appl Math, 5(1):32-38.

[25]Nie FP, Xiang SM, Jia YQ, et al., 2008. Trace ratio criterion for feature selection. 23^rd Int Conf on Artificial Intelligence, p.671-676.

[26]Nie FP, Xiang SM, Song YQ, et al., 2009. Orthogonal locality minimizing globality maximizing projections for feature extraction. Opt Eng, 48(1):017202.

[27]Nie FP, Huang H, Cai X, et al., 2010a. Efficient and robust feature selection via joint l>_2,1-norms minimization. 23^rd Int Conf on Neural Information Processing Systems, p.1813-1821.

[28]Nie FP, Xu D, Tsang IWH, et al., 2010b. Flexible manifold embedding: a framework for semi-supervised and unsupervised dimension reduction. IEEE Trans Image Proc, 19(7):1921-1932.

[29]Nie FP, Zeng ZN, Tsang IW, et al., 2011. Spectral embedded clustering: a framework for in-sample and out-of-sample spectral clustering. IEEE Trans Neur Netw, 22(11):1796-1808.

[30]Nie FP, Wang XQ, Jordan MI, et al., 2016a. The constrained Laplacian rank algorithm for graph-based clustering. 30^th AAAI Conf on Artificial Intelligence, p.1969-1976.

[31]Nie FP, Zhu W, Li XI, 2016b. Unsupervised feature selection with structured graph optimization. 30^th AAAI Conf on Artificial Intelligence, p.1302-1308.

[32]Peng HC, Long FH, Ding C, 2005. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Patt Anal Mach Intell, 27(8):1226-1238.

[33]Roweis ST, Saul LK, 2000. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500):2323-2326.

[34]Sun YJ, Todorovic S, Goodison S, 2010. Local-learning-based feature selection for high-dimensional data analysis. IEEE Trans Patt Anal Mach Intell, 32(9):1610-1626.

[35]Tan MK, Wang L, Tsang IW, 2010. Learning sparse SVM for feature selection on very high dimensional datasets. 27^th Int Conf on Machine Learning, p.1047-1054.

[36]Tenenbaum JB, de Silva V, Langford JC, 2000. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500):2319-2323.

[37]Tibshirani R, 1996. Regression shrinkage and selection via the Lasso. J R Stat Soc B, 58(1):267-288.

[38]Verleysen M, 2003. Learning high-dimensional data. In: Ablameyko S, Goras L, Gori M (Eds.), Limitations and Future Trends in Neural Computation. IOS Press, Amsterdam, p.141-162.

[39]Wang D, Nie FP, Huang H, 2014. Unsupervised feature selection via unified trace ratio formulation and K-means clustering (TRACK). European Conf on Machine Learning and Knowledge Discovery in Databases, p.306-321.

[40]Wu Y, Wang C, Bu JJ, et al., 2016. Group sparse feature selection on local learning based clustering. Neurocomputing, 171:1118-1130.

[41]Yang Y, Shen HT, Ma ZG, et al., 2011. $l_2,1$-norm regularized discriminative feature selection for unsupervised learning. 22^nd Int Joint Conf on Artificial Intelligence, p.1589-1594.

[42]Zeng H, Cheung YM, 2009. Feature selection for local learning based clustering. 13^th Pacific-Asia Conf on Advances in Knowledge Discovery and Data Mining, p.414-425.

[43]Zeng H, Cheung YM, 2011. Feature selection and kernel learning for local learning-based clustering. IEEE Trans Patt Anal Mach Intell, 33(8):1532-1547.

[44]Zhao Z, Liu H, 2007. Spectral feature selection for supervised and unsupervised learning. 24^th Int Conf on Machine Learning, p.1151-1157.

[45]Zou H, Hastie T, 2005. Regularization and variable selection via the elastic net. J R Stat Soc Ser B, 67(2):301-320.

Open peer comments: Debate/Discuss/Question/Opinion

<1>

Similar articles

- Go to

联合局部学习和组稀疏回归的无监督特征选择

Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article

Reference