Reducing the Dimensionality of Data with Neural Networks
Tóm tắt
High-dimensional data can be converted to low-dimensional codes by training a multilayer neural network with a small central layer to reconstruct high-dimensional input vectors. Gradient descent can be used for fine-tuning the weights in such “autoencoder” networks, but this works well only if the initial weights are close to a good solution. We describe an effective way of initializing the weights that allows deep autoencoder networks to learn low-dimensional codes that work much better than principal components analysis as a tool to reduce the dimensionality of data.
Từ khóa
Tài liệu tham khảo
D. DeMers, G. Cottrell, Advances in Neural Information Processing Systems 5 (Morgan Kaufmann, San Mateo, CA, 1993), pp. 580–587.
P. Smolensky, Parallel Distributed Processing: Volume 1: Foundations, D. E. Rumelhart, J. L. McClelland, Eds. (MIT Press, Cambridge, 1986), pp. 194–281.
G. E. Hinton, Neural Comput.14, 1711 (2002).
See supporting material on Science Online.
M. Welling, M. Rosen-Zvi, G. Hinton, Advances in Neural Information Processing Systems 17 (MIT Press, Cambridge, MA, 2005), pp. 1481–1488.
The MNIST data set is available at http://yann.lecun.com/exdb/mnist/index.html.
The Olivetti face data set is available at www.cs.toronto.edu/roweis/data.html.
The Reuter Corpus Volume 2 is available at http://trec.nist.gov/data/reuters/reuters.html.
We thank D. Rumelhart M. Welling S. Osindero and S. Roweis for helpful discussions and the Natural Sciences and Engineering Research Council of Canada for funding. G.E.H. is a fellow of the Canadian Institute for Advanced Research.