Variational inference for semiparametric Bayesian novelty detection in large datasets
Advances in Data Analysis and Classification - Trang 1-23 - 2023
Tóm tắt
After being trained on a fully-labeled training set, where the observations are grouped into a certain number of known classes, novelty detection methods aim to classify the instances of an unlabeled test set while allowing for the presence of previously unseen classes. These models are valuable in many areas, ranging from social network and food adulteration analyses to biology, where an evolving population may be present. In this paper, we focus on a two-stage Bayesian semiparametric novelty detector, also known as Brand, recently introduced in the literature. Leveraging on a model-based mixture representation, Brand allows clustering the test observations into known training terms or a single novelty term. Furthermore, the novelty term is modeled with a Dirichlet Process mixture model to flexibly capture any departure from the known patterns. Brand was originally estimated using MCMC schemes, which are prohibitively costly when applied to high-dimensional data. To scale up Brand applicability to large datasets, we propose to resort to a variational Bayes approach, providing an efficient algorithm for posterior approximation. We demonstrate a significant gain in efficiency and excellent classification performance with thorough simulation studies. Finally, to showcase its applicability, we perform a novelty detection analysis using the openly-available Statlog dataset, a large collection of satellite imaging spectra, to search for novel soil types.
Tài liệu tham khảo
Aliverti E, Russo M (2022) Stratified stochastic variational inference for high-dimensional network factor model. J Comput Graph Stat 31(2):502–511. https://doi.org/10.1080/10618600.2021.1984929, arXiv:2006.14217
Blei DM, Jordan MI (2006) Variational inference for Dirichlet process mixtures. Bayesian Anal 1(1):121–144. http://www.cs.berkeley.edu/$sim$blei/
Blei DM, Kucukelbir A, McAuliffe JD (2017) Variational inference: a review for statisticians. J Am Stat Assoc 112(518):859–877. https://doi.org/10.1080/01621459.2017.1285773
Boudt K, Rousseeuw PJ, Vanduffel S et al (2020) The minimum regularized covariance determinant estimator. Stat Comput 30(1):113–128. https://doi.org/10.1007/s11222-019-09869-x, arXiv:1701.07086
Bouveyron C (2014) Adaptive mixture discriminant analysis for supervised learning with unobserved classes. J Classif 31(1):49–84. https://doi.org/10.1007/s00357-014-9147-x. (link.springer.com/content/pdf/10.1007/s00357-014-9147-x.pdf)
Camilo M, Derek PT, Sina A et al (2011) How many species are there on earth and in the ocean? PLoS Biol 9(8):e1001,127
Cappozzo A, Greselin F, Murphy TB (2020) Anomaly and Novelty detection for robust semi-supervised learning. Stat Comput 30(5):1545–1571. https://doi.org/10.1007/s11222-020-09959-1. arxiv.org/abs/1911.08381 link.springer.com/10.1007/s11222-020-09959-1
Christenhusz MJ, Byng JW (2016) The number of known plants species in the world and its annual increase. Phytotaxa 261(3):201–217. https://doi.org/10.11646/phytotaxa.261.3.1
Denti F, Cappozzo A, Greselin F (2021) A two-stage Bayesian semiparametric model for novelty detection with robust prior information. Stat Comput 31(4):1–19
Escobar MD, West M (1995) Bayesian density estimation and inference using mixtures. J Am Stat Assoc 90(430):577–588. https://doi.org/10.1080/01621459.1995.10476550
Ferguson TS (1973) A Bayesian analysis of some nonparametric problems. Ann Stat 1(2):209–230. https://doi.org/10.1214/aos/1176342360, arXiv:arXiv:1011.1669v3
Finsinger W, Giesecke T, Brewer S et al (2017) Emergence patterns of novelty in European vegetation assemblages over the past 15000 years. Ecol Lett 20(3):336–346. https://doi.org/10.1111/ele.12731
Fop M, Mattei PA, Bouveyron C et al (2022) Unobserved classes and extra variables in high-dimensional discriminant analysis. Adv Data Anal Classif 16(1):55–92. https://doi.org/10.1007/s11634-021-00474-3, arXiv:2102.01982
Fowlkes EB, Mallows CL (1983) A method for comparing two hierarchical clusterings. J Am Stat Assoc 78(383):553–569. https://doi.org/10.1080/01621459.1983.10478008
Hinton G, van der Maaten L (2008) Visualizing Data using t-SNE. J Mach Learn Res 9:2579–2605. http://jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf%0Ahttp://www.jmlr.org/papers/v9/vandermaaten08a.html%5Cnfile:///Files/63/63E4B948-D809-4073-8CE0-E56194C96FD8.pdf
Hoffman M, Wang C, Paisley J (2003) Stochastic variational inference. J Mach Learn Res 1–52. arXiv:arXiv:1206.7051v1
Hubert L, Arabie P (1985) Comparing partitions. J Classif 2(1):193–218. https://doi.org/10.1007/BF01908075
Hubert M, Debruyne M, Rousseeuw PJ (2018) Minimum covariance determinant and extensions. Wiley Interdisciplinary Reviews: Comput Stat 10(3):1–11. https://doi.org/10.1002/wics.1421, arXiv:1709.07045
Jaakkola TS, Jordan MI (2000) Bayesian parameter estimation via variational methods. Stat Comput 10(1):25–37. https://doi.org/10.1023/A:1008932416310
Jordan MI, Ghahramani Z, Jaakkola TS et al (1999) Introduction to variational methods for graphical models. Mach Learn 37(2):183–233. https://doi.org/10.1023/A:1007665907178
Kalli M, Griffin JE, Walker SG (2011) Slice sampling mixture models. Stat Comput 21(1):93–105. https://doi.org/10.1007/s11222-009-9150-y
Markou M, Singh S (2003) Novelty detection: a review-part 1: statistical approaches. Signal Process 83(12):2481–2497. https://doi.org/10.1016/j.sigpro.2003.07.018. (linkinghub.elsevier.com/retrieve/pii/S0165168403002020)
Markou M, Singh S (2003b) Novelty detection: a review-part 2. Signal Process 83(12):2499–2521. https://doi.org/10.1016/j.sigpro.2003.07.019, https://linkinghub.elsevier.com/retrieve/pii/S0165168403002032
Nieman D, Szabo B, van Zanten H (2022) Contraction rates for sparse variational approximations in Gaussian process regression. J Mach Learn Res 23:1–26. arxiv:2109.10755
Ormerod JT, Wand MP (2010) Explaining variational approximations. Am Stat 64(2):140–153. https://doi.org/10.1198/tast.2010.09058
Ray K, Szabó B (2022) Variational Bayes for high-dimensional linear regression with sparse priors. J Am Stat Assoc 117(539):1270–1281. https://doi.org/10.1080/01621459.2020.1847121, arXiv:1904.07150
Rigon T (2023) An enriched mixture model for functional clustering. Appl Stoch Models Bus Ind 39(2):232–250
Rousseeuw PJ, Driessen KV (1999) A fast algorithm for the minimum covariance determinant estimator. Technometrics 41(3):212–223. https://doi.org/10.1080/00401706.1999.10485670
Sethuraman J (1994) A constructive definition of Dirichlet Process prior. Statistica Sinica 4(2):639–650. http://www.jstor.org/stable/24305538
Vatanen T, Kuusela M, Malmi E et al (2012) Semi-supervised detection of collective anomalies with an application in high energy particle physics. In: Proceedings of the international joint conference on neural networks, https://doi.org/10.1109/IJCNN.2012.6252712, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.223.215 &rep=rep1 &type=pdf
Vinh NX, Epps J, Bailey J (2009) Information theoretic measures for clusterings comparison: is a correction for chance necessary? ACM International Conference Proceeding Series 382:1–8. https://doi.org/10.1145/1553374.1553511
Wang B, Titterington D (2012) Convergence and asymptotic normality of variational Bayesian approximations for exponential family models with missing values. In: Chickering M, Halpern J (eds) Proceedings of the 20th conference in uncertainty in Artificial Intelligence. AUAI Press
Wang C (2012) Variational inference in nonconjugate models. J Mach Learn Res arXiv:arXiv:1209.4360v2
Woolhouse M, Scott F, Hudson Z et al (2012) Human viruses: discovery and emeraence. Philos Trans Roy Soc B: Biol Sci 367(1604):2864–2871. https://doi.org/10.1098/rstb.2011.0354
Zhang C, Butepage J, Kjellstrom H et al (2019) Advances in Variational Inference. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(8):2008–2026. https://doi.org/10.1109/TPAMI.2018.2889774, arXiv:1711.05597