Physics-based Penalization for Hyperparameter Estimation in Gaussian Process Regression

Ahmad, 2021, Revised learning based evolutionary assistive paradigm for surrogate selection (LEAPS2v2), Computers & Chemical Engineering, 152, 10.1016/j.compchemeng.2021.107385 Albert, 2020, Gaussian Process Regression for Data Fulfilling Linear Differential Equations with Localized Sources, Entropy, 22, 152, 10.3390/e22020152 Alifanov, 2012 Alves, 2022, A machine learning-based process operability framework using Gaussian processes, Computers & Chemical Engineering, 163, 10.1016/j.compchemeng.2022.107835 Bachoc, 2019, Maximum likelihood estimation for Gaussian processes under inequality constraints, Electronic Journal of Statistics, 13, 2921, 10.1214/19-EJS1587 Bayarri, 2007, A Framework for Validation of Computer Models, Technometrics, 49, 138, 10.1198/004017007000000092 Baydin, 2018, Automatic differentiation in machine learning: a survey, Journal of Marchine Learning Research, 18, 1 Benesty, 2009, Pearson correlation coefficient, 1 Berkenkamp, 2015, Safe and robust learning control with Gaussian processes, 2496 Blum, 2013, Optimization of Gaussian process hyperparameters using Rprop, 339 Bonzanini, 2021, Fast approximate learning-based multistage nonlinear model predictive control using Gaussian processes and deep neural networks, Computers & Chemical Engineering, 145, 10.1016/j.compchemeng.2020.107174 Boukouvala, 2012, Feasibility analysis of black-box processes using an adaptive sampling Kriging-based method, Computers & Chemical Engineering, 36, 358, 10.1016/j.compchemeng.2011.06.005 Bradley, 2022, Perspectives on the integration between first-principles and data-driven modeling, Computers & Chemical Engineering, 10.1016/j.compchemeng.2022.107898 Brooks, 1982, Streamline upwind/Petrov-Galerkin formulations for convection dominated flows with particular emphasis on the incompressible Navier-Stokes equations, Computer Methods in Applied Mechanics and Engineering, 32, 199, 10.1016/0045-7825(82)90071-8 Cao, 2013, Efficient optimization for sparse Gaussian process regression, Advances in Neural Information Processing Systems, 26 Chen, W., Xiong, Y., Tsui, K.-L., & Wang, S. (2008). A design-driven validation approach using Bayesian prediction models. Chen, 2018, How priors of initial hyperparameters affect Gaussian process regression models, Neurocomputing, 275, 1702, 10.1016/j.neucom.2017.10.028 Chib, 1995, Understanding the metropolis-hastings algorithm, The american statistician, 49, 327 Christov, 2013, On a difficulty in the formulation of initial and boundary conditions for eigenfunction expansion solutions for the start-up of fluid flow, Mechanics Research Communications, 51, 86, 10.1016/j.mechrescom.2013.05.005 Ciuperca, 2003, Penalized maximum likelihood estimator for normal mixtures, Scandinavian Journal of Statistics, 30, 45, 10.1111/1467-9469.00317 Cole, 2014, Maximum likelihood, profile likelihood, and penalized likelihood: a primer, American journal of epidemiology, 179, 252, 10.1093/aje/kwt245 Coles, 1999, Likelihood-based inference for extreme value models, Extremes, 2, 5, 10.1023/A:1009905222644 Constantinescu, 2013, Physics-based covariance models for Gaussian processes with multiple outputs, International Journal for Uncertainty Quantification, 3 Da Veiga, 2012, Gaussian process modeling with inequality constraints, 21, 529 Dai, 2022, A hybrid modeling framework using dimensional analysis for erosion predictions, Computers & Chemical Engineering, 156, 10.1016/j.compchemeng.2021.107577 Damianou, 2013, Deep gaussian processes, 207 Davis, 2007, A kriging method for the solution of nonlinear programs with black-box functions, AIChE Journal, 53, 2001, 10.1002/aic.11228 Duvenaud, 2011, Additive gaussian processes, Advances in Neural Information Processing Systems, 24 Eugene, 2020, Learning and optimization with Bayesian hybrid models, 3997 Firth, 1993, Bias reduction of maximum likelihood estimates, Biometrika, 80, 27, 10.1093/biomet/80.1.27 Fischer, B., Gorbach, N., Bauer, S., Bian, Y. A., & Buhmann, J. (2016). Model Selection for Gaussian Process Regression by Approximation Set Coding. Flyer, 2003, Accurate numerical resolution of transients in initial-boundary value problems for the heat equation, Journal of Computational Physics, 184, 526, 10.1016/S0021-9991(02)00034-7 Fornberg, 2004, On the nature of initial-boundary value solutions for dispersive equations, SIAM Journal on Applied Mathematics, 64, 546, 10.1137/S0036139902415853 Geyer, C. J., & Johnson, L. T. (2013). Mcmc: Markov chain monte carlo. In: R package version 0.9-2, URL http://CRAN.R-project.org/package=mcmc. Golchi, 2015, Monotone emulation of computer experiments, SIAM/ASA Journal on Uncertainty Quantification, 3, 370, 10.1137/140976741 Graepel, 2003, Solving noisy linear operator equations by Gaussian processes: Application to ordinary and partial differential equations, 3, 234 Grbić, 2013, Stream water temperature prediction based on Gaussian process regression, Expert systems with applications, 40, 7407, 10.1016/j.eswa.2013.06.077 Greenland, 2000, Problems due to small samples and sparse data in conditional logistic regression analysis, American journal of epidemiology, 151, 531, 10.1093/oxfordjournals.aje.a010240 Gulian, 2022, Gaussian process regression constrained by boundary value problems, Computer Methods in Applied Mechanics and Engineering, 388, 10.1016/j.cma.2021.114117 Gustafsson, O., Villani, M., & Stockhammar, P. (2020). Bayesian Optimization of Hyperparameters when the Marginal Likelihood is Estimated by MCMC. arXiv preprint arXiv:2004.10092. Jain, 2018, Learning and control using Gaussian processes, 140 Jensen, 2013, Bounded gaussian process regression, 1 Jidling, C., Wahlström, N., Wills, A., & Schön, T. (2017a). Linearly constrained Gaussian processes. Jidling, 2017, Linearly constrained Gaussian processes, Advances in Neural Information Processing Systems, 30 Joseph, 2020, Designing computer experiments with multiple types of factors: The MaxPro approach, Journal of Quality Technology, 52, 343, 10.1080/00224065.2019.1611351 Karvonen, T., & Oates, C. J. (2022). Maximum likelihood estimation in Gaussian process regression is ill-posed. arXiv preprint arXiv:2203.09179. Kennedy, 2001, Bayesian calibration of computer models, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63, 425, 10.1111/1467-9868.00294 Kevrekidis, 2017, REVISITING DIFFUSION, Quarterly of Applied Mathematics, 75, 581, 10.1090/qam/1473 Kim, J., & Choi, S. (2019). Practical Bayesian Optimization with Threshold-Guided Marginal Likelihood Maximization. arXiv preprint arXiv:1905.07540. Kim, 2022, Multi-Objective Bayesian Optimization for Design and Operating of Fluidized Bed Reactor, 49, 1297 Kocijan, 2003, Predictive control with Gaussian process models, 1, 352 Kong, 2018, Gaussian process regression for tool wear prediction, Mechanical systems and signal processing, 104, 556, 10.1016/j.ymssp.2017.11.021 Lange-Hegermann, 2018, Algorithmic linearly constrained Gaussian processes, Advances in Neural Information Processing Systems, 31 Lange-Hegermann, 2021, Linearly Constrained Gaussian Processes with Boundary Conditions, 130, 1090 Li, 2021, Deep Bayesian Gaussian processes for uncertainty estimation in electronic health records, Scientific reports, 11, 1 Liang, S., Jiang, S. W., Harlim, J., & Yang, H. (2021). Solving pdes on unknown manifolds with machine learning. arXiv preprint arXiv:2106.06682. Liu, 2009, Modularization in Bayesian analysis, with emphasis on analysis of computer models, Bayesian Analysis, 4 López-Lopera, 2018, Finite-dimensional Gaussian approximation with linear inequality constraints, SIAM/ASA Journal on Uncertainty Quantification, 6, 1224, 10.1137/17M1153157 Lorenzi, 2018, Constraining the dynamics of deep probabilistic models, 3227 Maatouk, 2017, Gaussian process emulators for computer experiments with inequality constraints, Mathematical Geosciences, 49, 557, 10.1007/s11004-017-9673-2 Manzhos, S., & Ihara, M. (2021). Rectangularization of Gaussian process regression for optimization of hyperparameters. arXiv preprint arXiv:2112.02467. Mattos, 2019, A stochastic variational framework for recurrent gaussian processes models, Neural Networks, 112, 54, 10.1016/j.neunet.2019.01.005 Mohammed, 2017, Over-fitting in model selection with Gaussian process regression, 192 Morris, 1993, Bayesian design and analysis of computer experiments: use of derivatives in surface prediction, Technometrics, 35, 243, 10.1080/00401706.1993.10485320 Nevin, 2021, Physics-informed Gaussian process regression for optical fiber communication systems, Journal of Lightwave Technology, 39, 6833, 10.1109/JLT.2021.3106714 Ng, 2022, Penalized maximum likelihood estimator for mixture of von Mises–Fisher distributions, Metrika, 1 Olofsson, 2018, Design of experiments for model discrimination using Gaussian process surrogate models, 44, 847 Olofsson, S., Schultz, E. S., Mhamdi, A., Mitsos, A., Deisenroth, M. P., & Misener, R. (2021). Using Gaussian Processes to Design Dynamic Experiments for Black-Box Model Discrimination under Uncertainty. arXiv preprint arXiv:2102.03782. Olson, 2004, Modeling a turbulent fibre suspension flowing in a planar contraction: The one-dimensional headbox, International Journal of Multiphase Flow, 30, 51, 10.1016/j.ijmultiphaseflow.2003.10.006 Pahari, 2021, Estimation of microstructural properties of wormlike micelles via a multi-scale multi-recommendation batch bayesian optimization, Industrial & Engineering Chemistry Research, 60, 15669, 10.1021/acs.iecr.1c03045 Papukdee, 2022, Penalized likelihood approach for the four-parameter kappa distribution, Journal of Applied Statistics, 49, 1559, 10.1080/02664763.2021.1871592 Paulson, 2022, COBALT: COnstrained Bayesian optimizAtion of computationaLly expensive grey-box models exploiting derivaTive information, Computers & Chemical Engineering, 160, 10.1016/j.compchemeng.2022.107700 Pensoneault, 2020, Nonnegativity-enforced Gaussian process regression, Theoretical and Applied Mechanics Letters, 10, 182, 10.1016/j.taml.2020.01.036 Petsagkourakis, 2021, Safe model-based design of experiments using Gaussian processes, Computers & Chemical Engineering, 151, 10.1016/j.compchemeng.2021.107339 Quirante, 2015, Optimization of chemical processes using surrogate models based on a Kriging interpolation, 37, 179 Rai, 2019, Gaussian process for estimating parameters of partial differential equations and its application to the Richards equation, Stochastic Environmental Research and Risk Assessment, 33, 1629, 10.1007/s00477-019-01709-8 Raissi, 2018, Hidden physics models: Machine learning of nonlinear partial differential equations, Journal of Computational Physics, 357, 125, 10.1016/j.jcp.2017.11.039 Raissi, 2017, Machine learning of linear differential equations using Gaussian processes, Journal of Computational Physics, 348, 683, 10.1016/j.jcp.2017.07.050 Raissi, 2019, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics, 378, 686, 10.1016/j.jcp.2018.10.045 Rasmussen, 2003, Gaussian processes in machine learning, 63 Riihimäki, 2010, Gaussian processes with monotonicity information, 645 Risken, 1996, Fokker-planck equation, 63 Roberts, 2013, Gaussian processes for time-series modelling, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371 Särkkä, 2011, 151 Scheffe, 1999, 72 Schweidtmann, 2021, Deterministic global optimization with Gaussian processes embedded, Mathematical Programming Computation, 13, 553, 10.1007/s12532-021-00204-y Snelson, 2003, Warped gaussian processes, Advances in Neural Information Processing Systems, 16 Solak, 2002, Derivative observations in Gaussian process models of dynamic systems, Advances in Neural Information Processing Systems, 15 Swiler, 2020, A survey of constrained Gaussian process regression: Approaches and implementation challenges, Journal of Machine Learning for Modeling and Computing, 1, 10.1615/JMachLearnModelComput.2020035155 Tamuri, 2014, A penalized-likelihood method to estimate the distribution of selection coefficients from phylogenetic data, Genetics, 197, 257, 10.1534/genetics.114.162263 Titsias, 2008, Markov chain Monte Carlo algorithms for Gaussian processes. Inference and Estimation in Probabilistic, Time-Series Models, 9, 298 Vessella, 2015, Stability Estimates for an Inverse Hyperbolic Initial Boundary Value Problem with Unknown Boundaries, SIAM J. Math. Anal., 47, 1419, 10.1137/140976212 Wang, 2021, Explicit estimation of derivatives from data and differential equations by gaussian process regression, International Journal for Uncertainty Quantification, 11 Wang, 2016, Estimating shape constrained functions using Gaussian processes, SIAM/ASA Journal on Uncertainty Quantification, 4, 1, 10.1137/140955033 Wang, 2010, A numerical method for solving the inverse heat conduction problem without initial value, Inverse Problems in Science and Engineering, 18, 655, 10.1080/17415971003698615 Wang, 2021, Variational system identification of the partial differential equations governing microstructure evolution in materials: Inference over sparse and spatially unrelated data, Computer Methods in Applied Mechanics and Engineering, 377, 10.1016/j.cma.2021.113706 Wiebe, 2022, A robust approach to warped Gaussian process-constrained optimization, Mathematical Programming, 196, 805, 10.1007/s10107-021-01762-8 Wilson, 2013, Gaussian process kernels for pattern discovery and extrapolation, 1067 Xiong, 2006, Fourier regularization method of a sideways heat equation for determining surface heat flux, Journal of Mathematical Analysis and Applications, 317, 331, 10.1016/j.jmaa.2005.12.010 Yang, 2021, Inference of dynamic systems from noisy and sparse data via manifold-constrained Gaussian processes, Proceedings of the National Academy of Sciences, 118 Yang, X., Tartakovsky, G., & Tartakovsky, A. (2018). Physics-informed kriging: A physics-informed Gaussian process regression method for data-model convergence. arXiv preprint arXiv:1809.03461. Zhang, 2001 Zhu, 1997, Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization, ACM Transactions on mathematical software (TOMS), 23, 550, 10.1145/279232.279236