Brain Encoding and Decoding in fMRI with Bidirectional Deep Generative Models

Kay, 2008, Identifying natural images from human brain activity, Nature, 452, 352, 10.1038/nature06713 Horikawa, 2017, Generic decoding of seen and imagined objects using hierarchical visual features, Nat Commun, 8, 15037, 10.1038/ncomms15037 Naselaris, 2011, Encoding and decoding in fMRI, NeuroImage, 56, 400, 10.1016/j.neuroimage.2010.07.073 Chen, 2014, Survey of encoding and decoding of visual stimulus via fMRI: an image analysis perspective, Brain Imaging Behav, 8, 7, 10.1007/s11682-013-9238-z Van Gerven, 2017, A primer on encoding models in sensory neuroscience, J Math Psychol, 76, 172, 10.1016/j.jmp.2016.06.009 Kay, 2013, A two-stage cascade model of BOLD responses in human visual cortex, PLoS Comput Biol, 9, 10.1371/journal.pcbi.1003079 St-Yves, 2018, The feature-weighted receptive field: an interpretable encoding model for complex feature spaces, NeuroImage, 180, 188, 10.1016/j.neuroimage.2017.06.035 Haxby, 2001, Distributed and overlapping representations of faces and objects in ventral temporal cortex, Science, 293, 2425, 10.1126/science.1063736 Haynes, 2006, Decoding mental states from brain activity in humans, Nat Rev Neurosci, 7, 523, 10.1038/nrn1931 Naselaris, 2009, Bayesian reconstruction of natural images from human brain activity, Neuron, 63, 902, 10.1016/j.neuron.2009.09.006 Horikawa, 2013, Neural decoding of visual imagery during sleep, Science, 340, 639, 10.1126/science.1234330 Miyawaki, 2008, Visual image reconstruction from human brain activity using a combination of multiscale local image decoders, Neuron, 60, 915, 10.1016/j.neuron.2008.11.004 Fujiwara, 2013, Modular encoding and decoding models derived from bayesian canonical correlation analysis, Neural Comput, 25, 979, 10.1162/NECO_a_00423 Yu S, Zheng N, Ma Y, Wu H, Chen B. A novel brain decoding method: a correlation network framework for revealing brain connections. 2017. arXiv:1712.01668. Schoenmakers, 2013, Linear reconstruction of perceived images from human brain activity, NeuroImage, 83, 951, 10.1016/j.neuroimage.2013.07.043 Schoenmakers, 2015, Gaussian mixture models and semantic gating improve reconstructions from human brain activity, Front Comput Neurosci, 8, 173, 10.3389/fncom.2014.00173 Cowen, 2014, Neural portraits of perception: reconstructing face images from evoked brain activity, NeuroImage, 94, 12, 10.1016/j.neuroimage.2014.03.018 Lee, 2016, Reconstructing perceived and retrieved faces from activity patterns in lateral parietal cortex, J Neurosci, 36, 6069, 10.1523/JNEUROSCI.4286-15.2016 Güçlütürk Y, Güçlü U, Seeliger K, Bosch S, Van Lier R, Van Gerven MA. Reconstructing perceived faces from brain activations with deep adversarial neural decoding. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, et al., editors. Advances in neural information processing systems 30 (NIPS 2017) La Jolla: Neural Information Processing Systems Foundation; 2017. p. 4249–60. Wen, 2018, Neural encoding and decoding with deep learning for dynamic natural vision, Cereb Cortex, 28, 4136, 10.1093/cercor/bhx268 Horikawa, 2017, Hierarchical neural representation of dreamed objects revealed by brain decoding with deep neural network features, Front Comput Neurosci, 11, 4, 10.3389/fncom.2017.00004 Naselaris, 2015, A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes, NeuroImage, 105, 215, 10.1016/j.neuroimage.2014.10.018 Zeidman, 2018, Bayesian population receptive field modelling, NeuroImage, 180, 173, 10.1016/j.neuroimage.2017.09.008 Güçlü, 2015, Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream, J Neurosci, 35, 10005, 10.1523/JNEUROSCI.5023-14.2015 Huth, 2016, Natural speech reveals the semantic maps that tile human cerebral cortex, Nature, 532, 453, 10.1038/nature17637 Shirer, 2012, Decoding subject-driven cognitive states with whole-brain connectivity patterns, Cereb Cortex, 22, 158, 10.1093/cercor/bhr099 Mokhtari, 2013, Decoding brain states using backward edge elimination and graph kernels in fMRI connectivity networks, J Neurosci Methods, 212, 259, 10.1016/j.jneumeth.2012.10.012 Yargholi, 2016, Brain decoding-classification of hand written digits from fMRI data employing Bayesian networks, Front Hum Neurosci, 10, 351, 10.3389/fnhum.2016.00351 Yargholi, 2016, Reconstruction of digit images from human brain fMRI activity through connectivity informed Bayesian networks, J Neurosci Methods, 257, 159, 10.1016/j.jneumeth.2015.09.032 Manning, 2018, A probabilistic approach to discovering dynamic full-brain functional connectivity patterns, NeuroImage, 180, 243, 10.1016/j.neuroimage.2018.01.071 Du, 2017, Sharing deep generative representation for perceived image reconstruction from human brain activity, 1049 Han, 2019, Variational autoencoder: an unsupervised model for modeling and decoding fMRI activity in visual cortex, NeuroImage, 198, 125, 10.1016/j.neuroimage.2019.05.039 Seeliger, 2018, Generative adversarial networks for reconstructing natural images from brain activity, NeuroImage, 181, 775, 10.1016/j.neuroimage.2018.07.043 Kuo, 2014, Decoding and encoding of visual patterns using magnetoencephalographic data represented in manifolds, NeuroImage, 102, 435, 10.1016/j.neuroimage.2014.07.046 LeCun, 2015, Deep learning, Nature, 521, 436, 10.1038/nature14539 Schmidhuber, 2015, Deep learning in neural networks: an overview, Neural Netw, 61, 85, 10.1016/j.neunet.2014.09.003 McCulloch, 1943, A logical calculus of the ideas immanent in nervous activity, Bull Math Biophys, 5, 115, 10.1007/BF02478259 Cox, 2014, Neural networks and neuroscience-inspired computer vision, Curr Biol, 24, R921, 10.1016/j.cub.2014.08.026 Cichy, 2016, Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence, Sci Rep, 6, 27755, 10.1038/srep27755 Eickenberg, 2017, Seeing it all: convolutional network layers map the function of the human visual system, NeuroImage, 152, 184, 10.1016/j.neuroimage.2016.10.001 DiCarlo, 2012, How does the brain solve visual object recognition?, Neuron, 73, 415, 10.1016/j.neuron.2012.01.010 DiCarlo, 2007, Untangling invariant object recognition, Trends Cogn Sci, 11, 333, 10.1016/j.tics.2007.06.010 Li, 2016, Visual information processing mechanism revealed by fMRI data, 85 Higgins I, Matthey L, Glorot X, Pal A, Uria B, Blundell C, et al. Early visual concept learning with unsupervised deep learning. 2016. arXiv:1606.05579. Kingma DP, Welling M. Auto-encoding variational Bayes. 2014. arXiv:1312.6114. Rezende DJ, Mohamed S, Wierstra D. Stochastic backpropagation and approximate inference in deep generative models. In: Ghahramani Z, Welling M, Cortes C, Lawrenc ND, Weinberger KQ, editors. Advances in neural information processing systems (NIPS 2014) La Jolla: Neural Information Processing Systems Foundation; 2014. p. 1278–86. Goodfellow I, Abadie JP, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrenc ND, Weinberger KQ, editors. Advances in Neural Information Processing Systems (NIPS 2014) La Jolla: Neural Information Processing Systems Foundation; 2014. p. 2672–80. St-Yves, 2018, Generative adversarial networks conditioned on brain activity reconstruct seen images Shen, 2019, End-to-end deep image reconstruction from human brain activity, Front Comput Neurosci, 13, 21, 10.3389/fncom.2019.00021 Kulkarni TD, Whitney WF, Kohli P, Tenenbaum J. Deep convolutional inverse graphics network. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R, editors. Advances in neural information processing systems 28 (NIPS 2015) La Jolla: Neural Information Processing Systems Foundation; 2015. p. 2539–47. Eslami SA, Heess N, Weber T, Tassa Y, Szepesvari D, Hinton GE, et al. Attend, infer, repeat: fast scene understanding with generative models. In: Lee DD, Sugiyama M, Luxburg UV, Guyon I, Garnett R, editors. Advances in neural information processing systems 29 (NIPS 2016) La Jolla: Neural Information Processing Systems Foundation; 2016. p. 3225–33. Norman, 2006, Beyond mind-reading: multi-voxel pattern analysis of fMRI data, Trends Cogn Sci, 10, 424, 10.1016/j.tics.2006.07.005 Isola P, Zhu J, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. 2016. arXiv:1611.07004. Liu M, Breuel T, Kautz J. Unsupervised image-to-image translation networks. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, et al., editors. Advances in neural information processing systems 30 (NIPS 2017) La Jolla: Neural Information Processing Systems Foundation; 2017. p. 700–8. Reed S, Akata Z, Yan X, Logeswaran L, Schiele B, Lee H. Generative adversarial text to image synthesis. 2016. arXiv:1605.05396. Hong S, Yang D, Choi J, Lee H. Inferring semantic layout for hierarchical text-to-image synthesis. 2018. arXiv:1801.05091. He D, Xia Y, Qin T, Wang L, Yu N, Liu T, et al. Dual learning for machine translation. In: Lee DD, Sugiyama M, Luxburg UV, Guyon I, Garnett R, editors. Advances in neural information processing systems 29 (NIPS 2016) La Jolla: Neural Information Processing Systems Foundation; 2016. p. 820–8. Xia Y, Qin T, Chen W, Bian J, Yu N, Liu T. Dual supervised learning. In: Proceedings of the 34th International Conference on Machine Learning; 2017 Aug 6–11; Sydney, Australia. Brookline: Microtome Publishing; 2017. p. 3789–98. Xia Y, Tan X, Tian F, Qin T, Yu N, Liu T. Model-level dual learning. In: Proceedings of the 35th International Conference on Machine Learning; 2018 Jul 10–15; Stockholm, Sweden. Brookline: Microtome Publishing; 2018. p. 5379–88. Zhu, 2017, Unpaired image-to-Image translation using cycle-consistent adversarial networks, 2242