DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning

Katayama, Y. et al. Real-time nanomicroscopy via three-dimensional single-particle tracking. Chem. Phys. Chem. 10, 2458–2464 (2009).

Manzo, C. & Garcia-Parajo, M. F. A review of progress in single particle tracking: from methods to biophysical insights. Rep. Prog. Phys. 78, 124601 (2015).

Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645 (2006).

Hess, S. T., Girirajan, T. P. & Mason, M. D. Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophysical J. 91, 4258–4272 (2006).

Rust, M. J., Bates, M. & Zhuang, X. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (storm). Nat. Methods 3, 793–796 (2006).

Sahl, S. J. & Moerner, W. Super-resolution fluorescence imaging with single molecules. Curr. Opin. Struct. Biol. 23, 778–787 (2013).

von Diezmann, A., Shechtman, Y. & Moerner, W. Three-dimensional localization of single molecules for super-resolution imaging and single-particle tracking. Chem. Rev. 117, 7244–7275 (2017).

Pavani, S. R. P. et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a Double-Helix point spread function. Proc. Natl Acad. Sci. USA 106, 2995–2999 (2009).

Huang, B., Wang, W., Bates, M. & Zhuang, X. Three-dimensional super-resolution imaging by stochastic optical reconstruction microscopy. Science 319, 810–813 (2008).

Shechtman, Y., Sahl, S. J., Backer, A. S. & Moerner, W. Optimal point spread function design for 3D imaging. Phys. Rev. Lett. 113, 133902 (2014).

Backer, A. S. & Moerner, W. Extending single-molecule microscopy using optical Fourier processing. J. Phys. Chem. B 118, 8313–8329 (2014).

Liu, S., Kromann, E. B., Krueger, W. D., Bewersdorf, J. & Lidke, K. A. Three-dimensional single-molecule localization using a phase retrieved pupil function. Opt. express 21, 29462–29487 (2013).

Babcock, H. P. & Zhuang, X. Analyzing single molecule localization microscopy data using cubic splines. Sci. Rep. 7, 552 (2017).

Li, Y. et al. Real-time 3D single-molecule localization using experimental point-spread functions. Nat. Methods 15, 367 (2018).

Aristov, A., Lelandais, B., Rensen, E. & Zimmer, C. Zola-3D allows flexible 3D localization microscopy over an adjustable axial range. Nat. Commun. 9, 2409 (2018).

Ferdman, B. et al. VIPR: vectorial implementation of phase retrieval for fast and accurate microscopic pixel-wise pupil estimation. Opt. Express 28, 10179–10198 (2020).

Shechtman, Y., Weiss, L. E., Backer, A. S., Sahl, S. J. & Moerner, W. Precise three-dimensional scan-free multiple-particle tracking over large axial ranges with tetrapod point spread functions. Nano Lett. 15, 4194–4199 (2015).

Min, J. et al. Falcon: fast and unbiased reconstruction of high-density super-resolution microscopy data. Sci. Rep. 4, 4577 (2014).

Boyd, N., Schiebinger, G. & Recht, B. The alternating descent conditional gradient method for sparse inverse problems. SIAM J. Optim. 27, 616–639 (2017).

Nehme, E., Weiss, L. E., Michaeli, T. & Shechtman, Y. Deep-storm: super-resolution single-molecule microscopy by deep learning. Optica 5, 458–464 (2018).

Sage, D. et al. Super-resolution fight club: assessment of 2D and 3D single-molecule localization microscopy software. Nat. Methods 16, 387 (2019).

Rivenson, Y., Zhang, Y., Günaydın, H., Teng, D. & Ozcan, A. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light-Sci. Appl. 7, 17141 (2018).

Nguyen, T., Xue, Y., Li, Y., Tian, L. & Nehmetallah, G. Deep learning approach for Fourier ptychography microscopy. Opt. Express 26, 26470–26484 (2018).

Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090 (2018).

Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F. & Johnson, G. R. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat. Methods 15, 917–920 (2018).

Krull, A., Buchholz, T.-O. & Jug, F. Noise2void-learning denoising from single noisy images. in Proc. IEEE Conference on Computer Vision and Pattern Recognition (eds. Davis, L., Torr, P. & Zhu, S. C.) 2129–2137 (2019).

Falk, T. et al. U-net: deep learning for cell counting, detection, and morphometry. Nat. Methods 16, 67–70 (2019).

Rivenson, Y. et al. Phasestain: the digital staining of label-free quantitative phase microscopy images using deep learning. Light-Sci. Appl. 8, 23 (2019).

Liu, T. et al. Deep learning-based super-resolution in coherent imaging systems. Sci. Rep. 9, 3926 (2019).

Smith, J. T. et al. Fast fit-free analysis of complex fluorescence lifetime imaging via deep learning. Proc. Natl Acad. Sci. USA 116, 24019–24030 (2019).

Boyd, N., Jonas, E., Babcock, H. P. & Recht, B. DeepLoco: fast 3D localization microscopy using neural networks. Preprint at bioRxiv https://doi.org/10.1101/267096 (2018).

Ouyang, W., Aristov, A., Lelek, M., Hao, X. & Zimmer, C. Deep learning massively accelerates super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018).

Diederic, B., Then, P., Jügler, A., Förster, R. & Heintzmann, R. cellSTORM: cost-effective super-resolution on a cellphone using dSTORM. PloS ONE 14, e0209827 (2019).

Newby, J. M., Schaefer, A. M., Lee, P. T., Forest, M. G. & Lai, S. K. Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D. Proc. Natl Acad. Sci. USA 115, 9026–9031 (2018).

Zelger, P. et al. Three-dimensional localization microscopy using deep learning. Opt. Express 26, 33166–33179 (2018).

Liu, K. et al. Fast 3D cell tracking with wide-field fluorescence microscopy through deep learning. Preprint at https://arXiv.org/abs/1805.05139 (2018).

Hershko, E., Weiss, L. E., Michaeli, T. & Shechtman, Y. Multicolor localization microscopy and point-spread-function engineering by deep learning. Opt. Express 27, 6158–6183 (2019).

Speiser, A., Turaga, S. C. & Macke, J. H. Teaching deep neural networks to localize sources in super-resolution microscopy by combining simulation-based learning and unsupervised learning. Preprint at https://arXiv.org/abs/1907.00770 (2019).

Zhang, P. et al. Analyzing complex single-molecule emission patterns with deep learning. Nat. methods 15, 913 (2018).

Chakrabarti, A. Learning sensor multiplexing design through back-propagation. in Advances in Neural Information Processing Systems (eds. Lee, D. D. et al.) 3081–3089 (Curran Associates, 2016).

Horstmeyer, R., Chen, R. Y., Kappes, B. & Judkewitz, B. Convolutional neural networks that teach microscopes how to image. Preprint at https://arXiv.org/abs/1709.07223 (2017).

Turpin, A., Vishniakou, I. & D Seelig, J. Light-scattering control in transmission and reflection with neural networks. Opt. Express 26, 30911–30929 (2018).

Haim, H., Elmalem, S., Giryes, R., Bronstein, A. M. & Marom, E. Depth estimation from a single image using deep learned phase coded mask. IEEE Trans. Comput. Imaging 4, 298–310 (2018).

He, L., Wang, G. & Hu, Z. Learning depth from single images with deep neural network embedding focal length. IEEE Trans. Image Process. 27, 4676–4689 (2018).

Sitzmann, V. et al. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37, 114 (2018).

Chang, J. & Wetzstein, G. Deep optics for monocular depth estimation and 3D object detection. in Proc. IEEE International Conference on Computer Vision (eds. Lee, K. M. et al.) 10193–10202 (2019).

Wu, Y., Boominathan, V., Chen, H., Sankaranarayanan, A. & Veeraraghavan, A. Phasecam3D: learning phase masks for passive single view depth estimation. in IEEE International Conference on Computational Photography (ed. Nedevschi, S.) 1–12 (2019).

Shechtman, Y., Weiss, L. E., Backer, A. S., Lee, M. Y. & Moerner, W. Multicolour localization microscopy by point-spread-function engineering. Nat. Photonics 10, 590 (2016).

Bickel, P. J. & Doksum, K. A. Mathematical Statistics: Basic Ideas and Selected Topics, Volumes I-II Package (Chapman and Hall/CRC, 2015).

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

Bronshtein, I. et al. Loss of lamin A function increases chromatin dynamics in the nuclear interior. Nat. Commun. 6, 8044 (2015).

Nahidiazar, L., Agronskaia, A. V., Broertjes, J., van den Broek, B. & Jalink, K. Optimizing imaging conditions for demanding multi-color super resolution localization microscopy. PLoS ONE 11, e0158884 (2016).

Ovesný, M., Křížek, P., Borkovec, J., Švindrych, Z. & Hagen, G. M. ThunderSTORM: a comprehensive ImageJ plug-in for PALM and STORM data analysis and super-resolution imaging. Bioinformatics 30, 2389–2390 (2014).

Schindelin, J. et al. Fiji: an open-source platform for biological-image analysis. Nat. Methods 9, 676 (2012).

Yu, F. & Koltun, V. Multi-scale context aggregation by dilated convolutions. Preprint at https://arXiv.org/abs/1511.07122v3 (2016).