Adaptive techniques for clustered N-body cosmological simulations

Springer Science and Business Media LLC - Tập 2 - Trang 1-16 - 2015
Harshitha Menon1, Lukasz Wesolowski1, Gengbin Zheng1, Pritish Jetley1, Laxmikant Kale1, Thomas Quinn2, Fabio Governato2
1Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, USA
2Department of Astronomy, University of Washington, Seattle, USA

Tóm tắt

ChaNGa is an N-body cosmology simulation application implemented using Charm++. In this paper, we present the parallel design of ChaNGa and address many challenges arising due to the high dynamic ranges of clustered datasets. We propose optimizations based on adaptive techniques. We evaluate the performance of ChaNGa on highly clustered datasets: a $z \sim0$ snapshot of a 2 billion particle realization of a 25 Mpc volume, and a 52 million particle multi-resolution realization of a dwarf galaxy. For the 25 Mpc volume, we show strong scaling on up to 128K cores of Blue Waters. We also demonstrate scaling up to 128K cores of a multi-stepping run of the 2 billion particle simulation. While the scaling of the multi-stepping run is not as good as single stepping, the throughput at 128K cores is greater by a factor of 2. We also demonstrate strong scaling on up to 512K cores of Blue Waters for two large, uniform datasets with 12 and 24 billion particles.

Tài liệu tham khảo

Planck Collaboration: Ade, PAR, Aghanim, N, Armitage-Caplan, C, Arnaud, M, Ashdown, M, Atrio-Barandela, F, Aumont, J, Baccigalupi, C, Banday, AJ, et al.: Planck 2013 results. XVI. Cosmological parameters (2013). arXiv:1303.5076 Angulo, RE, Springel, V, White, SDM, Jenkins, A, Baugh, CM, Frenk, CS: Scaling relations for galaxy clusters in the millennium-XXL simulation. Mon. Not. R. Astron. Soc. 426, 2046-2062 (2012). doi:10.1111/j.1365-2966.2012.21830.x Barnes, J, Hut, P: A hierarchical O(NlogN) force-calculation algorithm. Nature 324, 446-449 (1986) Brooks, A: Re-examining astrophysical constraints on the Dark Matter model (2014). arXiv:1407.7544 Brooks, AM, Governato, F, Booth, CM, Willman, B, Gardner, JP, Wadsley, J, Stinson, G, Quinn, T: The origin and evolution of the mass-metallicity relationship for galaxies: results from cosmological N-body simulations. Astrophys. J. Lett. 655, 17-20 (2007). doi:10.1086/511765 Christensen, C, Quinn, T, Governato, F, Stilp, A, Shen, S, Wadsley, J: Implementing molecular hydrogen in hydrodynamic simulations of galaxy formation. Mon. Not. R. Astron. Soc. 425, 3058-3076 (2012). doi:10.1111/j.1365-2966.2012.21628.x Collins, DC, Xu, H, Norman, ML, Li, H, Li, S: Cosmological adaptive mesh refinement magnetohydrodynamics with Enzo. Astrophys. J. Suppl. Ser. 186, 308-333 (2010). doi:10.1088/0067-0049/186/2/308 Davis, M, Efstathiou, G, Frenk, CS, White, SDM: The evolution of large-scale structure in a universe dominated by cold dark matter. Astrophys. J. 292, 371-394 (1985). doi:10.1086/163168 Dehnen, W: Towards optimal softening in three-dimensional N-body codes - I. Minimizing the force error. Mon. Not. R. Astron. Soc. 324, 273-291 (2001). doi:10.1046/j.1365-8711.2001.04237.x Ding, H-Q, Karasawa, N, Goddard, WA III: The reduced cell multipole method for Coulomb interactions in periodic systems with million-atom unit cells. Chem. Phys. Lett. 196, 6-10 (1992). doi:10.1016/0009-2614(92)85920-6 Durier, F, Dalla Vecchia, C: Implementation of feedback in smoothed particle hydrodynamics: towards concordance of methods. Mon. Not. R. Astron. Soc. 419, 465-478 (2012). doi:10.1111/j.1365-2966.2011.19712.x Ewald, PP: Die Berechnung optischer und elektrostatischer Gitterpotentiale. Ann. Phys. 369, 253-287 (1921). doi:10.1002/andp.19213690304 Frenk, CS, White, SDM: Dark matter and cosmic structure. Ann. Phys. 524, 507-534 (2012). doi:10.1002/andp.201200212 Gioachin, F, Sharma, A, Chakravorty, S, Mendes, C, Kale, LV, Quinn, TR: Scalable cosmology simulations on parallel machines. In: VECPAR 2006. LNCS, vol. 4395, pp. 476-489 (2007) Governato, F, Willman, B, Mayer, L, Brooks, A, Stinson, G, Valenzuela, O, Wadsley, J, Quinn, T: Forming disc galaxies in ΛCDM simulations. Mon. Not. R. Astron. Soc. 374, 1479-1494 (2007). doi:10.1111/j.1365-2966.2006.11266.x Governato, F, Zolotov, A, Pontzen, A, Christensen, C, Oh, SH, Brooks, AM, Quinn, T, Shen, S, Wadsley, J: Cuspy no more: how outflows affect the central dark matter and baryon distribution in Λ cold dark matter galaxies. Mon. Not. R. Astron. Soc. 422, 1231-1240 (2012). doi:10.1111/j.1365-2966.2012.20696.x Governato, F, Weisz, D, Pontzen, A, Loebman, S, Reed, D, Brooks, AM, Behroozi, P, Christensen, C, Madau, P, Mayer, L, Shen, S, Walker, M, Quinn, T, Wadsley, J: Faint dwarfs as a test of DM models: WDM vs. CDM (2014). arXiv:1407.0022 Guedes, J, Callegari, S, Madau, P, Mayer, L: Forming realistic late-type spirals in a ΛCDM universe: the Eris simulation. Astrophys. J. 742, 76 (2011). doi:10.1088/0004-637X/742/2/76 Habib, S, Morozov, V, Frontiere, N, Finkel, H, Pope, A, Heitmann, K: Hacc: extreme scaling and performance across diverse architectures. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. SC ’13, pp. 6-1610. ACM, New York (2013). doi:10.1145/2503210.2504566 Hopkins, PF: GIZMO: a new class of accurate. mesh-free hydrodynamic simulation methods (2014). arXiv:1409.7395 Hopkins, PF, Keres, D, Onorbe, J, Faucher-Giguere, C-A, Quataert, E, Murray, N, Bullock, JS: Galaxies on FIRE (Feedback In Realistic Environments): stellar feedback explains cosmologically inefficient star formation (2013). arXiv:1311.2073 Ishiyama, T, Nitadori, K, Makino, J: 4.45 Pflops astrophysical N-body simulation on K computer - the gravitational trillion-body problem (2012). arXiv:1211.4406 Jetley, P, Gioachin, F, Mendes, C, Kale, LV, Quinn, TR: Massively parallel cosmological simulations with ChaNGa. In: Proceedings of IEEE International Parallel and Distributed Processing Symposium 2008 (2008) Jetley, P, Wesolowski, L, Gioachin, F, Kalé, LV, Quinn, TR: Scaling hierarchical N-body simulations on GPU clusters. In: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis. SC ’10. IEEE Computer Society, Washington (2010) Kalé, LV, Sinha, A: Projections: a scalable performance tool. In: Parallel Systems Fair, International Parallel Processing Symposium, pp. 108-114 (1993) Keller, BW, Wadsley, J, Benincasa, SM, Couchman, HMP: A superbubble feedback model for galaxy simulations. Mon. Not. R. Astron. Soc. 442, 3013-3025 (2014). doi:10.1093/mnras/stu1058 Kim, J-H, Abel, T, Agertz, O, Bryan, GL, Ceverino, D, Christensen, C, Conroy, C, Dekel, A, Gnedin, NY, Goldbaum, NJ, Guedes, J, Hahn, O, Hobbs, A, Hopkins, PF, Hummels, CB, Iannuzzi, F, Keres, D, Klypin, A, Kravtsov, AV, Krumholz, MR, Kuhlen, M, Leitner, SN, Madau, P, Mayer, L, Moody, CE, Nagamine, K, Norman, ML, Onorbe, J, O’Shea, BW, Pillepich, A, Primack, JR, Quinn, T, Read, JI, Robertson, BE, Rocha, M, Rudd, DH, Shen, S, Smith, BD, Szalay, AS, Teyssier, R, Thompson, R, Todoroki, K, Turk, MJ, Wadsley, JW, Wise, JH, Zolotov, A (for the AGORA Collaboration29): The AGORA high-resolution galaxy simulations comparison project. Astrophys. J. Suppl. Ser. 210, 14 (2014). doi:10.1088/0067-0049/210/1/14 Mei, C, Zheng, G, Gioachin, F, Kalé, LV: Optimizing a parallel runtime system for multicore clusters: a case study. In: TeraGrid’10, Pittsburgh, PA, USA (2010) Mei, C, Sun, Y, Zheng, G, Bohm, EJ, Kalé, LV, Phillips, JC, Harrison, C: Enabling and scaling biomolecular simulations of 100 million atoms on petascale machines with a multicore-optimized message-driven runtime. In: Proceedings of the 2011 ACM/IEEE Conference on Supercomputing, Seattle, WA (2011) Menon, H, Jain, N, Zheng, G, Kalé, LV: Automated load balancing invocation based on application characteristics. In: IEEE Cluster, Beijing, China, vol. 12 (2012) Munshi, F, Governato, F, Brooks, AM, Christensen, C, Shen, S, Loebman, S, Moster, B, Quinn, T, Wadsley, J: Reproducing the stellar mass/halo mass relation in simulated LCDM galaxies: theory vs observational estimates. Astrophys. J. 766, 56 (2013). doi:10.1088/0004-637X/766/1/56 Pontzen, A, Governato, F: How supernova feedback turns dark matter cusps into cores. Mon. Not. R. Astron. Soc. 421, 3464-3471 (2012). doi:10.1111/j.1365-2966.2012.20571.x Pontzen, A, Governato, F: Cold dark matter heats up. Nature 506, 171-178 (2014). doi:10.1038/nature12953 Pontzen, A, Roškar, R, Stinson, GS, Woods, R: pynbody: N-Body/SPH analysis for Python. Astrophysics Source Code Library (2013). ascl:1305.002 Power, C, Navarro, JF, Jenkins, A, Frenk, CS, White, SDM, Springel, V, Stadel, J, Quinn, T: The inner structure of ΛCDM haloes - I. A numerical convergence study. Mon. Not. R. Astron. Soc. 338, 14-34 (2003). doi:10.1046/j.1365-8711.2003.05925.x Purcell, CW, Bullock, JS, Tollerud, EJ, Rocha, M, Chakrabarti, S: The Sagittarius impact as an architect of spirality and outer rings in the Milky Way. Nature 477, 301-303 (2011). doi:10.1038/nature10417 Quinn, T, Katz, N, Stadel, J, Lake, G: Time stepping N-body simulations (1997). arXiv:astro-ph/9710043 Quinn, TR, Jetley, P, Kale, LV, Gioachin, F: N-body simulations with ChaNGa. In: Kale, LV, Bhatele, A (eds.) Parallel Science and Engineering Applications: The Charm++ Approach. CRC Press, Boca Raton (2013) Reed, D, Gardner, J, Quinn, T, Stadel, J, Fardal, M, Lake, G, Governato, F: Evolution of the mass function of dark matter haloes. Mon. Not. R. Astron. Soc. 346, 565-572 (2003). doi:10.1046/j.1365-2966.2003.07113.x Ruan, JJ, Quinn, TR, Babul, A: The observable thermal and kinetic Sunyaev-Zel’dovich effect in merging galaxy clusters. Mon. Not. R. Astron. Soc. 432, 3508-3519 (2013). doi:10.1093/mnras/stt701 Schaye, J, Crain, RA, Bower, RG, Furlong, M, Schaller, M, Theuns, T, Dalla Vecchia, C, Frenk, CS, McCarthy, IG, Helly, JC, Jenkins, A, Rosas-Guevara, YM, White, SDM, Baes, M, Booth, CM, Camps, P, Navarro, JF, Qu, Y, Rahmati, A, Sawala, T, Thomas, PA, Trayford, J: The EAGLE project: simulating the evolution and assembly of galaxies and their environments (2014). arXiv:1407.7040 Sharma, A: Performance evaluation of tree structures and tree traversals for parallel n-body cosmological simulations. Master’s thesis, Department of Computer Science, University of Illinois at Urbana-Champaign (2006) http://charm.cs.uiuc.edu/papers/AmitMSThesis.html Shen, S, Wadsley, J, Stinson, G: The enrichment of the intergalactic medium with adiabatic feedback - I. Metal cooling and metal diffusion. Mon. Not. R. Astron. Soc. 407, 1581-1596 (2010). doi:10.1111/j.1365-2966.2010.17047.x Sinha, AB, Kale, LV, Ramkumar, B: A dynamic and adaptive quiescence detection algorithm. Technical Report 93-11, Parallel Programming Laboratory, Department of Computer Science, University of Illinois, Urbana-Champaign (1993) Solomonik, E, Kale, LV: Highly scalable parallel sorting. In: Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS) (2010) Springel, V: The cosmological simulation code GADGET-2. Mon. Not. R. Astron. Soc. 364, 1105-1134 (2005). doi:10.1111/j.1365-2966.2005.09655.x Springel, V: E pur si muove: Galilean-invariant cosmological hydrodynamical simulations on a moving mesh. Mon. Not. R. Astron. Soc. 401, 791-851 (2010). doi:10.1111/j.1365-2966.2009.15715.x Springel, V, White, SDM, Jenkins, A, Frenk, CS, Yoshida, N, Gao, L, Navarro, J, Thacker, R, Croton, D, Helly, J, Peacock, JA, Cole, S, Thomas, P, Couchman, H, Evrard, A, Colberg, J, Pearce, F: Simulations of the formation, evolution and clustering of galaxies and quasars. Nature 435, 629-636 (2005). doi:10.1038/nature03597 Stadel, JG: Cosmological N-body simulations and their analysis. PhD thesis, Department of Astronomy, University of Washington (March 2001) Stinson, G, Seth, A, Katz, N, Wadsley, J, Governato, F, Quinn, T: Star formation and feedback in smoothed particle hydrodynamic simulations - I. Isolated galaxies. Mon. Not. R. Astron. Soc. 373, 1074-1090 (2006). doi:10.1111/j.1365-2966.2006.11097.x Vogelsberger, M, Sijacki, D, Kereš, D, Springel, V, Hernquist, L: Moving mesh cosmology: numerical techniques and global statistics. Mon. Not. R. Astron. Soc. 425, 3024-3057 (2012). doi:10.1111/j.1365-2966.2012.21590.x Vogelsberger, M, Genel, S, Springel, V, Torrey, P, Sijacki, D, Xu, D, Snyder, GF, Nelson, D, Hernquist, L: Introducing the illustris project: simulating the coevolution of dark and visible matter in the Universe (2014). arXiv:1405.2921 Wadsley, JW, Stadel, J, Quinn, T: Gasoline: a flexible, parallel implementation of TreeSPH. New Astron. 9, 137-158 (2004) Wadsley, JW, Veeravalli, G, Couchman, HMP: On the treatment of entropy mixing in numerical cosmology. Mon. Not. R. Astron. Soc. 387, 427-438 (2008). doi:10.1111/j.1365-2966.2008.13260.x Warren, MS: 2HOT: an improved parallel hashed oct-tree N-body algorithm for cosmological simulation (2013). arXiv:1310.4502 Zheng, G: Achieving high performance on extremely large parallel machines: performance prediction and load balancing. PhD thesis, Department of Computer Science, University of Illinois at Urbana-Champaign (2005)