SAP Project:

Implementing Effects Due to Stellar Evolution in a Cosmological Simulation Code

Paul Ricker

National Center for Supercomputing Applications (NCSA)

University of Illinois at Urbana-Champaign

Material Associated with this Project: News article | Final Report

Research Objectives
SCIENTIFIC GOALS
As the most massive collapsed objects in the universe, clusters of galaxies offer particularly important insights into the cosmos as a whole. Because they require billions of years to virialize, clusters’ structural evolution is set by the rate of expansion of the universe, which in turn is determined by the amount of matter and energy in the Universe. Statisticalmeasures of clusters such as their mass distribution and power spectrum also reflect the influence of cosmology, since clusters are the most recent structures to have separated from the cosmic expansion. Their lateness on the cosmic stage makes them especially sensitive to the properties of dark energy. (Read more...)

COMPUTATIONAL GOALS AND METHODS
To study cluster evolution, we use a simulation framework called FLASH that was originally developed by the ASCI Center for Astrophysical Thermonuclear Flashes at the University of Chicago. FLASH (Fryxell et al. 2000) was designed to study X-ray bursts, novae, and Type Ia supernovae using AMR on large parallel computers. FLASH has been used to perform some of the largest AMR calculations ever attempted, earning its authors the 2000 Gordon Bell Prize (Calder et al. 2000). It is nearly unique among astrophysical codes in having been extensively validated against laboratory experiments (Calder et al. 2002). Since its beginning, FLASH has evolved into a general-purpose astrophysical simulation tool, including modules for gravity, magnetohydrodynamics, and particles. (Read more....)

POTENTIAL BENEFITS
We will develop a subgrid module for FLASH that will enable the following physics to be
accurately modeled within hydrodynamical simulations having spatial resolutions of order 100 pc or greater. As a starting point, we will use the hybrid multiphase star formation model described by Springel and Hernquist (2003) for smoothed-particle hydrodynamics. (Read more....)

COMPUTATIONAL APPROACH
FLASH is an MPI-based parallel application and should port readily to NCSA clusters.
We will target two platforms for porting, development, and optimization: tungsten, because of the large number of processors on this machine; and mercury, because of the large amount of memory available. This strategy will also allow us to compare the code’s performance on 32-bit and 64-bit platforms. (Read more....)

ACCOMPLISHMENTS AND SIGNIFICANCE
The development of subgrid modules can be done on smaller machines, but as noted in the proposal, the large spatial dynamic range required for cosmological simulations, even with AMR and subgrid modeling, requires production simulations to be carried out on large parallel machines. NCSA is one of the few places where these simulations can be carried out. It also has people with expertise in optimization on the target platforms, which will help us to improve the code's overall performance and do bigger (which in this field means better) calculations than we might otherwise have done.

Development of FLASH is going forward at the University of Chicago, but they are using the code to simulate Type Ia supernova explosions and are not interested in cosmological simulation. My group at UIUC is the main one driving development of particles, gravity, cosmological expansion, and stellar subgrid models in the code. However, we haven't the manpower or the optimization expertise to do this without help from NCSA.

FLASH is a community simulation code; it is available for free (download). More than 200 people have downloaded it so far. We get a steady stream of questions on the flash-users mailing list. Questions pertain to applications of FLASH in areas including cosmological large-scale structure, elliptical galaxies, radio jets in clusters of galaxies, and interstellar turbulence. It is catching on as an astrophysical community code. By developing stellar evolution subgrid models and improving the overall performance on NCSA platforms, we will be encouraging more of these users to consider running their applications at NCSA, and we will be improving the overall productivity of workers in astrophysical simulation. Improvements that we make will eventually find their way into the distributed version of the code so that others may benefit.

An additional benefit to the astronomy community will be online access to large simulation datasets. Once our results are published, we plan to make our simulations publicly available, if not in raw form then through the Laboratory for Cosmological Data Mining to enable scientists elsewhere to ask new questions of our simulation results.

Two of the deepest questions in cosmology right now are the nature of dark matter and the nature of dark energy. The rate of evolution of galaxy clusters and their statistics (primarily the mass distribution and power spectrum) are sensitive to the properties of both dark matter and dark energy. For this reason several large surveys in different wavebands (microwave, optical, X-ray) are being developed to obtain samples of more than 10^4 clusters out to a redshift of about 1. These surveys essentially measure surrogates for cluster mass, such as X-ray temperature or Sunyaev-Zel'dovich decrement.  To do accurate theoretical comparisons with these new observational datasets (and thus to use the data to derive constraints on the properties of, in particular, dark energy), it will be important to have simulations that (a) include comparable numbers of galaxy clusters and (b) self-consistently include the gas and galaxy physics that gives rise to these observable quantities. To do these simulations, we need to be able to run efficiently on hundreds of processors, and we need to include subgrid models for stellar evolution, calibrated against nearby galaxies and clusters of galaxies.

It may turn out that the solution to the question of dark energy involves revisions to our understanding of gravity. However, even in this case the code will be a useful laboratory for testing such ideas, because it is unlikely that our understanding of fluid dynamics and gravity on stellar scales would need to change.

PUBLICATIONS

• Allen, S. W. and Fabian, A. C. 1998, MNRAS, 297, L57
• Arnaud, M. and Evrard, A. E. 1998, MNRAS, 305, 631
• Berger, M. J. and Oliger, J. 1984, JCP, 53, 484
• Berger, M. J. and Colella, P. 1989, JCP, 82, 64
• Bertschinger, E. 2001, ApJS, 137, 1
• Bialek, J. J., Evrard, A. E., and Mohr, J. J. 2001, ApJ, 555, 597
• Blanton, E. L., Sarazin, C. L., and McNamara, B. R. 2003, ApJ, 585, 227
• Borgani, S., et al. 2001, ApJ, 559, L71
• Brandt, A. 1977, Math. Comp., 31, 333
• Bryan, G. L. 2000, ApJ, 544, L1
• Calder, A. C., et. al. 2000, in Proc. Supercomputing 2000.
• _________ 2002, ApJS, 143, 201
• Colella, P. and Woodward, P. 1984, JCP, 54, 174
• De Zeeuw, D. and Powell, K. G. 1993, JCP, 104, 56
• Forman, W., et al. 2002, in Lighthouses of the Universe (ESO), 51
• Fryxell, B., et al. 2000, ApJS, 131, 273
• Hockney, R. W. and Eastwood, J. W. 1988, Computer Simulation using Particles (IOP)
• Kaiser, N. 1986, MNRAS, 222, 323
• Knebe, A., Green, A., and Binney, J. 2001, MNRAS, 325, 845
• Kravtsov, A. V. 1999, PhD thesis, New Mexico State University
• L¨ohner, R. 1987, Comp. Meth. App. Mech. Eng., 61, 323
• MacNeice, P., et al. 2000, Comp. Phys. Comm., 126, 330
• Markevitch, M. 1998, ApJ, 504, 27
• Martin, D. 1998, PhD thesis, Univ. of California at Berkeley
• Martin, D. and Cartwright, K. 1996, Tech. Rep. UCB/ERL M96/66 (UC Berkeley)
• McNamara, B. R., et al. 2000, ApJ, 534, L135.
• _________ 2001, ApJ, 562, L149
• Metzler, C. and Evrard, A. E. astro-ph/9710324
• Mohr, J. J. and Evrard, A. E. 1997, ApJ, 491, 38
• Mohr, J. J., Mathiesen, B., and Evrard, A. E. 1999, ApJ, 517, 627
• Norman, M. L. and Bryan, G. L. 1999, in The Radio Galaxy Messier 87, H.-J. R¨oser and K. Meisenheimer, eds. (Springer), 106
• Pen, U. 1997, ApJ, 490, L127
• Quirk, J. J. 1991, PhD thesis, Cranfield Institute of Technology
• Ricker, P. M. and Sarazin, C. L. 2001, ApJ, 561, 621
• Sarazin, C. L. 1988, X-Ray Emission from Clusters of Galaxies (Cambridge)
• Shandarin, S. F. 1987, in Large Scale Structures of the Universe: Proc. 130th IAU Symposium (Kluwer), 273
• Springel, V. and Hernquist, L. 2003, MNRAS, 339, 289
• Voit, G. M., et al. 2002, ApJ, 576, 601
• Warren, M. S. and Salmon, J. K. 1993, in Proc. Supercomputing 1993 (IEEE Comp. Soc.), 12