SAP Project:

Simulation of Novel Biofuel Liquid Combustors: Toward Petaflop Computing

Farzad Mashayek

Department of Mechanical and Industrial Engineering

University of Illinois at Chicago

Research Objectives
SCIENTIFIC GOALS
The engines that propel ships, jets, and many other devices must rely heavily on combustion in a two-phase environment. Within the combustion chamber the liquid fuel, which is first atomized into small droplets, undergoes evaporation and reaction with the oxidizer gas. Under a current support from the Office of Naval Research (ONR), and in collaboration with the University of Minnesota, we are investigating novel ideas for increasing heat release while maintaining a stable flame in liquid-fuel dump combustors (see the schematic in figure 1). These ideas include

• Application of “counterflow” to increase turbulence levels while decreasing the size of the recirculation zone, thus increasing the flame speed and shortening the length of the combustor.
• Modification of the “step geometry” (as shown in figure 1) to help with flame anchoring.
• Implementation of “microjets” to enhance three-dimensionality of the flame at the step hence increasing flame stability.

In addition, we are investigating the application of “electrostatic atomization” to increase dispersion of the fuel droplets. In this method, the droplets carry similar charges which prevent them from agglomeration, thus improving the efficiency of combustion and reducing the emission of harmful gases and particulate matters. The electrostatic forces within the liquid fuel also significantly improve the atomization process at much lower back pressures. The electrostatic atomization is particularly effective for highly viscous liquids, making it an extremely viable choice for atomization of vegetable oils as an alternative fuel.

Figure 1: Schematic of liquid-fuel combustor with counterflow and modified step geometry.

 

COMPUTATIONAL GOALS AND METHODS
To enhance the rate of mixing and reaction, liquid-fuel combustors are designed to operate in the presence of turbulence. Turbulence, however, significantly adds to the complexity of the flow and to the challenges encountered in the theoretical description and practical prediction of the phenomena involved. These challenges are primarily a consequence of the numerous degrees of freedom associated with a turbulent flow, which has remained as one of the major unsolved problems in science. The addition of fuel droplets, with the associated phenomena of evaporation and chemical reaction, further introduces new sets of variables, often accompanied by large variations in scales.

With the advent of supercomputers and the rapid growth in computational power, direct numerical simulation (DNS) has emerged as a viable choice for prediction of turbulent flows in simple geometries. However, it seems unlikely that (at least for a foreseeable future) turbulent flows of practical interest can be fully accessed via DNS. Therefore, modeling is needed to decrease the number of degrees of freedom so that they can be resolved numerically with (preferably much) less effort. The models are generally derived following a long and complex mathematical procedure, subject to simplifying assumptions; thus, they must be assessed against more accurate data. Laboratory measurements can be used to generate the data for model validation; however, they are usually expensive to produce, and also mostly limited to somewhat global and average flow quantities. An alternative can be provided by DNS results for a preliminary assessment, as well as for the physical insight that is critical in the development of the models. This procedure could potentially postpone the need for experimental data till the final stages of model validation and the implementation for more practical flows.

The goal of this project is to implement DNS along with large-eddy simulation (LES) and Reynolds-averaged Navier-Stokes (RANS) modeling approaches in an intelligent manner for simulation of novel concepts in liquid-fuel combustors. For RANS, we will be using the Fluent commercial package which is available on NCSA machines. Our numerical approach for DNS and LES is discussed below.

Computational Approach
The multiscale, multiphysics nature of the problem demands that an efficient and accurate computational method be employed for the simulations. In recent years, high- order schemes within the h/p finite element framework have demonstrated great promise at tackling large scale simulations involving complex physics. Some of the attractive features of this method include minimal numerical dissipation, exponential convergence rate, and ability to deal with complex domains. Moreover the method is very amenable to efficient parallelization, a feature that is at the heart of high performance computing.

Therefore, for our simulations we employ a high-order approximation scheme on a fully unstructured grid with triangular/tetrahedral elements. As described before one of the design aspect is to modify the combustor geometry from the traditional backward-facing step configuration. The use of tetrahedral elements which are more boundary conforming than hexahedrons is advantageous in this context. However, the added challenge is to determine suitable approximation points (nodes) that would lead to stable and efficient numerical scheme. A sample grid is shown in figure 2. With a high-order polynomial approximation in place, the governing partial differential equations are discretized using the standard Galerkin technique. The semi-discrete equation is then solved in time using a low storage 4 th order Runge-Kutta scheme.

The parallel implementation is based on the domain decomposition of the underlying grid using graph partitioning programs (e.g. METIS). Non blocking MPI routines are employed for interprocessor communication. From a computational viewpoint the most time and memory intensive element of the algorithm is the computation of the spatial derivatives using matrix-vector products. Therefore, for performance gains we use linear algebra subroutines that are optimized for memory/cache usage.

             

Figure 2: A typical unstructured mesh in a two-dimensional dump-combustor.

 

POTENTIAL BENEFITS
The principal focus of this joint effort is to improve the performance of the code on large parallel computing resources. The potential benefits could be multi-pronged

1. Scaling behavior - Preliminary scale up studies has shown that there is room for improvement in the parallel performance of the code. The scale up analysis results are shown in Figure 3. It should be pointed out that the above study was done in a shared resources set up and may not reflect the true behavior of the code.

2. I/O performance – Current version of the code does not employ any parallel I/O libraries. Streamlining I/O feature could improve the code performance substantially.

3. Optimized computational subroutines – The code has been used with linear algebra packages like ATLAS (Linux clusters) and ESSL (IBM clusters) for matrix related operations. Other packages may be tested in-order to select the optimal package.

4. Post processing and data visualization – Post processing and data visualization are integral part of any numerical simulation. DNS/LES generates huge volumes of data, which are to be analyzed using advanced post processing tools. Moreover, multiphysics results often require that all the components be visualized at the same time for better understanding. A good example would be simultaneous visualization of the fuel droplets and the flame to study the correlation between the two. Smart techniques needs to be devised to achieve this.

Performance Study of the Code
This section provides a parallel performance analysis of the code. We simulated non-reacting flow through a 3D plane channel at Re = 3000. The table below enumerates the computational time and memory requirement for different processors. It should be pointed out that we do not run the simulation for the time necessary to attain quasi-steady state but only for 5 non-dimensional time units. Figure 3 shows the scale up nature of the code based on the wall clock times.

Case	Re	N (Grid points)	Proc	Peak Task Memory(MB)	Wall Clock time (hh:mm:ss)
1	3000	242,600	1	206.40	28:44:02
			2	103.20	14:22:01
			4	55.74	09:33:31
			8	32.30	04:42:03
			16	20.60	02:17:34

Scale-up to Higher Reynolds Numbers and Reacting Flows
The above data together with empirical scaling laws are used to estimate the total computational time necessary for laboratory scale and industrial scale Reynolds numbers. The real time necessary to advance one time step scales as, t ~ Re L 3.6 (for DNS) and Re L 2.4 (LES). The DNS scaling is used for the estimations. In practice, the total simulation time, which includes the time necessary to achieve statistical stationarity and subsequent computation of turbulent statistics is taken as 12 flow through times. Flow through time is that taken by a fluid particle to traverse the computation domain. Additional scaling considerations need to be accounted for simulations with chemical reaction and liquid droplets. Let S be the wall clock time necessary to advance a single transport equation by one time step. Then the total time necessary to advance the computation involving N chemical species and M step reaction mechanism will be (approximately) = S*(5+N+M+3*P). Here P is the factor which represents the time to advance all the droplets in one coordinate direction relative to S and taken as 0.25. In this analysis we consider 5 chemical species and a 5-step reaction for the combustion simulation.

Based on the above arguments, the following tables provides the estimated data for Reynolds number of 13,600, representative of a laboratory flow and 60,000 representative of an industrial case. The first row indicates the times for 16 processors (currently available resource) and the second row scales up the data to 160,000 processors (petascale resource).

Re = 13,600:

N (Grid Points)	Proc #	Peak Task Memory (MB)	Wall time for single computational time step (dd:hh:mm:ss) Non- reacting flow	Wall time for single computational time (dd:hh:mm:ss) Reacting flow	Total simulation time (dd: hh:mm:ss)
55,798,000	16	4738	04:09:28:12	13:20:08:24	1328:21:26:24
55,798,000	160,000	0.6	00:00:00:38	00:00:01:59	00:03:10:12

 

Re = 60,000:

N (Grid Points)	Proc #	Peak Task Memory (MB)	Wall time for single computational time step (dd:hh:mm:ss) Non-reacting flow	Wall time for single computational time (dd:hh:mm:ss) Reacting flow	Total simulation time (dd:hh:mm:ss)
1.17x10 10	16	994423	922:07:48:36	2905:07:53:24	278911:18:13:12
1.17x10 10	160,000	127	00:02:12:36	00:06:58:12	27:21:16:48

 

 

Figure 3. Code scale up study.

 

PUBLICATIONS
Featured Article: “Reacting Fuel Droplets in Combustion Engines,” envision , science magazine of the National Partnership for Advanced Computational Infrastructure (NPACI) & San Diego Supercomputing Center (SDSC), October-December 1999.

1. Elhami Amiri, A., Kazemzadeh Hannani, S., and Mashayek, F., “ Large-Eddy Simulation of Heavy-Particle Transport in Turbulent Channel Flow ,” Numerical Heat Transfer, Part B, 50 (4), 285-313, 2006 .
2. Shotorban B. and Mashayek, F., “A Stochastic Model for Particle Motion in Large-Eddy Simulation,” Journal of Turbulence, 7 (18), 1-13, 2006 .
3. Shotorban B. and Mashayek, F., “Modeling Subgrid-Scale Effects on Particles by Approximate Deconvolution,” Physics of Fluids, 17(8), 081701, 2005.
4. Sengupta, K., Russell, K., Minkowycz, W.J., and Mashayek, F., “Numerical Simulation Data for Assessment of Particle-Laden Turbulent Flow Models,” International Journal of Heat and Mass Transfer,48(15), 3035-3046, 2005.
5. Elhami Amiri, A., Kazemzadeh Hannani, S., and Mashayek, F., “Evaluation of a Fourth-Order Finite Volume Compact Scheme for LES with Explicit Filtering,” Numerical Heat Transfer, Part B, 48 (2), 147-164, 2005.
6. Jacobs, G.B., Kopriva, D.A. and Mashayek, F., “Validation Study of a Multidomain Spectral Code for Simulation of Turbulent Flows,” AIAA Journal,43(6), 1256-1264, 2005.
7. Jacobs, G.B., Kopriva, D.A., and Mashayek, F., “Compressible Subsonic Particle-Laden Flow over a Square Cylinder,” AIAAJournal of Propulsion and Power,20(2), 353-359, 2004.
8. Mashayek, F. and Pandya, R.V.R., “Analytical Description of Particle/Droplet-Laden Turbulent Flows,” Progress in Energy and Combustion Science, 29(4), 329-378, 2003.
9. Shotorban, B., Mashayek, F., and Pandya, R.V.R., “Temperature Statistics in Particle-Laden Turbulent Homogeneous Shear Flow,” International Journal of Multiphase Flow, 29(8), 1333-1353, 2003.
10. Jacobs, G.B., Kopriva, D.A., and Mashayek, F., “A Comparison of Outflow Boundary Conditions for the Multidomain Staggered-Grid Spectral Method,” Numerical Heat Transfer, Part B, 44(3), 225-251, 2003.
11. Pandya, R.V.R. and Mashayek, F., “Two-Fluid Large-eddy Simulation Approach for Particle-Laden Turbulent Flows,” International Journal for Heat and Mass Transfer, 45(24), 4753-4759, 2002.
12. Mashayek, F. and Taulbee, D.B., “A Four-Equation Model for Prediction of Gas-Solid Turbulent Flows,” Numerical Heat Transfer, Part B,41(2), 95-116, 2002.
13. Mashayek, F. and Taulbee, D.B., “Turbulent Gas-Solid Flows. Part I: Direct Simulations and Reynolds Stress Closures,” Numerical Heat Transfer, Part B, 41 (1), 1-29, 2002.
14. Mashayek, F. and Taulbee, D.B., “Turbulent Gas-Solid Flows. Part II: Explicit Algebraic Closures,” Numerical Heat Transfer, Part B, 41 (1), 31-52, 2002.
15. Liao, S., Mashayek, F., and Guo, D., “Numerical Simulations of Particle-Laden Axisymmetric Turbulent Flows,” Numerical Heat Transfer, Part A, 39(8), 847-855, 2001.
16. Mashayek, F. and Jacobs, G.B., “Temperature-Dependent Reaction in Droplet-Laden Homogeneous Turbulence,” Numerical Heat Transfer, Part A, 39, 101-121, 2001.
17. Mashayek, F., “Velocity and Temperature Statistics in Reacting Droplet-Laden Homogeneous Shear Turbulence,” AIAA Journal of Propulsion and Power,17(1), 197-202, 2001.
18. Barre, C., Mashayek, F., and Taulbee, D.B., “Statistics in Particle-Laden Plane Strain Turbulence by Direct Numerical Simulation,” International Journal of Multiphase Flow, 27(2), 347-378, 2001.
19. Mashayek, F., “Numerical Investigation of Reacting Droplets in Homogeneous Shear Turbulence,” Journal of Fluid Mechanics, 405, 1-36, 2000.
20. Jaberi, F.A. and Mashayek, F., “Temperature Decay in Two-Phase Turbulent Flows,” International Journal of Heat and Mass Transfer, 43(6), 993-1005, 2000.
21. Mashayek, F., “Simulations of Reacting Droplets Dispersed in Isotropic Turbulence,” AIAA Journal, 37(11), 1420-1425, 1999.
22. Taulbee, D.B., Mashayek, F., and Barre, C., “Simulation and Reynolds Stress Modeling of Particle-Laden Turbulent Shear Flows,” International Journal of Heat and Fluid Flow, 20(4), 368-373, 1999.
23. Jaberi, F.A., James, S., and Mashayek, F., “Direct Numerical Simulations of Parallel/Series Reactions in Turbulent Flows,” Chemical Engineering Communications, 173, 215-244, 1999.
24. Mashayek, F. and Jaberi, F.A., “Particle Dispersion in Forced Isotropic Low Mach Number Turbulent Flows,” International Journal of Heat and Mass Transfer, 42(15), 2823-2836, 1999.
25. Mashayek, F., “Droplet-Turbulence Interactions in Low Mach Number Homogeneous Shear Two-Phase Flows,” Journal of Fluid Mechanics, 367, 163-203, 1998.
26. Mashayek, F., “Direct Numerical Simulations of Evaporating Droplet Dispersion in Forced Low Mach Number Turbulence,” International Journal of Heat and Mass Transfer, 41(17), 2601-2617, 1998.
27. Mashayek, F., Jaberi, F.A., Miller, R.S., and Givi, P., “Dispersion and Polydispersity of Liquid Drops in Stationary Isotropic Turbulence,” International Journal of Multiphase Flow, 23(2), 337-355, 1997.