The TERRA Earth model was originally developed by John Baumgardner in 1985, and parallelized by Hans-Peter Bunge for RISC workstation clusters in 1993. Since that time the model has been continuously upgraded and enhanced, and is now widely used by geodynamicists. A version of this model is a designated high performance target code for the NASA HPCC Earth and Space Sciences Grand Challenge Project, due to its excellent efficiency on parallel machines. Recently the model demonstrated sustained performance of 100 GFLOPS on the 1024 processor NASA Goddard CRAY T3E. Over the past few months, TERRA has been ported to dedicated high-speed Beowulf clusters using the PGF90 Fortran 90 compiler from the Portland Group, Inc. A Beowulf cluster is a parallel high-performance computing system constructed using off-the-shelf PC hardware in a cluster configuration, typically late-generation Intel Pentium II systems interconnected using fast Ethernet. Terra is a finite element code defined on a nested geodesic grid, which is derived from the regular icosahedron [Fig 1]. The mesh allows an almost uniform discretization of the sphere and avoids the 'pole-problem' of traditional latitude-longitude grids. (In such grids the narrow polar cells impose un-necessarily small time-steps.) An efficient multigrid solver computes the velocity and pressure fields from equations of momentum and mass conservation at every time step. TERRA's ability to handle strong lateral variations of viscosity is especially important to model the deformation history of silicate rocks, which can change their strength by many orders of magnitude. In the TERRA model, the primitive equations that describe the dynamics and thermodynamics of the slow internal deformation of the earth are solved. These equations are essentially the Navier-Stokes equations for a highly viscous fluid. Because rocks deform very slowly (over many thousands of years), it is appropriate to neglect the effects from inertia and rotation. The slight variations in density, due to variations in pressure, temperature and chemistry of rocks, give rise to buoyancy forces, which allow the Earth interior to overturn slowly with time. Such overturns may have occurred as often as 20 times during Earth's history, taking about 200 million years each for their completion. They are the most effective mechanism to cool the planet, and also provide the driving force to move tectonic plates and rift the continents apart. The parallel version of TERRA presented here uses message-passing in conjunction with Fortran 90. The computational domain is decomposed and distributed across processors. Because communication is mostly local to connect subdomain boundaries, the parallel overhead shows a simple scaling that goes with the surface-to-volume ratio of each subdomain. Off-the-shelf Intel processor-based PC hardware now has sufficient speed and memory to handle relatively large subdomains, such that the penalty from communication is inconsequential. A parallel efficiency of more than 90 percent on dedicated Beowulf clusters can be obtained at modest cost. Porting TERRA to a Beowulf cluster turned out to be quite simple. PGF90
for Intel processor-based workstations proved to be the compiler of choice,
resulting in very fast machine code that significantly surpassed our expectations. For
example, we verified a sustained per-processor speed of more than 150 MFLOPS on a single
450 Mhz Intel Pentium II. This per-processor performance is comparable to some of the
fastest RISC processors available [table 1]. At 90 percent parallel efficiency, sustained
performance of 10 GFLOPS is expected on the new 72 processor Pentium II Beowulf cluster
under construction at Princeton's Geosciences
Department. Such high speed, but at reasonable cost, is essential when
attempting to perform large 3-D simulations at the level of university departments.
Table 1: TERRA Performance on various microprocessor-based parallel systems Equally important in our choice of PGF90 for Beowulf clusters was the wide availability of Portland Group High Performance Fortran (PGHPF) on high-end systems. With installations on 39 of the Top 100 systems worldwide, and nearly 20% of the Top 500, PGHPF is effectively a de-facto-standard. Using the PGI Fortran compilers has enabled our source-code to be ported across platforms at minimal cost. In the case of TERRA, we have run identical source-code on a dedicated 4 processor Pentium II cluster at the Department of Numerical Modelisation at the Institute de Physique du Globe in Paris using PGF90, as well as the 512 processor Edinburgh CRAY T3D using PGHPF operating in F90 mode. Presented here are simulations of the slow overturn of the Earth's mantle
(the 3000 km deep region between the surface and the molten iron core of the Earth) at a
snapshot in time. More than 10 million finite elements are needed to represent this
simulation. We show the temperature distribution using a simple color scheme: Blue is cold
and heavy, while red is hot and buoyant. The upper 200km of the Earth are removed, to gain
an un-obscured view of the Earth's interior. A simple reference model [Fig2] does not
reproduce the geologic observation of large oceanic and continental plates occasionally
disconnected by relatively few and elongated subduction zones. (Recall that the "Ring
of Fire" surrounding the Pacific ocean is powered by a prominent subduction system,
along which old oceanic plates sink back into the Earth's interior.) Instead, the surface
sinks back into the interior along relatively dense and pointlike structures, quite unlike
the plate tectonic style preferred on Earth. Here we have made rocks of the deep-Earth 30 times stronger than rocks of the shallower regions for all depth levels below 670 km. Such increase in the strength of deep-Earth rocks is strongly suggested by many different geophysical observations. The one-parameter change results in a convection structure that is radically different from the simple reference model shown above. The blue and cold downwelling regions are now organized along a widely spaced and interconnected system of downwelling lines, remarkably similar to the pattern preferred by subduction zones on Earth. The result demonstrates that the deep Earth plays an important role in structuring the scale and pattern of geologic motion at the surface. It also shows how geodynamicists can use computer simulations to infer the secrets of the deep Earth interior. Hans-Peter Bunge is a researcher and faculty member in the Department of Geosciences at Princeton University. He can be reached by email at bunge@geo.princeton.edu. Acknowledgement: This work was generously supported by the Institute of Geophysics and Planetary Physics (IGPP) and the Advanced Computing Laboratory (ACL) both at Los Alamos National Laboratory (LANL). All computer simulations were performed at the ACL.
|
