A Parallel 3-Dimensional HYDROGEOCHEM and an Application to a
Proposed Waste Disposal Site at the Oak Ridge National Laboratory
J. P. Gwo (Center for Computational Sciences, Oak Ridge National
Laboratory*, P.O. Box 2008, MS6203, Oak Ridge, TN 37831,
email:g4p@ornl.gov)
G.-T. Yeh (Department of Civil and
Environmental Engineering, Pennsylvania State University, University
Park, PA 16802. email:gty@arlvax.arl.psu.edu)
Abstract
The objectives of this study are to (1) parallelize a 3-dimensional
hydrogeochemistry code and (2) to apply the parallel code to a proposed
waste disposal site at the Oak Ridge National Laboratory (ORNL). The
2-dimensional hydrogeochemistry code HYDROGEOCHEM, developed at the
Pennsylvania State University, for coupled subsurface solute transport
and chemical equilibrium processes, was first modified to accommodate
3-dimensional problem domains. A bi-conjugate gradient stabilized
linear matrix solver was then incorporated to solve the matrix
equation, whose coefficient matrix is asymmetric due to the
upstream-weighting scheme used in the transport modules. We chose to
parallelize the 3-dimensional code on the Intel Paragons at ORNL by
using an HPF (high performance FORTRAN) compiler developed at PGI. The
data- and task-parallel algorithms available in the HPF compiler prove
to be highly efficient for the geochemistry calculation. This
calculation can be easily implemented in HPF formats and is perfectly
parallel because the chemical speciation on one finite-element node is
virtually independent of those on the others. The parallel code was
then applied to the Melton Branch subwatershed at ORNL. A 56 species
uranium tailing problem was simulated to demonstrate the capability of
the parallel code. The problem domain consists of 12,090 finite
element nodes and 7 major chemical components: calcium, carbonate,
uranium, phosphate, sulfate, proton, and ferrous. Computations were
distributed over various number of processors on the Paragons to obtain
speed-up statistics.
Parallelism in Computational Hydrogeochemistry: benefit and
cost
Parallelism in computational hydrogeochemistry provides a framework for
the vertical integration of biological, physical and chemical processes
modeling from microscopic to macroscopic scales. Realistic
representation of large-scale multiple-objective groundwater resources
management problems becomes possible and the analysis of these problems
becomes achievable within economical time frames. Chemical equilibrium
calculations of a hydrogeological system are intrinsically and
perfectly parallel. This property, married with high performance
parallel computers, eases the high memory and CPU time requirements
that may otherwise overwhelm most computer platforms available today.
Speed-up obtained in computation largely reduces the cost of design and
planning, and scale-up in the size of problems and the underlying
physical and chemical processes provides scientists a vehicle to see
not only the trees but also the forest, examining the contribution of
individual component processes to the behavior of a hydrogeological
system. Cost of these benefits, however, may involve communication
overhead among processors, high labor cost or the lack of optimization
and easy-to-use parallelization tools, and parallel interpretation
tools that may facilitate the design and analysis of large-scale
multiple-objective problems. The following figure is a schematic
representation of dividing the MBSW site into subdomains and mapping
them onto an array of parallel processors on the Paragons.
Hydrogeology of the Disposal Site
The Melton Branch subwatershed (MBSW) is a forested catchment with
steep slopes and a shallow (<1 m) soil profile, overlying intensively
folded and faulted sedimentary rocks. The vadose zone reaches a
maximum thickness of ~20 m, locally decreasing to <1 m (following
topographic trends). Mean annual precipitation is on the order of 1300
mm/y; episodic, high intensity recharge events (often related to
precipitation events) characterize the region.
Saprolite, a highly
weathered shale that contains lenses of clay, comprises a 0.5-3 m zone
between the soil horizon and underlying (unweathered) shale
(see the figure).
The original structure of the parent material (e.g.
lamination, bedding planes) has been well-preserved in the saprolite.
Many of the bedding planes, fractures, and other macroporous surfaces
are coated with secondary Fe- and Mn-oxides and translocated clay
minerals. The MBSW has been the subject of experimental and theoretical
research focused on preferential flow and solute transport in the
vadose and shallow groundwater zone at a variety of scales. Careful
measurements of physical transport parameters (e.g. hydraulic
conductivity, porosity) have been made at both the laboratory and field
scales.
Hydrogeochemistry Modeling of the Melton Branch Site
Flow field calculation of the MBSW site was
performed on the Intel Paragons at ORNL, using the variably saturated
groundwater flow code PFEM which was the parallel version of 3DFEMWATER
developed by G.-T. Yeh at the Pennsylvania State University. These
velocity fields became one of the input data used by the parallel code,
PHGC3D, for hydrogeochemistry calculation on the Paragons. An uranium
mill tailing problem with 56 chemical species and a leaking source on
the southeast corner of the site was simulated using a computational
grid of 10500 finite elements. The concentration iosurfaces of four
chemical components, carbonate, iron, proton, and uranium, at
approximately 50 days after the release of the chemicals are shown
here.
Code Performance
The computation times of an 8 time-step simulation, including chemistry
equilibrium, I/O and bi-conjugate gradient solve, are reported here.
The code performance peaked at 65 nodes for this particular problem.
The chemistry equilibrium module scaled linearly as expected. The
degradation of the bi-conjugate gradient solver incurred by
communication overhead indicates that there is room for further
improvement. However, the major target of the next phase code
optimization will be the disk I/O.
Summary
An application of a parallel supercomputer hydrogeochemistry code,
PHGC3D, to a proposed waste disposal site at the Oak Ridge National
Laboratory is used to demonstrate the possibility of vertically
integrating the modeling of microscopic and macroscopic processes in a
hydrogeological system. With the potential of recent development in
high performance computation, we are looking at tremendous amount of
productivity gains in terms of facilitating the decision making,
planning, and managing of our precious water resources.
Acknowledgment
This work was funded in part by the ORNL Partnership in Computational
Sciences (PICS) program, supported by the Department of Energy's
Mathematical, Information, and Computational Sciences (MICS) Division
of the Office of Computational and Technology Research.
*Managed by Lockheed Martin Energy Research Corp. for the U.S.
Department of Energy under contract number DE-AC05-96OR22464.