A Parallel 3-Dimensional HYDROGEOCHEM and an Application to a Proposed Waste Disposal Site at the Oak Ridge National Laboratory

J. P. Gwo (Center for Computational Sciences, Oak Ridge National Laboratory*, P.O. Box 2008, MS6203, Oak Ridge, TN 37831, email:g4p@ornl.gov)

G.-T. Yeh (Department of Civil and Environmental Engineering, Pennsylvania State University, University Park, PA 16802. email:gty@arlvax.arl.psu.edu)


The objectives of this study are to (1) parallelize a 3-dimensional hydrogeochemistry code and (2) to apply the parallel code to a proposed waste disposal site at the Oak Ridge National Laboratory (ORNL). The 2-dimensional hydrogeochemistry code HYDROGEOCHEM, developed at the Pennsylvania State University, for coupled subsurface solute transport and chemical equilibrium processes, was first modified to accommodate 3-dimensional problem domains. A bi-conjugate gradient stabilized linear matrix solver was then incorporated to solve the matrix equation, whose coefficient matrix is asymmetric due to the upstream-weighting scheme used in the transport modules. We chose to parallelize the 3-dimensional code on the Intel Paragons at ORNL by using an HPF (high performance FORTRAN) compiler developed at PGI. The data- and task-parallel algorithms available in the HPF compiler prove to be highly efficient for the geochemistry calculation. This calculation can be easily implemented in HPF formats and is perfectly parallel because the chemical speciation on one finite-element node is virtually independent of those on the others. The parallel code was then applied to the Melton Branch subwatershed at ORNL. A 56 species uranium tailing problem was simulated to demonstrate the capability of the parallel code. The problem domain consists of 12,090 finite element nodes and 7 major chemical components: calcium, carbonate, uranium, phosphate, sulfate, proton, and ferrous. Computations were distributed over various number of processors on the Paragons to obtain speed-up statistics.

Parallelism in Computational Hydrogeochemistry: benefit and cost

Parallelism in computational hydrogeochemistry provides a framework for the vertical integration of biological, physical and chemical processes modeling from microscopic to macroscopic scales. Realistic representation of large-scale multiple-objective groundwater resources management problems becomes possible and the analysis of these problems becomes achievable within economical time frames. Chemical equilibrium calculations of a hydrogeological system are intrinsically and perfectly parallel. This property, married with high performance parallel computers, eases the high memory and CPU time requirements that may otherwise overwhelm most computer platforms available today. Speed-up obtained in computation largely reduces the cost of design and planning, and scale-up in the size of problems and the underlying physical and chemical processes provides scientists a vehicle to see not only the trees but also the forest, examining the contribution of individual component processes to the behavior of a hydrogeological system. Cost of these benefits, however, may involve communication overhead among processors, high labor cost or the lack of optimization and easy-to-use parallelization tools, and parallel interpretation tools that may facilitate the design and analysis of large-scale multiple-objective problems. The following figure is a schematic representation of dividing the MBSW site into subdomains and mapping them onto an array of parallel processors on the Paragons.

Hydrogeology of the Disposal Site

The Melton Branch subwatershed (MBSW) is a forested catchment with steep slopes and a shallow (<1 m) soil profile, overlying intensively folded and faulted sedimentary rocks. The vadose zone reaches a maximum thickness of ~20 m, locally decreasing to <1 m (following topographic trends). Mean annual precipitation is on the order of 1300 mm/y; episodic, high intensity recharge events (often related to precipitation events) characterize the region. Saprolite, a highly weathered shale that contains lenses of clay, comprises a 0.5-3 m zone between the soil horizon and underlying (unweathered) shale (see the figure). The original structure of the parent material (e.g. lamination, bedding planes) has been well-preserved in the saprolite. Many of the bedding planes, fractures, and other macroporous surfaces are coated with secondary Fe- and Mn-oxides and translocated clay minerals. The MBSW has been the subject of experimental and theoretical research focused on preferential flow and solute transport in the vadose and shallow groundwater zone at a variety of scales. Careful measurements of physical transport parameters (e.g. hydraulic conductivity, porosity) have been made at both the laboratory and field scales.

Hydrogeochemistry Modeling of the Melton Branch Site

Flow field calculation of the MBSW site was performed on the Intel Paragons at ORNL, using the variably saturated groundwater flow code PFEM which was the parallel version of 3DFEMWATER developed by G.-T. Yeh at the Pennsylvania State University. These velocity fields became one of the input data used by the parallel code, PHGC3D, for hydrogeochemistry calculation on the Paragons. An uranium mill tailing problem with 56 chemical species and a leaking source on the southeast corner of the site was simulated using a computational grid of 10500 finite elements. The concentration iosurfaces of four chemical components, carbonate, iron, proton, and uranium, at approximately 50 days after the release of the chemicals are shown here.

Code Performance

The computation times of an 8 time-step simulation, including chemistry equilibrium, I/O and bi-conjugate gradient solve, are reported here. The code performance peaked at 65 nodes for this particular problem. The chemistry equilibrium module scaled linearly as expected. The degradation of the bi-conjugate gradient solver incurred by communication overhead indicates that there is room for further improvement. However, the major target of the next phase code optimization will be the disk I/O.


An application of a parallel supercomputer hydrogeochemistry code, PHGC3D, to a proposed waste disposal site at the Oak Ridge National Laboratory is used to demonstrate the possibility of vertically integrating the modeling of microscopic and macroscopic processes in a hydrogeological system. With the potential of recent development in high performance computation, we are looking at tremendous amount of productivity gains in terms of facilitating the decision making, planning, and managing of our precious water resources.


This work was funded in part by the ORNL Partnership in Computational Sciences (PICS) program, supported by the Department of Energy's Mathematical, Information, and Computational Sciences (MICS) Division of the Office of Computational and Technology Research.

*Managed by Lockheed Martin Energy Research Corp. for the U.S. Department of Energy under contract number DE-AC05-96OR22464.