Computational Earth Sciences research at ORNL encompasses many important aspects of global and regional Earth system model development and analysis. We focus on numerical methods development and implementation, data analytics, verification and validation of Earth system components, and the development of methods to characterize stochastic behavior. Significant progress in the areas of scalable time stepping algorithms, utilization of hybrid architectures to enable efficient and effective use of leadership class computing architectures, and algorithms to transport of large sets of aerosol and chemical species is underway. Data analytics research includes involvement with an early warning detection system to monitor forest health, development of clustering algorithms to identify geographic ecoregions, and coordinated efforts to link observational networks with simulation through model-informed data collection. Global Earth system model evaluation is becoming more crucial as models become increasingly complex, and advancements in tools and methods for component, fully coupled, and intermodel comparison are underway. Moving forward, Earth system models that imbed a stochastic representation of variable Earth system behavior such as cloud physics are of interest, and increasingly, Earth science research at ORNL is addressing this for the model representation of features, sensitivity analysis, and uncertainty quantification. Computational Earth sciences research at ORNL is driven by the large scale science questions that scalable, efficient, and validated simulation efforts can address, and recent high-impact experiments to characterize and compare regional models, intermodel comparisons, and the investigation of aerosol sensitivities are some fruits of these model development foci. See the project summaries below for details of this research.
Climate Science for a Sustainable Energy Future (CSSEF) (contacts Forrest Hoffmann/Richard Mills)
The Climate Science for a Sustainable Energy Future (CSSEF) is a collaborative project among Oak Ridge, Argonne, Brookhaven, Lawrence Berkeley, Lawrence Livermore, Pacific Northwest, and Sandia national laboratories together with the National Center for Atmospheric Research to transform the climate model development and testing process and thereby accelerate the development of the Community Earth System Model's sixth-generation version, CESM3, scheduled to be released for predictive simulation in the 5 to 10 year time frame. Four research themes are addressed in the project:
- a focused effort for converting observational data sets into specialized, multi- variable data sets for model testing and improvement,
- development of model development testbeds in which model components and sub-models can be rapidly prototyped and evaluated,
- research to enhance numerical methods and computational science research focused on enabling climate models that use future computing architectures, and
- research to enhance efforts in uncertainty quantification for climate model simulations and predictions.
These four themes are mutually reinforcing and tightly coupled around three overarching research directions:
- the development, implementation, and testing of variable-resolution methodologies that enable computationally efficient simulation of the climate system at regional scales,
- improvement of the representation of the hydrological cycle and quantification of the sources of certainty in its simulation, and
- the reduction and quantification of uncertainties in carbon cycle and other biogeochemical feedbacks in the terrestrial ecosystem.
The CSSEF will be structured to first deploy expertise in the research theme areas across the development of the atmosphere, ocean, sea ice, and land surface model components, and later to the fully coupled system. The CSSEF project addresses the DOE Office of Biological and Environmental Research's Long-Term Measure to "Deliver improved scientific data and models about the potential response of the Earth's climate and terrestrial biosphere to increased greenhouse gas levels for policy makers to determine safe levels of greenhouse gases in the atmosphere."
DataONE (contact John Cobb)
DataONE is a cyber repository that provides universal access to data about life on earth and the environment that sustains it. DataONE supports environmental science by: (1) engaging the relevant science, data, and policy communities; (2) providing easy, secure, and persistent storage of data; and (3) disseminating integrated and user-friendly tools for data discovery, analysis, visualization, and decision-making.
The foundation for success with DataONE is the established partnerships among participating organizations that have decades-long expertise in a wide range of fields that includes: existing archive initiatives, libraries, environmental observing systems and research networks, data and information management, science synthesis centers, and professional societies. DataONE engages its community of partners through working groups focused on identifying, describing, and implementing the DataONE cyber-infrastructure, governance, and sustainability models. These working groups, which consist of a diverse group of graduate students, educators, government and industry representatives, and leading computer, information, and library scientists will: (1) perform cutting edge computer science, informatics, and social science research related to all stages of the data life cycle; (2) develop DataONE interfaces and prototypes; (3) adopt/adapt interoperability standards; (4) create value-added technologies (e.g., semantic mediation, scientific workflow, and visualization) that facilitate data integration, analysis, and understanding; (5) address socio-cultural barriers to sustainable data preservation and data sharing; and (6) promote the adoption of best practices for managing the full data life cycle.
The defining purpose of DataONE is to enable discovery and universal access to data about life on Earth from around the world through DataONE.org. DataONE achieves this vision by providing transformational tools that shape scientific understanding of Earth processes from local to global scales; offering researchers education and training in various domains to enhance scientific enquiry; combining expertise and resources across diverse communities to collectively educate, advocate, and support trustworthy stewardship of scientific data; and presenting incentives and infrastructure for sharing data from federally funded researchers in academia. These objectives work in tandem to strengthen environmental research with an end goal of enabling discoveries that transform our understanding of ecological processes and conserve life on earth and the environment that sustains it.
DataONE is currently developing infrastructure to provide support for the entire data life cycle, from obtaining observations, compiling documentation and metadata (information about the data product) that will help users understand the data, and other tools and services to allow users to explore the data holdings, access relevant data, and perform some analysis and visualization. The infrastructure is slated for release in December 2011.
DataONE has established a number of Working Groups in Community Engagement and in CyberInfrastructure to engage the international research community in identifying the best solutions for managing the data life cycle. One of the first DataONE working groups is the Exploration, Visualization, and Analysis (EVA) group that examines data intensive science. Steve Kelling (Cornell Lab of Ornithology) and Bob Cook (ESD/ORNL) are the co-chairs of the EVA Working. The first exemplar dealt with the environmental factors that affect migratory bird distributions in the conterminous US (http://ebird.org/content/ebird/news/ebird-animated-occurrence-maps) . The methods used in exploring, visualizing and analyzing the bird observation and environmental data will inform the development of DataONE tools and services. The next exemplar for the EVA Working Group will likely be associated with evaluation of carbon cycle models using a wide range of observations.
NGEE-Arctic (contact Richard Mills)
"...improving climate model predictions through advanced understanding of coupled processes in Arctic terrestrial ecosystems."
Increasing our confidence in climate projections for high-latitude regions of the world will require a coordinated set of investigations that target improved process understanding and model representation of important ecosystem-climate feedbacks. The Next-Generation Ecosystem Experiments (NGEE Arctic) seeks to address this challenge by quantifying the physical, chemical, and biological behavior of terrestrial ecosystems in Alaska. Initial research will focus on the highly dynamic landscapes of the North Slope (Barrow, Alaska) where thaw lakes, drained thaw lake basins, and ice-rich polygonal ground offer distinct land units for investigation and modeling. A focus on scaling based on investigations within these geomorphological units will allow us to deliver a process-rich ecosystem model, extending from bedrock to the top of the vegetative canopy, in which the evolution of Arctic ecosystems in a changing climate can be modeled at the scale of a high resolution Earth System Model grid cell (i.e., 30x30 km grid size). This vision includes mechanistic studies in the field and in the laboratory; modeling of critical and interrelated water, nitrogen, carbon, and energy dynamics; and characterization of important interactions from molecular to landscape scales that drive feedbacks to the climate system. A suite of fine-, intermediate-, and climate-scale models will be used to guide observations and interpret data, while process studies will serve to initialize state variables in models, provide new algorithms and process parameterizations, and evaluate model performance.
Research sponsored by the Office of Biological and Environmental Research within the U.S. Department of Energy's Office of Science. The NGEE project is a collaboration among scientists and engineers at Oak Ridge National Laboratory, Los Alamos National Laboratory, Brookhaven National Laboratory, Lawrence Berkeley National Laboratory, University of Alaska Fairbanks and our partners at leading universities and other state and federal agencies. ORNL is managed by UT-Battelle, LLC.
Biogeochemistry (contact Forrest Hoffman)
The Applying Computationally Efficient Schemes for BioGeochemical Cycles (ACES4BGC) Project seeks to advance the predictive capabilities of Earth System Models (ESMs) by reducing two of the largest sources of uncertainty, aerosols and biospheric feedbacks, with a highly efficient computational approach. In particular, this project will implement and optimize new computationally efficient tracer advection algorithms for large numbers of tracer species; add important biogeochemical interactions between the atmosphere, land, and ocean models; and apply uncertainty quantification (UQ) techniques to constrain process parameters and evaluate uncertainties in feedbacks between biogeochemical cycles and the climate system. The resulting improvements to the Community Earth System Model (CESM) will deliver new scientific capabilities and significantly improve the representation of biogeochemical interactions at the canopy-to-atmosphere, rivers-to-coastal oceans, and open oceans-to-atmosphere interfaces. ACES4BGC partners modelers with decades of cumulative research experience and a team of computer and computational scientist building scalable solvers and tools, developing advanced UQ methods, and applying technologies for performance optimization through U.S. DOE SciDAC Institutes. The project began April 15, 2012.
Performance Engineering of the Community Climate System Model (PECCSM) (contact Patrick Worley)
"Performance Engineering of the Community Climate System Model" (PECCSM) was a project associated with the SciDAC2 Science Application project "A Scalable and Extensible Earth System Model for Climate Change Science" (SEESM) during its final two years. There was a predecessor Science Application Partnership (SAP) project called "Performance Engineering for the Next Generation Community Climate System Model" (PENG) that addressed performance engineering issues during the first three years of SEESM. You may know the history of SAPs in SciDAC2 better than I, but the SAP program wound down for the most part after the first three years of SciDAC2. The mid-project review of SEESM recognized important contributions by PENG and encouraged its continuation, if at all possible, because of the continued importance of performance engineering to SEESM. Based on this, the PECCSM proposal was developed and, subsequently, funded.
Both PENG and PECCSM had well-defined research plans and statements of work, but ultimately both were intended to be flexible enough to address performance issues that arose within SEESM. I believe that we were reasonably successful in this.
The goals of PENG were to identify and address structural impediments to performance and performance scalability in the Community Climate System Model (CCSM), especially with regards to "next generation" problem scenarios and HPC systems. At the time, this led naturally to a focus on the performance of the atmosphere, to helping port the model to and optimize performance on the IBM BG/L and BG/P and the Cray XT4 and XT5, and to helping with the development of a new parallel I/O layer (PIO).
During this period there were a number of new numerical methods being developed for the atmosphere dynamics, because the existing algorithm (finite volume based method, FV, on a longitude-latitude grid) was not viewed as being sufficiently scalable for next generation architectures and problem sizes. We felt that it was important to establish exactly what the performance capabilities of the existing algorithm were, both to provide a realistic baseline for determining whether the new methods were in fact superior and because the existing FV algorithm would be used for both production and other aspects of model development for at least the duration of SEESM. (The latter turned out to be true. The transition to a new atmosphere dynamics algorithm is just now taking place.) The FV performance optimization work and dynamics algorithm performance comparisons began in PENG, and were finished in PECCSM. The attempt to extend the performance and performance scalability of the FV algorithm led to the introduction of a number of scalability enhancements that had general applicability. The work also helped established that at least one of the new algorithms had superior performance at scale.
Building on the PENG work, PECCSM was able to focus more on the performance of the full coupled model, still in the context of large problem sizes and large scale parallelism. Due to changes in the ocean grid and in the target platforms, performance of the ocean component went from scaling very well to scaling relatively poorly in high resolution configurations, and so ocean (and sea ice) components became important targets for performance analysis and optimization. As part of this, significant effort went into node level performance optimization, including introducing OpenMP parallelism where it did not yet exist and further optimizing it where it did. (This also increased exploitable parallelism and/or decreased MPI overhead, so was more than just an optimization of node performance.) With the imminent public release of the new model and its use in AR5, it was also critical to determine how best to configure the model for performance (there being many options) for production scenarios and what performance to expect on target platforms. Finally, work began on further analyzing and optimizing performance of the expected successor to the FV dynamics algorithm, a spectral element method on a cubed sphere grid.
Multi Scale (contact Kate Evans)
Multiscale Methods for Accurate, Efficient, and Scale-Aware Models of the Earth System
The DOE BER SciDAC funded Multiscale project is tasked to improve global Earth system models by addressing the representation of multiscale interactions among small-scale features and large-scale structures of the ocean and atmosphere. The primary goal is to produce better models for these critical processes and constituents from ocean-eddy and cloud-system to global scales through improved physical and computational implementations.
The ORNL contribution to this project is to address the time stepping methods used to integrate the model across the multiple ranges of space and time scales in the atmosphere. These include improvements to the integration of the large-scale dynamics through non-CFL limited time-stepping methods and their extension to improve the methods used to integrate the diagnostic cloud physics packages within the model. These packages are concurrently being redesigned to allow scale-aware behavior as the model is refined globally and regionally. As part of these efforts, research into scalable preconditioners and the optimized use of generic solver libraries will be key focus areas.
OLCF CAAR project: acceleration of CAM-SE using GPU
Future success in climate simulation requires not only added spatial resolution but improvement in the fidelity of physical processes modeled, which in turn increases the number of processes to be modeled, such as chemical tracer quantities to be transported. Since transport is the dominant computational cost for added tracers, OLCF's CAAR effort has ported the tracer transport routines to utilize GPUs. This port has been performed using PGI's CUDA FORTRAN language. Memory optimizations and overlapping of PCI-express transfers, MPI transfers, and message packing and unpacking routines have led to over 2.5x reduction in full CAM-SE runtime compared to the CPU-only code run on an XE6 with a 14km science problem with 100 tracers. Currently, further changes to the codebase are being re-ported into the tracer routines on GPUs, and an effort is underway to translate the CUDA FORTRAN code into a more readable, portable, and maintainable OpenACC port.
Ultra High Resolution Global Climate Simulation (contact Kate Evans)
The problem of predicting climate change and its consequences is motivated by the increasingly urgent need to adapt to near term trends in climate change and the potential changes in the frequency and intensity of extreme events. This project is developing the scientific framework to determine the benefit of employing very-high-resolution global models to investigate regional-scale phenomena. The team will test the hypothesis that higher resolution models are necessary to accomplish the related objectives of 1) the explicit simulation of non-linear phenomena and interactions on the small scale that have feedbacks on large scale climate features; and 2) the accurate and explicit simulation of local to regional scale phenomena, including low-probability, high-impact hydrological events. The focus will be sarigorous evaluation of our hypothesis with high-resolution simulations of observed climate and variability will be the focus. Using a series of stand-alone component and ensemble coupled present-day climate simulations, the project partners will perform comparisons of high- and low-resolution model configurations to determine the potential advantages of high resolution simulations. They will investigate the role of air-sea interaction, the quality of the basic state, and resolution sensitivity on a hierarchy of modes of variability, including the distribution and evolution of extreme events in control, historical, and time-slice experiments of future climate change. An integral component of this project will be the development of new objective diagnostics and metrics to gauge the potential benefit of employing high resolution to improve the representation of regional scale phenomena, especially those related to the hydrological cycle.
The pathway to the project's goals requires an improved understanding of critical sensitivities in model formulation that have been identified from preliminary very-high resolution coupled simulations. Our experimental protocol will enable the project team to investigate these sensitivities, which include the interaction of physics with the choice of dynamical core of the atmospheric model and the initialization of the ocean model, both of which contribute to the development of model biases relative to observations. Implicit on this pathway toward developing a more realistic very-high resolution coupled model for regional projections of climate change will be the exploration of the benefit of very-high resolution contributed by the atmospheric and ocean models, respectively.
Developing a coupled ocean-atmosphere model provides a great challenge to modelers. This challenge is amplified when development is performed at very-high resolution, as scales of variability are resolved that heretofore have not been simulated in coupled mode, and due to the extreme computing that is required to produce such simulations. The climates of these very-high resolution runs each appear relatively stable. Thus, this project will investigate the sources of bias that complicate the simulation of the coupled system, including ocean initialization, parameterized physics and its interaction with the dynamical core. Additionally, the relative influence of high resolution from the atmospheric and ocean models will be explored through combinations of uncoupled and coupled runs in which the resolution of either one or both of the model components is decreased. The enhanced understanding and improved performance that we gain from the initial phase of experimentation will enable the project team to use the very-high resolution coupled model with increased confidence for the investigation of natural variability and anthropogenic forced perturbations in experiments run as part of the World Climate Research Programme's Working Group on Coupled Models Fifth Coupled Model Intercomparison Project (CMIP5) protocol.
PISCEES: Predicting Ice Sheet and Climate Evolution at Extreme Scales (contact Kate Evans)
The DOE BER SciDAC funded PICSEES project, led from Los Alamos National Laboratory, is tasked with developing and applying robust, accurate, and scalable dynamical cores (“dycores”) for ice sheet modeling on structured and unstructured meshes with adaptive refinements, evaluating ice sheet models using new tools and data sets for verification and validation (V&V) and uncertainty quantification (UQ), and integrating these models and tools in the Community Ice Sheet Model (CISM) and Community Earth System Model (CESM). Mass loss from the Greenland and Antarctic ice sheets is accelerating, and although ice sheet models have improved in recent years, much work is needed to make these models reliable and efficient on continental scales and to quantify their uncertainties.
The ORNL contribution to this project is to develop tools and frameworks for V&V and UQ. We will create verification test suites consisting of analytical and manufactured solutions, and we will compile the best available data sets for model validation. These tools will be assembled in a post-processing package that can be used to formally evaluate CISM in standalone runs and as a CESM component. Observational data sets are providing a target for initialization of the coupled model. In the next we will use the evaluation framework to spin up ice sheets to a state that closely resembles present conditions, while remaining in near balance with CESM surface forcing. We are providing a coherent structure for ongoing collaboration among glaciologists, climate modelers, and computational scientists and work closely with the SciDAC institutes (FASTMath, QUEST, and SUPER) and the community of CISM and CESM developers. All source code for the Land Ice Verification and Validation framework can be found within the CISM repo, which can be accessed from the CESM 1.2 repository: http://www.cesm.ucar.edu/models/cesm1.1/.
Stochastic Parameterization of the Influence of Sub-grid Scale Land Heterogeneity on Convection in a Climate Model (Contact: Salil Mahajan)
This project is investigating stochastic parameterization methodologies that capture the sub-grid scale heat flux variability associated with land surface heterogeneity in the Community Earth System Model (CESM). Currently, latent heat fluxes in CESM are computed as an average over various plant functional types (PFTs) within the grid box. This approach at best approximates the bulk effect of the latent heat flux. The missing sub-grid scale variability in the surface fluxes to the atmosphere in the model could potentially impact local convective processes limiting the variability of simulated precipitation. We are currently investigating the effect of the addition of a stochastic term that approximates the spatial sub-grid scale variability within a model grid-box in the heat flux computations in a single column version of the model forced with prescribed dynamical constraints. The project is tasked with the development of a prototype robust workflow to support the automatic initialization and deployment of single column land atmosphere ensembles based on Uncertainty Quantification (UQ) parametric sampling strategies, and a semi automated sufficiency analysis of ensemble results based on adaptive stochastic sampling strategies to define and initiate supplementary ensemble members. The ultimate goal of the project is to set up the global model with a stochastic forcing capability for the land fluxes to the atmosphere and perform a limited series of simulations to determine the global impact of the stochastic forcing on climate simulations.
This project is funded by the laboratory under the Laboratory Directed Research and Development (LDRD) program.
Impacts of Aerosols and Air-Sea Interaction on Community Earth System Model (CESM) Biases in the Western Pacific Warm Pool Region (Contact: Salil Mahajan)
The realistic simulation of clouds and precipitation over the Tropical Western Pacific warm pool (WPWP) region remains a challenge in this new era of higher resolution climate models (horizontal spatial resolution of 1° and higher). Coupled model simulations typically show biases in the zonal sea surface temperature (SST) gradient as well as the meridional SST gradient over the equatorial/tropical Pacific ocean, both of which might lead to biases in the SST-sensitive tropical atmosphere via air-sea coupled processes. However, atmosphere-only simulations with prescribed observed SST also suffer from negative biases in total cloud amounts and biases in the location and intensity of the inter-tropical convergence zone (ITCZ). DOE's ARM sites over the tropical Western Pacific (TWP) region provide vast continuous multivariate observations of meteorological conditions since 1996 over the region - including those related to tropospheric aerosols, clouds, surface radiation and precipitation - presenting an opportunity to carefully analyze atmospheric and air-sea coupled processes and improve their representation in climate models.
Contrary to assumptions in most of the IPCC AR4 models, the relationship between aerosols and clouds/precipitation is non-monotonic, because of the simultaneous direct, semi-direct and indirect effects of aerosols on the surface, clouds and precipitation. The microphysical interactions between aerosols and clouds have only recently been incorporated into global climate model (GCM) simulations. It is important to understand these simulated interactions and compare them with observations to improve the representation of climate processes in GCMs and hence improve climate predictions.
The WPWP regions bears an influx of aerosols from Southeast Asia along with local sources from the Western Pacific islands and can play an important role in the regional climate. Furthermore, the WPWP region exhibits strong air-sea coupling, and any study of the origin of regional biases needs to take these interactions into account.
We propose to conduct a hierarchical coupled modeling study using a suite of experiments conducted with a high-resolution configuration of CESM1.0 to understand the impact of major aerosol species on the WPWP region and compare model output with observations from ARM surface instruments as well as multi-sensor satellite observations. In addition to the coupled CESM integrations, we also intend to carry out additional mechanistic integrations using the uncoupled CAM5 atmospheric model, intermediate coupled models where CAM5 is coupled to simplified ocean models and the single column atmospheric model (SCAM) configuration in CESM1.0. In particular, we will couple CAM5 to a slab ocean model (SOM) and a reduced-gravity ocean model (RGOM) that can be configured to simulate both the mean state as well as important features of tropical variability such as the El Nino-Southern Oscillation (ENSO). This hierarchy of mechanistic integrations will facilitate the understanding of the interaction between aerosols and the atmospheric column, as well as the interaction between the atmospheric column and the underlying ocean, in determining the tropical climate state. Insights gained from analyzing these integrations can be used to improve model parameterization of aerosol effects, and thus help alleviate model biases associated with clouds and precipitation.
This project is funded by the US DOE Office of Science, Biological and Environmental Research (BER) program.