CUMULVS: Collaborative User Migration, User Library for Visualization and Steering

DOE 2000 - ACTS Toolkit Research

FY 1998 Highlights


CUMULVS is an ongoing project at Oak Ridge National Laboratory (ORNL) as part of the DOE 2000 Advanced Computational Testing and Simulation (ACTS) Toolkit. The CUMULVS work focuses on the integration of collaborative computational steering tools, interactive visualization tools, and checkpointing / fault tolerance tools into runtime systems, frameworks and applications.

The ACTS Toolkit

The ACTS Toolkit has the goal to provide an integrated set of software tools, algorithms, and environments that accelerate the adoption and use of advanced computing by DOE programs. The long-term goal of this work is to bring together the major developers and users of high-performance software tools and numerical software to develop and implement a common tool integration interface. The project consists of three distinct and complementary parts: software development, numerical kernels, and runtime support. This work is a collaborative effort involving Oak Ridge National Laboratory (ORNL), Los Alamos National Laboratory (LANL), Lawrence Livermore National Laboratory (LLNL), Sandia National Laboratories (SNL), Lawrence Berkeley National Laboratory (LBL), Argonne National Laboratory (ANL), and several university partners.

Several ACTS tools from the DOE laboratories have already been integrated and made interoperable to share portions of their functionality. A more general integration solution is being designed via the Common Component Architecture (CCA) for specifying how high performance software components interface and attach to each other across DOE (see below).


This report includes:
CUMULVS Approach
FY 1998 Accomplishments
Future Plans
Downloading and Using CUMULVS
Development Team Participants

CUMULVS Approach

CUMULVS is a software infrastructure for the development of collaborative environments. Developers of high-performance scientific software can utilize CUMULVS to dynamically attach front-end viewer programs to a running simulation and interact with it on-the-fly. CUMULVS supports interactive run-time visualization and remote computational steering of distributed applications by multiple collaborators, and provides a mechanism for constructing fault-tolerant, migrating applications in heterogeneous distributed computing environments. CUMULVS is also being applied to the coupling of models for hybrid or cooperative simulations, and can assist in the sharing and interpolation of data fields among disparate models.

The architecture of CUMULVS is described in the figure below:

CUMULVS is the infrastructure "glue" that allows independent front-end viewer programs to attach to the tasks that make up a distributed application. The application tasks can be running on any combination of Unix and Windows machines, and the viewer programs can run locally with the application tasks or remotely over a network. There can be any number of different viewers attached at any one time, each potentially viewing its own sub-region of a different data array. The viewer programs can use any graphical system for rendering data field views, including AVS, Tcl/Tk, a virtual reality interface, or some application-specific custom GUI constructed by the user.

The CUMULVS system exists in three distinct pieces, as an application-side library, a viewer-side library, and a separate fault recovery daemon (not shown). By compiling the appropriate CUMULVS libraries into the application tasks and viewer programs, CUMULVS can transparently attach and detach viewers for run-time visualization and computational steering. Once the application program has been instrumented to identify and describe any data fields or parameters of interest, the CUMULVS library can automatically extract any desired data as needed. The application-side and the viewer-side libraries communicate by invoking the necessary protocols to pass information back and forth for viewing or steering, under the control of the simple, high-level library interface at the viewer.

Another way to consider the CUMULVS viewer interaction is in terms of a distributed data array, as shown in the above example. A very large array can be decomposed among some number of application tasks. CUMULVS understands these decompositions, and so each viewer program can extract its desired array elements from the tasks where those elements are stored. The viewers provide access to the data array in global coordinates, as if it were a single, contiguous array.

Multiple, remote collaborators can all attach to the same application and coordinate its control cooperatively. Each scientist can simultaneously "steer" a different parameter. These parameters can be scientific in nature, controlling physical aspects of the experiment, or can be algorithmic parameters that control the numerics of the computation itself. Such interactive control over an application can be used to explore "What If?" analyses, even for non-physical effects that could not be duplicated in a physical experiment. In addition, steering can be used to keep a simulation on track by making minor adjustments while it is running. This type of control also produces a more efficient experimentation cycle by cropping off experiments gone awry. This is especially useful for long-running software experiments.

As a means for application-directed fault tolerance, CUMULVS provides a heterogeneous checkpointing facility. Using the same infrastructure that allows an application to declare data arrays for visualization and parameters for steering, the application can mark the variables that contribute to the minimal program state. CUMULVS can then automatically extract these variables and save them in a checkpoint file for restart or migration. CUMULVS handles the collection and organization of the checkpoint data, and provides an automatic run-time fault recovery system.

Because the application describes to CUMULVS (in the data array instrumentation) what the semantics are regarding the decomposition of it data arrays, CUMULVS can provide an efficient and flexible checkpointing mechanism. Rather than saving a full core image of the application, CUMULVS can save just the necessary variables, resulting in significantly smaller checkpoint files. The semantic information also allows CUMULVS to manipulate the checkpoint data for heterogeneous restart and migration, and can even reconfigure a checkpoint for restart using a different machine topology or decomposition type.

The CUMULVS run-time fault recovery system consists of one checkpointing daemon (CPD) per host, for collecting and providing checkpoints, and monitoring the application and system for failures. The CPDs also act as a "console" for manually controlling any restarts or migrations as desired. The CPDs themselves comprise a fault-tolerant application that can handle the failure of any subset of CPDs or hosts. The CPDs coordinate the redundancy of the checkpoint data among the different hosts, using a "ring" topology.

FY 1998 Accomplishments

- PVMPI (MPI_Connect) package created to allow vendors' optimized MPI implementations to interoperate.

- CUMULVS integrated with MPI to provide visualization, computational steering and fault tolerance for MPI applications.

- CUMULVS Checkpointing system prototype completed and released to the public as of July 1998.

- CUMULVS Particle-Based Viewer Programs developed as an extension to the contiguous data array viewers, to provide visualization support for particle-based applications such as Smooth Particle Hydrodynamics (SPH).

- Integrated CUMULVS with NERSC's Cray T3E to allow remote monitoring and visualization of T3E applications over a standard network.

- Collaborated with NCSA on the construction of collaborative, 3-D and Virtual Reality viewers for CUMULVS.

- Common Component Architecture (CCA) Forum was formed, with ORNL as a founding member.

The initial release of the PVMPI package was completed and installed for production use in the Department of Defense (DOD) Major Shared Resource Center (MSRC) sites at CEWES, ASC and ARL. The system supports transparent operation on top of multiple MPI implementations, or point-to-point communication among distinct MPI task groups. Public release of PVMPI will be made during the first quarter of 1999.

CUMULVS has now been extended to provide visualization, steering and checkpointing for MPI applications. Due to the static nature of MPI and its lack of support for persistent data and any fault tolerance features, CUMULVS uses PVM in conjunction with MPI to interface with MPI applications. Work is in progress to design a possible "reduced-functionality" version of CUMULVS that uses MPI exclusively (see future plans below).

The primary focus of CUMULVS development for FY 1998 was on completing the design and implementation of the checkpointing facility. The remaining functions required for a production checkpointing system were added to CUMULVS, and the run-time fault recovery system was thoroughly tested and made more robust. Several case studies were performed to pinpoint areas for performance enhancement and user interface improvements (see future plans below).

To support visualization of particle-based applications, as supported now via the generalized particle decomposition interface co-developed between ORNL and Sandia, several CUMULVS viewers have been extended to handle particle data. The AVS viewer module and Tcl/Tk slicer viewer now extract and display particle data, even in conjunction with other contiguous data fields.

In a collaboration among ORNL, LBL and LLNL, CUMULVS was ported to the Cray T3E system at NERSC to provide remote monitoring of T3E applications from a user's desktop. CUMULVS was integrated with the optimized Cray PVM implementation, and was utilized to coordinate and collect data from application tasks for run-time visualization. CUMULVS' capability to select subregions of data at varying degrees of data granularity allows efficient use of standard networks for remote monitoring.

In conjunction with the National Center for Supercomputing Applications (NCSA), ACTS funding was used to support the development of a wide variety of CUMULVS viewer programs and projects. A set of Tcl/Tk language bindings were created to allow direct access to CUMULVS library routines from Tcl/Tk applications. A coordinated viewer was developed for the Air Force Research Laboratory (ARFL), using VTK and Tango, to allow a team of collaborators in an interactive scenario to adjust their view regions in unison under the control of a central viewer. A 3-D virtual reality viewer for CUMULVS, constructed using OpenGL and the Cave libraries, was used to visualize a Chesapeake Bay simulation on the ImmersaDesk. Windows-based viewers using the Microsoft Foundation Classes (MFC) are under development. CUMULVS is being applied to a Neutron Star Collision simulation at the State University of New York (SUNY) and to several DOD codes at the Aeronautical Systems Center (ASC).

A new approach to tool integration was spawned during the FY 1998 ACTS work - the Common Component Architecture (CCA). Al Geist from ORNL is a founding member of the CCA forum. The CCA is a specification for how software components and tools define their interfaces and plug together. The CCA is a fundamentally different and more general approach to tool integration than manually interfacing individual pairs of tools. It centers on the concept of a "gPort" that specifies how data flows in and out of application and tool components. The CCA will significantly expedite the incorporation of CUMULVS fault tolerance into the PAWS Event and Control Monitor, and the direct instrumentation for CUMULVS of the data distribution components and numerical kernels in PADRE and InDEPS (POET). These tool integration tasks will be accomplished easily when CUMULVS is made CCA-compliant (see future plans below).

Future Plans

Our ongoing work on CUMULVS and the ACTS Toolkit continues to emphasize the integration of run-time visualization, coordinated computation steering and application fault tolerance capabilities into ACTS' runtime, numerics, and framework components. Our research will continue to expand the CUMULVS runtime environment to increase its applicability to a wider base of applications and tools.

Extend the CUMULVS Application Programming Interface (API) to include support for dynamic data redistribution as found in Adaptive Mesh Refinement (AMR) algorithms.

- CUMULVS Messaging Substrates:
Extend CUMULVS to communicate over alternate internal message-passing substrates (as used for attaching to application tasks), including Nexus messaging and possibly MPI.

- CUMULVS Checkpointing Optimization:
Extend and optimize the CUMULVS checkpointing system to support the parallel writing of checkpoint data and to provide adjustable levels of redundancy / fault tolerance.

Bring CUMULVS into compliance with the Common Component Architecture (CCA) specification, allowing CUMULVS to interoperate with other tools, such as Harness, PAWS, and InDEPS.

- CUMULVS Instrumentation:
Create a Problem Solving Environment (PSE) and pre-processor to assist in instrumenting applications for computational steering, interactive visualization and fault tolerance.

- Model Coupling Using CUMULVS:
The CUMULVS system is being applied to experiment with the coupling of data fields across disparate models, as used in hybrid or cooperative simulations.

The data decomposition interface in CUMULVS will be extended to encompass dynamic data redistribution for applications that use Adaptive Mesh Refinement (AMR) algorithms. This will require extensions to the base CUMULVS data collection facility, as well as new protocols to notify attached viewers of underlying mesh changes.

CUMULVS will be generalized to work with alternate message-passing substrates such as Nexus and possibly MPI. CUMULVS currently uses PVM for its internal messaging, process control, persistent data and fault notification, but can attach to applications using any other communication system as well. This simultaneous use of multiple communication systems is functional yet cumbersome. The Nexus communication system, part of the Globus work at Argonne National Laboratory (ANL), should provide sufficient functionality to support CUMULVS' internal communication needs without PVM. Porting CUMULVS to use Nexus would open the door to a large community of application developers. MPI, by itself, does not provide adequate system dynamics, process control, or fault notification for the full CUMULVS functionality. Therefore, for CUMULVS to function as a stand-alone MPI interface, a possibly "reduced-functionality" MPI implementation needs to be explored and/or defined.

To improve the usefulness and feasibility of using CUMULVS for application fault tolerance, the CUMULVS checkpointing facility will be optimized. Checkpointing overhead will be reduced by integrating parallel I/O technology, to more efficiently write checkpoint data to disk. The use of a parallel file system will be explored, along with other portable approaches for coordinating the simultaneous writing of each task's checkpoint data. The run-time fault recovery system will also be generalized to provide a spectrum of adjustable redundancy levels. This will control the checkpointing overhead by allowing an application to customize its desired fault recovery requirements.

Next, a more fundamental step towards interoperability will be taken to bring CUMULVS into compliance with the new Common Component Architecture (CCA) specification. The CCA will provide an interface for dynamically attaching to an application to share data and cooperate. Tools such as CUMULVS will use the CCA as a foundation for attaching to generic CCA-compliant applications for visualization and steering. The data decomposition interface that CUMULVS uses to extract data fields will be extended to accept CCA-compatible information from applications, to identify what data is available and how it is distributed among tasks. This will effectively make CUMULVS interoperable with any other potentially CCA-compliant tools, such as PAWS and InDEPS (POET). The CCA also provides a more general approach to plugging together components in the Harness environment, a project underway at ORNL, the University of Tennessee and Emory University. The CUMULVS capabilities will integrate more easily into Harness using the standardized specification layer provided by the CCA.

In addition to expanding the CUMULVS system to work with extant CCA-compatible applications, a Problem Solving Environment (PSE) will be designed to assist in instrumenting new or legacy applications for interfacing with CUMULVS or other CCA-compliant tools. This PSE will likely consist of a graphical user interface (GUI) and a source code pre-processor. The PSE will assist the applications programmer in identifying and describing any relevant data fields or steerable parameters in a simulation. The PSE will also expedite the instrumentation of applications for fault tolerance by providing a means for selecting the minimal program state to checkpoint and helping to insert checkpointing operations and requests.

As part of a national climate modeling effort, CUMULVS is being applied to experiment with the coupling of models. Two different models, e.g. an Ocean model and an Atmospheric model, might be coupled together to share and correlate a common data field, such as Temperature. This typically requires that CUMULVS extract the data from one model's decomposition and redistribute it according to the other model's task topology. As seen in the figure below, this is a natural extension to the fundamental CUMULVS viewer protocols, if the one-to-many attachment is enhanced to many-to-many. Additionally, however, many problems remain to be solved, such as generic approaches to interpolating data in time and space. This arises when two models do not share the same spatial grid, or if they compute on dramatically different simulation time scales.

Downloading and Using CUMULVS

To allow interaction at run-time, an application must be instrumented to identify and define any relevant data fields and parameters. This amounts to describing any distributed data decompositions, and providing the local data type and memory allocation of each array. Similarly, any parameters, which could be manipulated to steer an application or simulation, must be defined. A typical instrumentation requires tens of lines of code and usually only takes a few hours, depending on the user's knowledge of the given application code.

The latest version of the CUMULVS software can be downloaded from the CUMULVS Home Page. This software works on most Unix platforms, and there is also a preliminary distribution for Windows platforms.

Currently, CUMULVS is built on top of PVM, so you must download and install PVM to use it. CUMULVS works with either the PVM 3.3.11 or PVM 3.4.* releases.

The CUMULVS software comes with several pre-defined viewer programs, including an AVS viewer, a Tcl/Tk slicer, and a simple text-based viewer. To use the AVS viewer, you must first purchase AVS for your system. To use the Tcl/Tk viewer, you must download and install the Tcl/Tk system. CUMULVS will work with versions Tcl 7.4 / Tk 4.0 or later. Additional information on Tcl/Tk can be found at

Development Team Participants

Jim Kohl (ORNL)
Al Geist (ORNL)
Conrad Albrecht-Buehler (ORNL)
Dave Semeraro (NCSA)
Phil Papadopoulos (UCSD)
Rob Armstrong (Sandia)

For more details on CUMULVS or questions,

Distributed Computing Group
Computer Science and Mathematics Division
Oak Ridge National Laboratory
P.O. Box 2008, Bldg 6012, MS 6367
Oak Ridge, TN 37831-6367

Research supported by the Mathematics, Information and Computational Sciences Office, Office of Advanced Scientific Computing Research, Office of Energy Research, U. S. Department of Energy, under contract No. DE-AC05-96OR22464 with Lockheed Martin Energy Research Corporation.
Last modified: May 10, 1999 by Kohl.