Probe: Final Probe Projects Archival

The Probe project was closed out on September 30, 2003, with the following projects still active. Most of them are still active under other funding arrangements.

Projects as of September 30, 2003

• Research into distributed data cluster analysis

  As part of our work in DOE’s Probe project, we have developed communication-inexpensive methods (RACHET) that will enable scientists to perform cluster analysis and dimension reduction of distributed data on a computational grid or Internet. Cluster analysis and dimension reduction are fundamental to discovery and visualization of structure in high-dimensional data. These computationally demanding methods are used across many data-intensive applications ranging from astrophysics and climate simulations to neutron physics experiments and biological databases. To analyze these simulated or collected data, researchers previously required transferring such large amounts of data to a very powerful central computer. For massive distributed data sets, this approach is either impossible or impractical. The central idea behind the distributed methods of RACHET is that a software code not the data is moved to a remote host that is close to the data. Unlike parallel methods, the code performs local analyses on local data without any communication. The code transfers only minimal summary information to a merger site (e.g., desktop) where these summaries are combined into a global analysis. These distributed methods are being developed for a variety of well-known data analysis techniques.

• New linux nodes

  We have acquired four "new" systems - four of the 64 dual Pentium-III nodes that comprised the "HiTORQ" cluster supercomputer. We expect to install Redhat linux 7.2 on the nodes and link them via 100 megabit Ethernet to the network. We intend to use two of the nodes to investigate the "DataSpace" project in collaboration with Dr. Robert Grossman of the University of Illinois at Chicago. The other two nodes will be used in work on the Hierarchical Resource Manager (HRM) work at Lawrence Berkeley Laboratory; HRM is an element of two SciDAC projects. The nodes will also be useful in distributed cluster analysis activities.

• HPSS front end

  Users of supercomputers often wish to send large files to remote HPSS installations. They could invoke the hsi application on the supercomputer to do the transfer, but because of wide-area network latencies and bandwidth the transfers will take a long time, tying up supercomputer resources and researcher time.

  To avoid tying up the full resources associated with the job and the researcher's time, we are developing a "front end buffer". Users will be able to invoke a command that will queue a transfer, then exit. The transfer will take place later (fifo), freeing the user, and consume far fewer resources. It will be of value when HPSS is unavailable; the supercomputer job can continue without encountering an HPSS transfer error.

• Grid

  Two Grid projects have entered ORNL's Probe installation. One project is the DOE Science Grid; the other is Earth Systems Grid II. As part of that work, Globus II is being installed and tested on an ORNL IBM RS/6000 44P-170 ("sneezy") and a Sun E250 ("sleepy"). Probe staff is involved to the degree that system administration is needed, but other staff (both in Oak Ridge and elsewhere) are doing the investigations.

• GridFTP

  The HPSS parallel ftp application and the GridFTP application have strong similarities, but they are not identical. There is considerable interest in both communities for compatibility between the codes and discussions about what should be done, and by whom.

  Both Probe sites are interested in working on the project. Both sites are HPSS developers, and so have access to the HPSS code, and both sites have involvement in Grid activities. Our role is to be determined.

• NISTNET

  Nistnet is a WAN emulation package run on a "router" in a virtual network. To establish it, we first set up a virtual network involving two linux nodes, configuring one as a router. We verified initial operation by including an old laptop in the mini-network. We then sent packets from one linux node through the router linux node to the laptop using nistnet to emulate a wide-area transfer. We've now replaced the laptop with another linux box.

  Having established this emulator, we next will investigate "dynamic right-sizing" as described by Wu Feng.