Robert E. Gillen

I’m sitting at the airport in New Orleans, after having attended the first half of the ACM/IEEE 2010 Super Computing conference. This was the first time I have attended this conference, and it was certainly interesting to participate.

During the workshop I participated in on Sunday (Petascale Data Analytics on Clouds: Trends, Challenges, and Opportunities), there arose a conversation regarding the Amazon EC2 “cluster compute instances” and their having reached a spot on the Top 500 list. What surprised me, however, was not that they were mentioned (I actually expected them to receive more attention than they did), but that they were described as not being “real” cloud computing. The point was made that they represented some sort of special configuration that was done just for the tests and that the offering was somehow significantly different than the rest of the general populous could acquire. The two primary individuals involved in the exchange have significant history in classic HPC and have, at least a degree of “anti-cloud” bias, but I am responsible for helping influence the viewpoint of one of these folks so I’ve been thinking a bit over the past few days about how to properly articulate the inaccuracies of the argument… and wondering if it really matters anyway.

Recently I've been working on a desktop application named BiLab. BiLab is an interactive workbench for computational biologists similar in some ways to MatLab but focused (not surprisingly) on the computational biologist or molecular biologist. The goal is to tie together a number of existing and proven tools into a single interface allowing the user to interact with numerous tools from one application. Additionally, the tool includes interpreters which allow the output from one tool to be used as input to the next, saving the user the headache of manipulating file formats.

STAHC is a rather simple tool that allows applications that are not HPC aware but have problem sets that can be data parallelized to benefit from the computational power available in the cloud. The user of the tool completes a manifest file which is then fed into STAHC. STAHC packages up the source application and input data, deploys it to a cloud computing platform, calls the provisiong APIs of that platform to deploy the application and input data to the specified number of nodes, executes the computation, aggregates the output data and downloads it after which the cloud platform is torn down. All of this occurs without the domain scientist having to adjust or port his code to any cloud specific interfaces.

One issue of significance in cloud computing is that of exposing large datasets in a cloud friendly fashion. Exposed Walrus is a set of tools that extends the existing Walrus platform (S3-compatible storage implementation in the open source Eucalyptus platform) and allows an organization to publish large collections of data with an S3-compatible interface in situ. So long as the original data store is visible from the Walrus server, these tools allow that data to be proxied through and published to consumers as if it had been directly loaded into the Walrus platform.

Developing for the cloud often introduces new challenges and one of them is overcoming (or becoming comfortable with) the opacity of the machines on which one's code is running. The Window Azure GAC Viewer is a utility made available to the community that allows them to view a number of screens giving them low-level details about a typical machine running in Windows Azure. Additionally, a user can have the tool screen his project file to verify that he has the appropriate project references in place.

Occasionally our email system strips large or otherwise suspicious file attachements. If you would like to send me a large file or anything else that seems to have trouble getting through email, please use our file upload system.