Home > PC >

Originally appeared in March 11, 2005
HPCwire

Cray Makes Waves With X1E Upgrade

Cray has been making some big news this week. The provider of high-performance supercomputer systems recently appointed a new president, Peter Ungaro, and announced the availability of the Cray X1E supercomputer, a major upgrade to the company's record-setting Cray X1 scalable vector product. The new system nearly triples the peak performance and price performance of the presiding Cray X1 system that has set records on weather, engineering and scientific research applications.

Steve Scott, Cray's chief technology officer and chief architect of the Cray X1 series, recently spoke with HPCwire to provide more details about the announcement and future Cray plans.

=================================================

HPCwire: The Cray X1E supercomputer is a major upgrade to the Cray X1 product. What are the main differences?

Steve Scott: The primary difference is that by exploiting the next-generation integrated circuit technology, the Cray X1E system
nearly triples the peak performance and compute density of the Cray X1 product. We boosted the peak speed of each processor by almost 50% and placed two processors on each multi- chip module, instead of one in the Cray X1 system. The net result is a peak performance of 18 gigaflops per processor, which makes this the world's fastest processor. On a single eight-processor module, you have 144 peak gigaflops; and in the same physical cabinet, you go from 819 gigaflops with Cray X1 processors to 2.3 teraflops using Cray X1E processors.

These are substantial advances in computing power and compute density; and as you know, the more powerful the individual processors, the easier it is to scale up applications. The Cray X1E processors are extremely latency tolerant and are coupled with a very high bandwidth memory system and interconnect. The result is a system designed to scale efficiently on the most challenging class of problems.

HPCwire: The Cray X1E uses dual-core vector processors to gain compute density, and at Linuxworld last month, Cray became one of the first to demonstrate dual-core Opterons on the Cray XD1 supercomputer. What do you find to be the major advantages and disadvantages of dual core?

Scott: The biggest advantage of dual core in the Cray X1E system is doubling the compute density without a corresponding increase in complexity. There are also advantages for power consumption and clock speed, by exploiting locality on the silicon. This complements the already excellent power efficiency of the vector processors used in the system.

The greatest disadvantage of going dual core is that you reduce the ratio of memory bandwidth to peak computational speed. In that regard, it helps tremendously to start with strong bandwidth. We designed the Cray X1 and our Opteron-based products, the Cray XD1 and Cray XT3, with dual core in mind. Let me give you an example of what this means. In a typical single-core cluster, you have four microprocessors sharing about six gigabytes per second of bandwidth on a bus. With dual core on that same cluster, you now have eight processors sharing the original bandwidth, which means you have less than one gigabyte per second of bandwidth per processor. Contrast that with a Cray X1E dual core system, where two processors share 35 gigabytes of bandwidth, giving each of them more than 17 gigabytes. This has major implications for the kinds of problems you can tackle and scale up efficiently in practice.

HPCwire: On that topic, how has the predecessor Cray X1 system performed in practice?

Scott: The Cray X1 system has produced record results on really large, difficult scientific and engineering applications. Army High Performance Computing Research Center researchers ran a challenging CFD code at more than one sustained teraflop on only 256 processors. They used the MM5 model to complete a 24-hour weather forecast for the continental U.S. at five-kilometer resolution in a very short time. Five-kilometer resolution takes eight times the computing power of the 10-kilometer grid spacing that's typically been the highest resolution for this model. AHPCRC also ran MM5 at 2.5-kilometer resolution, which takes 64 times the computation of the 10-kilometer model, and they're in the process of verifying that MM5 is still accurate at grid spacing this fine.

Oak Ridge researchers have run scientific applications up to 25 times faster than before with the Cray X1 system. They achieved a 50 percent performance improvement, on a processor-to-processor basis, over the Earth Simulator on the POP ocean modeling code. The Cray X1's overall scores on the HPC Challenge benchmark, as reported by customers, are the best for any HPC system. With problems that are really big and really difficult, especially problems with irregular memory references, the Cray X1 does exceptionally well.

HPCwire: Talk about initial customers for the Cray X1E and what they will be doing with the system.

Scott: We can talk about four of the initial customers. The Cray X1E system at KMA, the Korea Meteorological Administration, will be one of the world's biggest weather forecasting systems. KMA and Cray will also rely on this X1E for work in the Earth System Research Center that we'll jointly operate to advance the state of the art in atmospheric modeling in the East Asia Pacific region. Warsaw University's Interdisciplinary Centre for Mathematical and Computational Modeling received the first Cray X1E system in December 2004 and will use it for leading-edge research in mathematics and natural and computational science, including bioinformatics. Spain's National Institute of Meteorology will use their Cray X1E system for operational weather forecasting and for research in climate and atmospheric modeling. Oak Ridge will be getting a 20-teraflop X1E system this year, along with a 20-teraflop Cray XT3 massively parallel processing system. This is part of the DOE's plan to build the world's most powerful open scientific computer at Oak Ridge. Any researcher in the world will be able to use the Oak Ridge computing resources, as long as the problems are worthy and will be published in the open literature.

HPCwire: Do you have any performance results yet for the Cray X1E?

Scott: We've gotten some impressive results in internal testing at Cray. It's too soon for results on standard benchmarks

HPCwire: Will the recent purchasers of the Cray X1 system now need to upgrade? Could they have just waited until the Cray X1E was released to buy their supercomputing power?

Scott: Cray X1 customers don't need to upgrade, but many of them are: Because of the easy transition from the Cray X1 to the Cray X1E system, there wasn't much point in delaying Cray X1 purchases to wait for the Cray X1E. It made more sense to start running applications on the Cray X1 to get an immediate performance benefit, and then upgrade later for a big performance boost with no application impact.

HPCwire: So how difficult is it then to upgrade physically from the Cray X1 to the Cray X1E system? What about from an applications standpoint?

Scott: It's a simple board-level upgrade. The Cray X1E uses the same cabinets, network and I/O infrastructure as the Cray X1 system. The systems are binary compatible with each other, so nothing has to be done to the applications.

HPCwire: Some say the market for traditional vector supercomputers is dying. How long will Cray keep producing vector systems?

Scott: People have been saying that since long before the Cray X1 came out, but vector systems continue to sell. We'll keep producing them as long as they keep providing better performance for many of our customers' most crucial applications. There's really no such thing as a separate "vector market." Cray sells vector systems into the part of the HPC market that demands strong system balance and high sustained performance on very challenging applications.

We do believe, however, that traditional vector systems are outdated. Vector supercomputers used to require programmers to write differently than for their non-vector systems. The Cray X1 and Cray X1E products are highly scalable and have hierarchical memory system architectures that reward locality, so optimizing for these systems is similar to optimizing for other parallel architectures. When customers optimize codes for their Cray X1s, they typically find that these codes run faster on their scalar systems as well, whether the scalar systems are from Cray or other vendors.

Vector processing has some important advantages that fit in well with where integrated circuit technology is heading. Vectors give you a high degree of parallelism within the processor, with low complexity and power consumption. This is why many microprocessor vendors are starting to exploit vector processing, though not to the same extent Cray does. A vector processor can be very powerful because it needs to process far fewer instructions and perform much less dependency-checking than a scalar processor. For array- based calculations typical of most scientific codes, the control complexity is much lower on vector processors. Also, a typical microprocessor can handle perhaps 16 outstanding memory references, while a Cray X1E dual-core vector processor set can continue executing with up to 4,000 outstanding references. Vector processors can keep a lot more balls in the air.

HPCwire: So what are the target applications for the Cray X1E?

Scott: Classified applications, weather and climate, CFD, especially safety problems with complex physics involving turbulent flows. All of the applications that run well on the Cray X1 system will benefit from the Cray X1E. The compute-bound codes will benefit most from the processor upgrade.

HPCwire: The Cray X1E and Cray XT3 are both high-end systems. When and why would a customer choose one over the other?

Scott: No single architecture is best for all HPC codes, although strong balance and bandwidth are beneficial in all architectures and we engineer these into every one of our products. The Cray X1E system will deliver superior sustained performance on applications that are associated with really big data, irregular memory access or really big loop nests. The applications customers assign to the Cray X1E often represent a small percentage of their codes, but a high percentage of their cycles. High-end applications that are scalar-dominated or have more regular access patterns are a better fit for the Cray XT3 supercomputer. Customers generally know which system they're looking for, based on the nature of their workloads.

HPCwire: But how do you respond to customers who have a mix of application types?

Scott: Cray offers the widest choice of HPC products in the market. Some customers use more than one Cray product to handle their user requirements and workloads. They often use clusters, too, for their less-demanding requirements.

HPCwire: Is Cray still targeting sustained petaflops performance on real applications by 2010? What will achieving this mean for the industry?

Scott: Absolutely. Cray was the first vendor to break the sustained gigaflop and teraflop barriers on full 64-bit applications, and we're committed to sustained petaflops performance by 2010. We made that commitment when we announced the Cray X1 back in 2002. Cray was one of three HPC vendors selected for Phase II of the DARPA HPCS program, for our "Cascade" initiative. We're on track toward sustained petaflops speed on real-world applications by the end of the decade.

In 1999, the President's Information Technology Advisory Committee established the U.S. government's goal to attain a sustained petaflop on real applications by 2010. Their report said trans-petaflop systems will be crucial for more accurate weather and climate forecasting, advanced manufacturing, new pharmaceuticals, scientific research and other nationally important, strategic applications. In the Petaflops II conference and other meetings, experts identified a longer list of problems that could benefit from petaflop speed, including full-fidelity automotive crash testing, advanced aircraft and spacecraft design, rapid detection of wildfires, virtual surgery planning, national economic modeling, combating pandemics and bioterrorism and improving electrical power generation and distribution.

HPCwire: Where does Cray go from here? What comes after the Cray X1E system?

Scott: We have a strong roadmap for both our vector and scalar products going forward, and we'll be moving toward tighter integration of these capabilities in future systems. We won't say much more about that publicly for competitive reasons, but I've participated in quite a few NDA briefings with customers and others, and there's a high level of excitement about what Cray is planning in the next few years. HPCwire: We predicted that Cray would be making some big news in 2005, and it looks like we were right! Thanks for speaking with us here at HPCwire.

Copyright 1993-2004, HPCwire. All Rights Reserved.
Mirrored with permission.