Home > PC >
Originally appeared in March 11, 2005
Cray Makes Waves With X1E Upgrade
Cray has been making some big news this week. The provider of
high-performance supercomputer systems recently appointed a new
president, Peter Ungaro, and announced the availability of the Cray
X1E supercomputer, a major upgrade to the company's record-setting
Cray X1 scalable vector product. The new system nearly triples the
peak performance and price performance of the presiding Cray X1 system
that has set records on weather, engineering and scientific research
Steve Scott, Cray's chief technology officer and chief architect of
the Cray X1 series, recently spoke with HPCwire to provide more
details about the announcement and future Cray plans.
HPCwire: The Cray X1E supercomputer is a major upgrade to the Cray X1
product. What are the main differences?
Steve Scott: The primary difference is that by exploiting the
next-generation integrated circuit technology, the Cray X1E system
nearly triples the peak performance and compute density of the Cray X1
product. We boosted the peak speed of each processor by almost 50% and
placed two processors on each multi- chip module, instead of one in
the Cray X1 system. The net result is a peak performance of 18
gigaflops per processor, which makes this the world's fastest
processor. On a single eight-processor module, you have 144 peak
gigaflops; and in the same physical cabinet, you go from 819 gigaflops
with Cray X1 processors to 2.3 teraflops using Cray X1E processors.
These are substantial advances in computing power and compute density;
and as you know, the more powerful the individual processors, the
easier it is to scale up applications. The Cray X1E processors are
extremely latency tolerant and are coupled with a very high bandwidth
memory system and interconnect. The result is a system designed to
scale efficiently on the most challenging class of problems.
HPCwire: The Cray X1E uses dual-core vector processors to gain compute
density, and at Linuxworld last month, Cray became one of the first to
demonstrate dual-core Opterons on the Cray XD1 supercomputer. What do
you find to be the major advantages and disadvantages of dual core?
Scott: The biggest advantage of dual core in the Cray X1E system is
doubling the compute density without a corresponding increase in
complexity. There are also advantages for power consumption and clock
speed, by exploiting locality on the silicon. This complements the
already excellent power efficiency of the vector processors used in
The greatest disadvantage of going dual core is that you reduce the
ratio of memory bandwidth to peak computational speed. In that regard,
it helps tremendously to start with strong bandwidth. We designed the
Cray X1 and our Opteron-based products, the Cray XD1 and Cray XT3,
with dual core in mind. Let me give you an example of what this means.
In a typical single-core cluster, you have four microprocessors
sharing about six gigabytes per second of bandwidth on a bus. With
dual core on that same cluster, you now have eight processors sharing
the original bandwidth, which means you have less than one gigabyte
per second of bandwidth per processor. Contrast that with a Cray X1E
dual core system, where two processors share 35 gigabytes of
bandwidth, giving each of them more than 17 gigabytes. This has major
implications for the kinds of problems you can tackle and scale up
efficiently in practice.
HPCwire: On that topic, how has the predecessor Cray X1 system
performed in practice?
Scott: The Cray X1 system has produced record results on really large,
difficult scientific and engineering applications. Army High
Performance Computing Research Center researchers ran a challenging
CFD code at more than one sustained teraflop on only 256 processors.
They used the MM5 model to complete a 24-hour weather forecast for the
continental U.S. at five-kilometer resolution in a very short time.
Five-kilometer resolution takes eight times the computing power of the
10-kilometer grid spacing that's typically been the highest resolution
for this model. AHPCRC also ran MM5 at 2.5-kilometer resolution, which
takes 64 times the computation of the 10-kilometer model, and they're
in the process of verifying that MM5 is still accurate at grid spacing
Oak Ridge researchers have run scientific applications up to 25 times
faster than before with the Cray X1 system. They achieved a 50 percent
performance improvement, on a processor-to-processor basis, over the
Earth Simulator on the POP ocean modeling code. The Cray X1's overall
scores on the HPC Challenge benchmark, as reported by customers, are
the best for any HPC system. With problems that are really big and
really difficult, especially problems with irregular memory
references, the Cray X1 does exceptionally well.
HPCwire: Talk about initial customers for the Cray X1E and what they
will be doing with the system.
Scott: We can talk about four of the initial customers. The Cray X1E
system at KMA, the Korea Meteorological Administration, will be one of
the world's biggest weather forecasting systems. KMA and Cray will
also rely on this X1E for work in the Earth System Research Center
that we'll jointly operate to advance the state of the art in
atmospheric modeling in the East Asia Pacific region. Warsaw
University's Interdisciplinary Centre for Mathematical and
Computational Modeling received the first Cray X1E system in December
2004 and will use it for leading-edge research in mathematics and
natural and computational science, including bioinformatics. Spain's
National Institute of Meteorology will use their Cray X1E system for
operational weather forecasting and for research in climate and
atmospheric modeling. Oak Ridge will be getting a 20-teraflop X1E
system this year, along with a 20-teraflop Cray XT3 massively parallel
processing system. This is part of the DOE's plan to build the world's
most powerful open scientific computer at Oak Ridge. Any researcher in
the world will be able to use the Oak Ridge computing resources, as
long as the problems are worthy and will be published in the open
HPCwire: Do you have any performance results yet for the Cray X1E?
Scott: We've gotten some impressive results in internal testing at
Cray. It's too soon for results on standard benchmarks
HPCwire: Will the recent purchasers of the Cray X1 system now need to
upgrade? Could they have just waited until the Cray X1E was released
to buy their supercomputing power?
Scott: Cray X1 customers don't need to upgrade, but many of them are:
Because of the easy transition from the Cray X1 to the Cray X1E
system, there wasn't much point in delaying Cray X1 purchases to wait
for the Cray X1E. It made more sense to start running applications on
the Cray X1 to get an immediate performance benefit, and then upgrade
later for a big performance boost with no application impact.
HPCwire: So how difficult is it then to upgrade physically from the
Cray X1 to the Cray X1E system? What about from an applications
Scott: It's a simple board-level upgrade. The Cray X1E uses the same
cabinets, network and I/O infrastructure as the Cray X1 system. The
systems are binary compatible with each other, so nothing has to be
done to the applications.
HPCwire: Some say the market for traditional vector supercomputers is
dying. How long will Cray keep producing vector systems?
Scott: People have been saying that since long before the Cray X1 came
out, but vector systems continue to sell. We'll keep producing them as
long as they keep providing better performance for many of our
customers' most crucial applications. There's really no such thing as
a separate "vector market." Cray sells vector systems into the part
the HPC market that demands strong system balance and high sustained
performance on very challenging applications.
We do believe, however, that traditional vector systems are outdated.
Vector supercomputers used to require programmers to write differently
than for their non-vector systems. The Cray X1 and Cray X1E products
are highly scalable and have hierarchical memory system architectures
that reward locality, so optimizing for these systems is similar to
optimizing for other parallel architectures. When customers optimize
codes for their Cray X1s, they typically find that these codes run
faster on their scalar systems as well, whether the scalar systems are
from Cray or other vendors.
Vector processing has some important advantages that fit in well with
where integrated circuit technology is heading. Vectors give you a
high degree of parallelism within the processor, with low complexity
and power consumption. This is why many microprocessor vendors are
starting to exploit vector processing, though not to the same extent
Cray does. A vector processor can be very powerful because it needs to
process far fewer instructions and perform much less
dependency-checking than a scalar processor. For array- based
calculations typical of most scientific codes, the control complexity
is much lower on vector processors. Also, a typical microprocessor can
handle perhaps 16 outstanding memory references, while a Cray X1E
dual-core vector processor set can continue executing with up to 4,000
outstanding references. Vector processors can keep a lot more balls in
HPCwire: So what are the target applications for the Cray X1E?
Scott: Classified applications, weather and climate, CFD, especially
safety problems with complex physics involving turbulent flows. All of
the applications that run well on the Cray X1 system will benefit from
the Cray X1E. The compute-bound codes will benefit most from the
HPCwire: The Cray X1E and Cray XT3 are both high-end systems. When and
why would a customer choose one over the other?
Scott: No single architecture is best for all HPC codes, although
strong balance and bandwidth are beneficial in all architectures and
we engineer these into every one of our products. The Cray X1E system
will deliver superior sustained performance on applications that are
associated with really big data, irregular memory access or really big
loop nests. The applications customers assign to the Cray X1E often
represent a small percentage of their codes, but a high percentage of
their cycles. High-end applications that are scalar-dominated or have
more regular access patterns are a better fit for the Cray XT3
supercomputer. Customers generally know which system they're looking
for, based on the nature of their workloads.
HPCwire: But how do you respond to customers who have a mix of
Scott: Cray offers the widest choice of HPC products in the market.
Some customers use more than one Cray product to handle their user
requirements and workloads. They often use clusters, too, for their
HPCwire: Is Cray still targeting sustained petaflops performance on
real applications by 2010? What will achieving this mean for the
Scott: Absolutely. Cray was the first vendor to break the sustained
gigaflop and teraflop barriers on full 64-bit applications, and we're
committed to sustained petaflops performance by 2010. We made that
commitment when we announced the Cray X1 back in 2002. Cray was one of
three HPC vendors selected for Phase II of the DARPA HPCS program, for
our "Cascade" initiative. We're on track toward sustained petaflops
speed on real-world applications by the end of the decade.
In 1999, the President's Information Technology Advisory Committee
established the U.S. government's goal to attain a sustained petaflop
on real applications by 2010. Their report said trans-petaflop systems
will be crucial for more accurate weather and climate forecasting,
advanced manufacturing, new pharmaceuticals, scientific research and
other nationally important, strategic applications. In the Petaflops
II conference and other meetings, experts identified a longer list of
problems that could benefit from petaflop speed, including
full-fidelity automotive crash testing, advanced aircraft and
spacecraft design, rapid detection of wildfires, virtual surgery
planning, national economic modeling, combating pandemics and
bioterrorism and improving electrical power generation and
HPCwire: Where does Cray go from here? What comes after the Cray X1E
Scott: We have a strong roadmap for both our vector and scalar products going
forward, and we'll be moving toward tighter integration of these capabilities
in future systems. We won't say much more about that publicly for competitive
reasons, but I've participated in quite a few NDA briefings with customers and
others, and there's a high level of excitement about what Cray is planning in
the next few years.
HPCwire: We predicted that Cray would be making some big news in 2005,
and it looks like we were right! Thanks for speaking with us here at
Copyright 1993-2004, HPCwire. All Rights Reserved.
Mirrored with permission.