PSTSWM on the Cray X1

Introduction

PSTSWM was originally designed as a testbed for parallel spectral algorithms on the sphere. Its serial performance, especially its memory access pattern, is similar to that of the spectral dynamical core used in the NCAR global atmospheric models. By scaling the problem size (horizontal and vertical resolutions), the effect of the memory subsystem on performance can be ascertained. Running multiple versions of the serial code simultaneously allows us to examine the performance impact of memory contention as well. The X1 performance data was collected by Patrick H. Worley in April, May, and June of 2003.

PSTSWM is dominated by floating point multiple/add operations, and can also take advantage of math library FFT routines. (Fortran implementations of the FFT routines were used for the Cray X1 experiments described below.) PSTSWM also exhibits little reuse of operands as it sweeps through the field arrays, making significant demands on the memory subsystem. By these characteristics, the code should run well on vector systems. However, in the standard version of PSTSWM, all array dimensions and loop bounds are determined at runtime. This makes it difficult for the compiler to identify which loops to vectorize or parallelize.

Architecture Summary

The primary strategy for utilizing the eight vector units of a single MSP is parallelization ("streaming") across outer loops and vectorization within inner loops. However, Cray does support the option of treating each SSP as a separate processor. The Ecache has sufficient single-stride bandwidth to saturate the vector units of the MSP. The Ecache is needed because the bandwidth to main memory is not enough to saturate the vector units without data reuse - memory bandwidth is roughly half the saturation bandwidth.

Four MSPs and a flat, shared memory of 16 GB form a Cray X1 node. The memory banks of a node provide 200 GB/s of bandwidth, enough to saturate the paths to the local MSPs and service requests from remote MSPs. Each bank of shared memory is connected to a number of banks on remote nodes, with an aggregate bandwidth of roughly 50 GB/s between nodes. This represents a byte per flop of interconnect bandwidth per computation rate, compared to 0.25 bytes per flop on the Earth Simulator and less than 0.1 bytes per flop expected on an IBM p690 with the maximum number of Federation connections. The collected nodes of an X1 have a single system image.

Experiment Overview

Eight horizontal resolutions were used:

• T5 (16x8 horizontal grid)
• T10 (32x16 horizontal grid)
• T21 (64x32 horizontal grid)
• T42 (128x64 horizontal grid)
• T85 (256x128 horizontal grid)
• T170 (512x256 horizontal grid)
• T340 (1024x512 horizontal grid)
• T680 (2048x1024 horizontal grid)

Our experiments describe the performance of a version of PSTSWM that was ported to the Cray X1 in April, 2003. Profiling was used to indicate the routines that were most important to optimize, and numerous experiments involving a variety of index and loop ordering and compiler directives were used to improve performance of these routines. All changes were local to these routines, and the global data structures were not altered. After these optimizations, the same routines were still the most important, but performance was improved significantly.