-
Performance Studies using
PSTSWM
|
|
|
Communication overhead for parallel runs of PSTSWM on a given platform is a function of the parallel algorithm, the parallel algorithm implementation, and the problem granularity (problem size and number of processors). One aspect of the parallel implementation that we have found to be important on some platforms is the message-passing protocol. Some indication of the variation that can occur from the choice of message-passing protcol can be seen from the basic point-to-point communication performance results described here. In these studies, we examine the impact of the choice of protocol in more detail, looking at the effect on the performance of specific parallel algorithm options in PSTSWM.
In the spectral transform method used in PSTSWM, fields are transformed at each timestep between the physical (longitude-latitude-vertical) and the Fourier (wavenumber-latitude-vertical) domains using Fourier transforms in the longitude direction, and between the Fourier and spectral (spectral coefficients - vertical) domains using a Legendre transform in the latitude direction. All parallel algorithms in PSTSWM are based on decomposing the different computational domains onto a logical two-dimensional grid of processors, PXx PY. In each domain two of the domain dimensions are decomposed across the processor grid, for example, assigning longitude-latitude patches of the physical domain to individual processors, but leaving one domain dimension undecomposed.
Two general types of parallel algorithms are used in PSTSWM: transpose and distributed. In a transpose algorithm, the decomposition is ``rotated'' before a transform begins, to ensure that all data needed to compute a particular transform is local to a single processor. In a distributed algorithm the original decomposition of the domain is retained, and communication is performed to allow the processors to cooperate in the calculation of a transform.
Three transpose algorithms are examined, each of which is functionally equivalent to MPI_ALLTOALLV:
Four distributed Legendre transform algorithms are examined, the first three of which are functionally equivalent to MPI_ALLREDUCE:
Each of these algorithms can be implemented using two general classes of protocols: unordered and ordered. While not all protocols are available for all message-passing transport layers, they are drawn from those described below. Examples are given for the SWAP command using MPI commands. Similar definitions hold for the SENDRECV command. Note that the examples have been simplified (to save room) and do not accurately represent the MPI implementations. In particular, handshaking messages required for correct use of the ready send command have been omitted.
Unordered |
Ordered |
|||||||||||||||||||||
|
|
|||||||||||||||||||||
|
|
|||||||||||||||||||||
|
|
|||||||||||||||||||||
|
|
|||||||||||||||||||||
|
|
|||||||||||||||||||||
|
|
|||||||||||||||||||||
|
|
To examine the performance issues in these different implementation options, we run one or more of the following experiments.
In actual practice, each parallel algorithm will run on subsets of processors in a two-dimensional virtual processor grid. Thus Experiment A does not take into account the performance effect of a realistic assignment of processes to processors when the virtual processor grid does not match the physical interconnection topology, nor the contention caused by multiple subsets running the same parallel algorithm. This experiment also overestimates the importance of the performance variation due to the choice of communication protocol. The measured runtimes will be smaller than those for the corresponding "real" two-dimensional runs since they do not include the communication cost for the parallel algorithm in the other direction.
Experiment B allows us to examine one example of the contention for resources when running multiple copies of the parallel algorithm. It is not the same amount of contention as would be seen in a real run for the 16 and 32 processor cases, but it should provide us with some information. Note that this experiment underestimates the performance variation due to the communication protocol for the 8x8 case in that the execution in a realistic run will be smaller. The default parallel algorithm for the other dimension uses a communication protocol that is generally one of the worst performers. It was chosen because it is the only option that is supported across all of the message passing libraries that PSTSWM supports.
Results are presented as scatterplots for each parallel platform, set of experiments, parallel algorithm, problem size, and number of processors. The x-axis is the protocol option (0-6) and the y-axis is the execution time for 12 timesteps of PSTSWM using the standard benchmark problem case (global steady state nonlinear zonal geostrophic flow). Each x-axis tick mark will have results for unordered, unordered/overlap; ordered, and ordered/overlap protocols. Some of the algorithms have results presented for different degrees of overlap, with overlap indicating a modest degree of overlap and OVERLAP indicating the maximum possible. The ringpipe algorithm also has additional send-ahead, send-ahead/overlap, and send-ahead/OVERLAP options, in which the message send and message receive are separated by significant computation. Three separate experimental results are presented for each transpose algorithm. The second uses a Px1 decomposition, representing the parallel Fourier transform. The other two use a 1xP decomposition, representing a parallel Legendre transform, but where the first assumes a distributed parallel Fourier transform while the other assumes a transpose parallel Fourier transform. Note that these three transpose experiments differ primarily in the size of the messages being transmitted.
Statistics summarizing the data are presented below each plot:
A summary of the results is also provided for each platform and message-passing library combination. For each parallel algorithm, we indicate
If results for more than one message-passing library are provided for a given platform, then performance differences between the libraries are discussed. In particular, we compare the optimal protocols, the sensitivity to choice of protocol, and the sensitivity to contention and process placement.
Note that timings are for the entire code, and that percentage differences in communication costs will generally be much higher than the percentage differences in total runtime reported for comparable tests cases. Changing the communication protocols does not change the complexity of the runs and does not significantly change the total amount of data moved. What is affected is the number and order of handshaking messages, and whether bidirectional bandwidth or communication/computation overlap are attempted. The one case where the computation may be affected is the distributed FFT dfft. In order to exploit overlap, the block FFT is divided into two blocks. On some machines and with some problem granularities, transforming two small blocks results in a higher computational rate than transforming a single large block. This should be taken into account when comparing dfft protocols.
Also note that when comparing between platforms, the sensitivity is a strong function of the communication/computation ratio, and a platform with a slow processor may show significantly less performance sensitivity than one with a fast processor. Thus high performance sensitivity to the communication protocol is more of a feature than a bug, but it is a feature that should be recognized if goos performance is to be achieved.
| A: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| B: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| C: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| A: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| B: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| C: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| A1: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| A2: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| A1: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| A2: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| A1: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| A2: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| A3: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| B1: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| B2: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
| C: | DFFT | EXCHSUM | HALFSUM | ||
| RINGPIPE | RINGSUM | ||||
| LOGTRANS (1) | SRTRANS (1) | SWTRANS (1) | |||
| LOGTRANS (2) | SRTRANS (2) | SWTRANS (2) | |||
| LOGTRANS (3) | SRTRANS (3) | SWTRANS (3) |
PSTSWM Performance Page 