Performance Studies using CCM/MP-2D

Compaq AlphaServer SC Parallel Algorithms for CCM/MP-2D

(Results based on November, 1999 PSTSWM Experiments)

The AlphaServer SC is a distributed-memory parallel architecture utilizing high-end workstation-class processors interconnected by a Quadrics (fat-tree) switch. The SC used in these experiments was a Compaq development machine with 8 4-way SMP nodes. We used the results of our November, 1999 studies of PSTSWM performance to identify the following parallel algorithms for CCM/MP-2D on the AlphaServer SC.

When referring to the parallel algorithms and their implementations, we use the following shorthand:

• The name of the individual parallel algorithms, e.g., srtrans, is described on the PSTSWM protocol web pages.
• "overlap" refers to a communication protocol that posts one or more send or receive requests as early as possible, in an attempt to overlap communication with computation or to hide latency. The default case is "no overlap".
• "ordered" refers to a communication protocol that does not attempt to exploit bidirectional bandwidth in a swap or send-receive operation, instead having one processor send and the other receive, followed by the reverse when the first send is complete. The default is "unordered", i.e., logically sending both directions simultaneously.
• "ready send" refers to a communication protocol that uses the MPI_RSEND or MPI_IRSEND and additional handshaking messages to control precisely when messages are sent and received.
• A parallel algorithm for CCM/MP-2D is specified as a vector consisting of the codes for the individual parallel algorithms, in the following order:

  (parallel FFT, parallel Legendre transform, parallel semi-Lagrangian transport, physics load balancing)

The parallel algorithms chosen for the AlphaServer SC experiments are listed below. Seven different algorithms were examined, three distributed FT/distributed Legendre transform algorithms:

d0: (df0 , dl0 , ds0 , lb0)

d1: (df1 , dl1 , ds1 , lb1)

d2: (df2 , dl2 , ds2 , lb2)

and four transpose FFT/distributed Legendre transform algorithms:

t0: (tf0 , dl0 , ds0 , lb0)

t1: (tf0 , dl1 , ds0 , lb0)

t2: (tf1 , dl1 , ds1 , lb1)

t3: (tf2 , dl2 , ds2 , lb2)

where the codes for the individual parallel algorithms are as follows:

Distributed FFT

• df0 - ordered nonblocking receive, nonblocking send protocol: (1,3)
• df1 - ordered nonblocking receive, overlap protocol: (3,2)
• df2 - MPI_SENDRECV-based communication protocol: (0,6)

Transpose FFT

• tf0 - srtrans implemented using a nonblocking receive, nonblocking send overlap protocol: (4,3)
• tf1 - srtrans implemented using a nonblocking send protocol: (0,1)
• tf2 - MPI_ALLTOALLV-based parallel algorithm: (0,6)

Distributed Legendre Transform

• dl0 - halfsum implemented using MPI_SENDRECV: (20,6)
• dl1 - halfsum implemented using a nonblocking receive, ready send overlap protocol: (20,3,4)
• dl2 - MPI_ALLREDUCE-based parallel algorithm: (0,6)

Distributed semi-Lagrangian

• ds0 - nonblocking receive, nonblocking send (overlap) protocol: (0,3)
• ds1 - nonblocking send protocol: (0,1) for sendrecv communication and ordered, noblocking receive protocol: (1,2) for swap communication
• ds2 - MPI_SENDRECV-based communication protocol: (0,6)

Physics load balancing

• lb0 - nonblocking receive, nonblocking send (overlap) protocol: (0,3)
• lb1 - ordered, noblocking receive protocol: (1,2)
• lb2 - MPI_SENDRECV-based communication protocol: (0,6)

d0, t0, and t1 use protocols that the PSTSWM experiments indicate are best for "small" granularity problems, while d1 and t2 use protocols that are best for "large" grain problems. d2 and t3 are the standard MPI protocols that one would choose without tuning.

---

Patrick H. Worley / ( worleyph@ornl.gov)
Last Modified Monday, 15-Jul-2002 10:03:35 EDT.
86302 accesses since 1/2/96.