PSTSWM Paragon Point-to-Point Communication Performance

Performance Studies using

PSTSWM

Intel Paragon SWAP Performance

(unordered swap of 128KB message using MPI)

Date/Person:	February 2, 1998 / P. Worley
Platform:	Intel Paragon XP/S 150 MP at Oak Ridge National Laboratory:
	1024 MP nodes (3 50-MHz iPSC/860 processors per node)
Environment:	Paragon OSF/1 Release 1.0.4 Server 1.4 R1_4_5
	f77/Paragon Paragon Version R5.0.3
Communication Library:	MPI
SWAP size:	16384 REAL*8 floating point values each direction
Message size:	Largest - 16384 REAL*8 floating point values
	Smallest - 16 REAL*8 floating point values
Processors:	0 and 1
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Results:

unordered simple swap

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 14.25 219.29 6.1%

10 iter. 14.25 214.36 6.8%

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 43.15 158.09 41.6%

10 iter. 55.16 158.07 23.9%

1 iter. w/overlap 43.87 184.86 35.0%

10 iter. w/overlap 51.49 161.75 31.0%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 65.31 136.26 25.9%

10 iter. 64.82 134.97 30.2%

1 iter. w/overlap 65.18 148.29 26.9%

10 iter. w/overlap 69.64 131.19 31.0%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 69.60 135.78 22.4%

10 iter. 67.47 135.19 31.4%

1 iter. w/overlap 67.65 191.48 38.0%

10 iter. w/overlap 76.42 171.35 31.4%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 65.18 276.69 25.3%

10 iter. 66.75 275.85 20.3%

1 iter. w/overlap 67.14 160.86 25.4%

10 iter. w/overlap 70.33 137.23 29.6%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 65.75 275.36 17.7%

10 iter. 67.63 274.93 23.1%

1 iter. w/overlap 69.03 203.03 37.0%

10 iter. w/overlap 76.26 176.18 33.1%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 65.67 149.82 24.0%

10 iter. 63.76 148.24 28.0%

DISCUSSION

Unordered simple swap uses MPI_BSEND, which involves an additional memory-to-memory copy and significantly degrades achieveable bandwidth.
Message-passing performance under OSF on the Paragon degrades significantly when attempting simultaneous send and receive on the same node (due to thrashing when accessing an address table?). This can be seen easily by comparing bandwidth estimates for these unordered results with bandwidth estimates for the ordered SWAP protocols. (Note that this is not as significant a problem when running the SUNMOS operating system from Sandia National Laboratory on the Paragon.)

Performance Studies using

PSTSWM

Intel Paragon SWAP Performance

(unordered swap of 128KB message using MPI)

DISCUSSION

Patrick H. Worley / ( worleyph@ornl.gov) Last Modified Monday, 15-Jul-2002 10:29:10 EDT. 86852 accesses since 1/2/96.

Patrick H. Worley / ( worleyph@ornl.gov)
Last Modified Monday, 15-Jul-2002 10:29:10 EDT.
86852 accesses since 1/2/96.