PSTSWM Paragon Point-to-Point Communication Performance

Performance Studies using

PSTSWM

Intel Paragon SWAP Performance

(unordered swap of 8KB message using MPI)

Date/Person:	January 23, 1998 / P. Worley
Platform:	Intel Paragon XP/S 150 MP at Oak Ridge National Laboratory:
	1024 MP nodes (3 50-MHz iPSC/860 processors per node)
Environment:	Paragon OSF/1 Release 1.0.4 Server 1.4 R1_4_5
	f77/Paragon Paragon Version R5.0.3
Communication Library:	MPI
SWAP size:	1024 REAL*8 floating point values each direction
Message size:	Largest - 1024 REAL*8 floating point values
	Smallest - 1 REAL*8 floating point values
Processors:	0 and 1
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Results:

unordered simple swap

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 11.64 220.33 16.5%

10 iter. 11.76 222.31 20.3%

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 36.32 160.02 35.5%

10 iter. 41.25 159.14 40.1%

1 iter. w/overlap 27.43 184.93 42.7%

10 iter. w/overlap 39.26 158.53 38.0%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 41.58 130.00 33.0%

10 iter. 51.51 130.75 44.5%

1 iter. w/overlap 44.96 140.35 38.5%

10 iter. w/overlap 55.27 126.70 47.1%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 47.85 129.58 37.8%

10 iter. 44.09 130.88 47.8%

1 iter. w/overlap 31.02 182.71 55.8%

10 iter. w/overlap 51.16 166.01 52.2%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 34.92 275.11 58.6%

10 iter. 36.30 271.68 60.2%

1 iter. w/overlap 55.00 153.33 51.5%

10 iter. w/overlap 55.64 133.11 45.8%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 35.27 274.11 59.0%

10 iter. 38.73 270.30 63.9%

1 iter. w/overlap 41.25 192.87 48.6%

10 iter. w/overlap 59.06 170.22 61.4%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 28.91 143.15 52.9%

10 iter. 39.88 141.62 59.1%

DISCUSSION

Unordered simple swap uses MPI_BSEND, which involves an additional memory-to-memory copy and significantly degrades achieveable bandwidth.
Message-passing performance under OSF on the Paragon degrades when attempting simultaneous send and receive on the same node (due to thrashing when accessing an address table?). This can be seen by comparing bandwidth estimates for these unordered results with bandwidth estimates for the ordered SWAP protocols. (Note that this is not as significant a problem when running the SUNMOS operating system from Sandia National Laboratory on the Paragon.)

Performance Studies using

PSTSWM

Intel Paragon SWAP Performance

(unordered swap of 8KB message using MPI)

DISCUSSION

Patrick H. Worley / ( worleyph@ornl.gov) Last Modified Monday, 15-Jul-2002 10:29:12 EDT. 86604 accesses since 1/2/96.

Patrick H. Worley / ( worleyph@ornl.gov)
Last Modified Monday, 15-Jul-2002 10:29:12 EDT.
86604 accesses since 1/2/96.