PSTSWM Paragon Point-to-Point Communication Performance

Performance Studies using

PSTSWM

Intel Paragon SWAP Performance

(unordered swap of 2MB message using MPI)

Date/Person:	January 30, 1998 / P. Worley
Platform:	Intel Paragon XP/S 150 MP at Oak Ridge National Laboratory:
	1024 MP nodes (3 50-MHz iPSC/860 processors per node)
Environment:	Paragon OSF/1 Release 1.0.4 Server 1.4 R1_4_5
	f77/Paragon Paragon Version R5.0.3
Communication Library:	MPI
SWAP size:	262144 REAL*8 floating point values each direction
Message size:	Largest - 262144 REAL*8 floating point values
	Smallest - 256 REAL*8 floating point values
Processors:	0 and 1
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Results:

unordered simple swap

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 14.65 162.75 27.3%

10 iter. 14.66 162.41 25.9%

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 55.25 157.04 19.5%

10 iter. 58.26 151.92 21.7%

1 iter. w/overlap 55.22 183.10 19.2%

10 iter. w/overlap 58.13 155.99 21.1%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 73.18 63.88 27.8%

10 iter. 72.91 61.12 29.0%

1 iter. w/overlap 74.58 100.26 6.9%

10 iter. w/overlap 74.05 86.60 4.0%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 72.25 90.27 16.2%

10 iter. 72.51 82.17 22.6%

1 iter. w/overlap 80.09 157.52 21.1%

10 iter. w/overlap 74.07 141.76 27.0%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 72.03 245.72 11.4%

10 iter. 72.20 245.96 10.0%

1 iter. w/overlap 73.98 112.90 6.2%

10 iter. w/overlap 73.26 91.40 2.9%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 72.67 245.68 9.4%

10 iter. 72.34 244.66 11.0%

1 iter. w/overlap 77.79 167.12 21.2%

10 iter. w/overlap 74.10 147.17 28.8%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

1 iter. 73.15 82.51 24.3%

10 iter. 73.86 83.40 21.3%

DISCUSSION

Unordered simple swap uses MPI_BSEND, which involves an additional memory-to-memory copy and significantly degrades achieveable bandwidth.
Message-passing performance under OSF on the Paragon degrades significantly when attempting simultaneous send and receive on the same node (due to thrashing when accessing an address table?). This can be seen easily by comparing bandwidth estimates for these unordered results with bandwidth estimates for the ordered SWAP protocols. (Note that this is not as significant a problem when running the SUNMOS operating system from Sandia National Laboratory on the Paragon.)

Performance Studies using

PSTSWM

Intel Paragon SWAP Performance

(unordered swap of 2MB message using MPI)

DISCUSSION

Patrick H. Worley / ( worleyph@ornl.gov) Last Modified Monday, 15-Jul-2002 10:29:08 EDT. 86389 accesses since 1/2/96.

Patrick H. Worley / ( worleyph@ornl.gov)
Last Modified Monday, 15-Jul-2002 10:29:08 EDT.
86389 accesses since 1/2/96.