COMMTEST SP3-375 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM SP3-375 Winterhawk II SWAP Performance

(unordered swap of 8KB message using MPI within a node)

(performance measured per processor when all processors in node communicating)

Date/Person:	May 25, 2000 / P. Worley
Platform:	IBM SP3 at Oak Ridge National Laboratory (eagle.ccs.ornl.gov):
	160 4-way Winterhawk II SMP nodes (375MHz POWER3 with 8MB L2 cache)
Environment:	AIX 4.3.3; PSSP 3.1.1
Communication Library:	shared memory MPI within a node, MPI over the switch between nodes, using User Space protocol in both cases
SWAP size:	1024 REAL*8 floating point values each direction
Message size:	Largest - 1024 REAL*8 floating point values
	Smallest - 1 REAL*8 floating point values
Processors:	0 and 1
	2 and 3
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

unordered simple swap

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 39.83 13.66 18.5%

1 iter. 152.79 13.93 42.9%

10 iter. 206.47 13.35 35.8%

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 41.83 14.06 11.9%

1 iter. 175.41 13.85 29.7%

10 iter. 246.61 13.79 41.5%

cache inv. w/overlap 38.85 14.71 20.4%

1 iter. w/overlap 186.90 14.41 32.6%

10 iter. w/overlap 227.99 14.27 39.7%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 45.06 13.39 11.3%

1 iter. 216.61 13.20 34.9%

10 iter. 232.89 13.15 39.1%

cache inv. w/overlap 51.20 13.65 10.6%

1 iter. w/overlap 181.62 13.34 52.2%

10 iter. w/overlap 232.85 13.27 37.7%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 42.67 14.91 12.3%

1 iter. 189.67 14.79 33.9%

10 iter. 243.88 14.83 44.2%

cache inv. w/overlap 45.19 15.41 16.2%

1 iter. w/overlap 188.51 15.35 35.3%

10 iter. w/overlap 242.15 15.15 44.8%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 43.20 24.60 6.5%

1 iter. 153.86 24.11 45.3%

10 iter. 229.25 24.13 38.4%

cache inv. w/overlap 51.51 14.27 10.2%

1 iter. w/overlap 236.57 13.65 20.8%

10 iter. w/overlap 280.37 13.66 24.9%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 40.05 25.78 11.6%

1 iter. 165.42 25.63 46.4%

10 iter. 209.79 25.80 43.7%

cache inv. w/overlap 48.86 15.72 11.2%

1 iter. w/overlap 195.44 15.34 29.2%

10 iter. w/overlap 254.81 15.16 26.3%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 48.02 13.43 10.7%

1 iter. 222.56 13.30 36.1%

10 iter. 216.82 13.32 40.2%

unordered swap using nonblocking sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 42.04 35.64 10.8%

1 iter. 188.74 33.76 38.9%

10 iter. 214.05 33.96 44.4%

cache inv. w/overlap 40.27 36.10 11.0%

1 iter. w/overlap 140.65 34.24 57.4%

10 iter. w/overlap 204.39 33.92 42.3%

unordered swap using nonblocking receive with sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 38.57 35.82 14.2%

1 iter. 188.51 34.24 39.4%

10 iter. 210.86 34.94 45.0%

cache inv. w/overlap 48.01 36.43 10.7%

1 iter. w/overlap 163.67 35.83 47.6%

10 iter. w/overlap 209.28 34.92 44.6%

unordered swap using nonblocking sync. send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 41.67 40.00 10.2%

1 iter. 182.69 37.55 41.9%

10 iter. 217.68 38.12 50.7%

cache inv. w/overlap 44.64 43.62 18.1%

1 iter. w/overlap 194.46 39.04 46.3%

10 iter. w/overlap 182.69 39.97 47.4%

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.4196e-02 1.3863e-05 1.15 0.69 0.79 1.88

16 7.2914e-03 1.4241e-05 2.25 0.67 0.76 1.80

32 3.8052e-03 1.4864e-05 4.31 0.65 0.73 1.82

64 2.0241e-03 1.5814e-05 8.09 0.64 0.73 1.72

128 1.1301e-03 1.7658e-05 14.50 0.60 0.62 1.55

256 7.1011e-04 2.2191e-05 23.07 0.47 0.57 1.21

512 5.0245e-04 3.1403e-05 32.61 0.40 0.45 1.00

1024 4.2447e-04 5.3059e-05 38.60 0.24 0.28 0.57

2048 3.7245e-04 9.3112e-05 43.99 0.21 0.25 0.41

4096 3.4117e-04 1.7059e-04 48.02 0.20 0.21 0.32

8192 3.6426e-04 3.6426e-04 44.98 0.11 0.08 0.26

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 0 1 3

16 0 2 6 1 3

32 6 2 0 1 3

64 6 0 2 1 3

128 2 6 0 1 3

256 6 2 0 1 3

512 6 2 0 3 1

1024 6 2 3 1 0

2048 6 2 3 0 1

4096 6 2 3 0 5

8192 6 4 2 7 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 3 4 5

16 3 4 5

32 3 4 5

64 1 3 5

128 1 1 5

256 1 3 5

512 1 2 5

1024 2 2 5

2048 1 2 5

4096 1 1 6

8192 1 2 9

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.3592e-02 1.3274e-05 1.21 0.70 0.82 1.85

16 6.8348e-03 1.3349e-05 2.40 0.70 0.82 1.85

32 3.4263e-03 1.3384e-05 4.78 0.70 0.83 1.87

64 1.7531e-03 1.3696e-05 9.35 0.69 0.82 1.84

128 8.9798e-04 1.4031e-05 18.25 0.66 0.77 1.79

256 4.6032e-04 1.4385e-05 35.59 0.66 0.83 1.73

512 2.4989e-04 1.5618e-05 65.56 0.62 0.72 1.59

1024 1.4596e-04 1.8245e-05 112.25 0.58 0.63 1.49

2048 9.9043e-05 2.4761e-05 165.42 0.39 0.42 0.92

4096 7.3617e-05 3.6808e-05 222.56 0.38 0.45 0.77

8192 8.6383e-05 8.6383e-05 189.67 0.13 0.15 0.38

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 1 0 3

16 2 6 1 0 3

32 2 6 1 0 3

64 2 6 1 0 3

128 2 6 0 1 3

256 2 6 0 1 3

512 6 2 0 1 3

1024 2 6 3 1 0

2048 2 6 3 0 1

4096 6 2 1 3 5

8192 3 7 8 6 9

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 2 5

16 2 2 5

32 1 2 5

64 2 2 5

128 1 4 5

256 1 2 5

512 1 2 5

1024 1 2 4

2048 2 2 5

4096 1 2 2

8192 3 5 8

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.3536e-02 1.3218e-05 1.21 0.71 0.83 1.89

16 6.8041e-03 1.3289e-05 2.41 0.70 0.83 1.88

32 3.3909e-03 1.3246e-05 4.83 0.71 0.84 1.87

64 1.7301e-03 1.3516e-05 9.47 0.69 0.81 1.84

128 8.8249e-04 1.3789e-05 18.57 0.68 0.79 1.81

256 4.5793e-04 1.4310e-05 35.78 0.66 0.75 1.76

512 2.4361e-04 1.5225e-05 67.26 0.63 0.75 1.67

1024 1.4097e-04 1.7621e-05 116.22 0.54 0.65 1.38

2048 8.8372e-05 2.2093e-05 185.40 0.46 0.53 1.18

4096 6.6436e-05 3.3218e-05 246.61 0.32 0.30 0.80

8192 7.1468e-05 7.1468e-05 229.25 0.08 0.09 0.13

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 0 1 3

16 2 6 0 1 3

32 2 6 0 1 3

64 2 6 0 1 3

128 2 6 0 1 3

256 2 6 0 1 3

512 2 6 0 1 3

1024 6 2 1 3 0

2048 2 6 3 1 0

4096 1 3 2 6 0

8192 4 9 3 7 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 4 5

16 2 4 5

32 2 3 5

64 2 3 5

128 2 3 5

256 2 3 5

512 1 2 5

1024 2 2 5

2048 1 2 5

4096 1 2 5

8192 1 1 10

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.4247e-02 1.3913e-05 1.15 0.57 0.14 2.03

16 7.2591e-03 1.4178e-05 2.26 0.55 0.13 1.86

32 3.7651e-03 1.4707e-05 4.35 0.52 0.14 1.80

64 2.0036e-03 1.5653e-05 8.18 0.52 0.12 1.79

128 1.1640e-03 1.8188e-05 14.08 0.44 0.09 1.49

256 6.8404e-04 2.1376e-05 23.95 0.43 0.14 1.26

512 4.9585e-04 3.0991e-05 33.04 0.30 0.09 0.96

1024 3.9032e-04 4.8790e-05 41.98 0.31 0.28 0.85

2048 3.5298e-04 8.8245e-05 46.42 0.19 0.15 0.46

4096 3.2000e-04 1.6000e-04 51.20 0.19 0.22 0.35

8192 3.1809e-04 3.1809e-04 51.51 0.17 0.15 0.40

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 0 4 1

16 2 4 0 6 1

32 2 6 0 4 1

64 2 6 4 0 1

128 4 6 2 0 5

256 2 6 4 5 3

512 4 0 6 3 2

1024 2 6 5 3 4

2048 2 4 3 5 6

4096 2 4 3 5 6

8192 4 5 2 8 3

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 4 7

16 2 4 7

32 2 4 7

64 1 3 7

128 3 4 7

256 1 3 7

512 1 3 7

1024 1 2 5

2048 1 2 7

4096 1 1 7

8192 1 1 7

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.3754e-02 1.3432e-05 1.19 0.56 0.14 1.92

16 6.9240e-03 1.3524e-05 2.37 0.56 0.13 1.91

32 3.4957e-03 1.3655e-05 4.69 0.57 0.14 1.88

64 1.7488e-03 1.3663e-05 9.37 0.57 0.15 1.95

128 8.9809e-04 1.4033e-05 18.24 0.55 0.15 1.92

256 4.7043e-04 1.4701e-05 34.83 0.53 0.14 1.80

512 2.5830e-04 1.6144e-05 63.43 0.46 0.14 1.61

1024 1.5138e-04 1.8923e-05 108.23 0.43 0.12 1.32

2048 9.4255e-05 2.3564e-05 173.83 0.39 0.19 1.13

4096 8.6915e-05 4.3457e-05 188.51 0.17 0.11 0.40

8192 6.9255e-05 6.9255e-05 236.57 0.34 0.27 0.76

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 4 0 1

16 2 6 4 0 1

32 6 2 4 1 0

64 2 6 4 1 0

128 6 2 4 0 1

256 2 6 0 4 1

512 6 2 4 0 1

1024 6 2 4 5 3

2048 6 2 4 1 0

4096 3 2 6 4 1

8192 4 6 5 9 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 3 7

16 2 3 7

32 2 3 7

64 1 3 7

128 2 4 7

256 2 2 7

512 2 3 7

1024 1 3 6

2048 2 2 7

4096 1 3 7

8192 1 1 4

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.3668e-02 1.3347e-05 1.20 0.55 0.14 1.98

16 6.8732e-03 1.3424e-05 2.38 0.55 0.13 1.95

32 3.4370e-03 1.3426e-05 4.77 0.55 0.13 2.02

64 1.7451e-03 1.3634e-05 9.39 0.54 0.13 1.98

128 8.9326e-04 1.3957e-05 18.34 0.54 0.13 1.94

256 4.7007e-04 1.4690e-05 34.85 0.49 0.11 1.73

512 2.4835e-04 1.5522e-05 65.97 0.48 0.10 1.66

1024 1.3936e-04 1.7420e-05 117.56 0.44 0.12 1.37

2048 9.0000e-05 2.2500e-05 182.04 0.34 0.09 1.13

4096 6.0840e-05 3.0420e-05 269.29 0.37 0.24 0.89

8192 5.8436e-05 5.8436e-05 280.37 0.35 0.37 0.65

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 0 4 1

16 2 6 0 4 1

32 6 2 4 0 1

64 2 6 4 0 1

128 2 6 4 0 1

256 6 2 4 0 1

512 2 6 4 0 1

1024 2 6 4 3 0

2048 2 6 4 1 0

4096 6 3 2 1 5

8192 4 5 1 8 6

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 4 7

16 2 4 7

32 2 4 7

64 2 4 7

128 2 4 7

256 3 4 7

512 2 3 7

1024 1 3 7

2048 3 3 7

4096 1 1 6

8192 1 1 3

DISCUSSION

The SP3 Winterhawk II processing node consists of four processors. In these tests, we examine MPI communication performance between processors in a node when all processors in the node are communicating. The performance is reported for a single processor pair (logical processors 0 and 1), representing the latency and bandwdith that would typically appear in a highly synchronous communication-intensive operation, such as a collective communication command.