COMMTEST SP3-375 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM SP3-375 Winterhawk II SWAP Performance

(ordered swap of 128KB message using MPI within a node)

(performance measured per processor when all processors in node communicating)

Date/Person:	May 25, 2000 / P. Worley
Platform:	IBM SP3 at Oak Ridge National Laboratory (eagle.ccs.ornl.gov):
	160 4-way Winterhawk II SMP nodes (375MHz POWER3 with 8MB L2 cache)
Environment:	AIX 4.3.3; PSSP 3.1.1
Communication Library:	shared memory MPI within a node, MPI over the switch between nodes, using User Space protocol in both cases
SWAP size:	16384 REAL*8 floating point values each direction
Message size:	Largest - 16384 REAL*8 floating point values
	Smallest - 16 REAL*8 floating point values
Processors:	0 and 1
	2 and 3
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

ordered simple swap

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 87.34 9.59 17.0%

1 iter. 374.83 9.52 38.2%

10 iter. 371.35 9.59 36.7%

ordered swap using nonblocking send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 84.54 10.09 20.2%

1 iter. 363.23 10.17 36.5%

10 iter. 380.56 10.27 37.5%

cache inv. w/overlap 83.27 10.74 18.1%

1 iter. w/overlap 349.87 10.90 35.4%

10 iter. w/overlap 375.91 11.07 37.3%

ordered swap using nonblocking receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 85.88 10.47 20.4%

1 iter. 374.89 10.36 38.6%

10 iter. 363.19 10.49 35.4%

cache inv. w/overlap 86.27 10.51 19.9%

1 iter. w/overlap 373.92 10.90 37.8%

10 iter. w/overlap 377.43 10.69 37.1%

ordered swap using nonblocking send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 83.92 11.31 21.3%

1 iter. 365.17 11.35 35.7%

10 iter. 378.52 11.18 37.0%

cache inv. w/overlap 82.68 11.79 22.0%

1 iter. w/overlap 365.06 12.01 35.8%

10 iter. w/overlap 371.44 12.06 36.9%

ordered swap using ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 86.13 17.73 27.2%

1 iter. 314.15 16.65 17.9%

10 iter. 374.99 16.70 16.6%

cache inv. w/overlap 89.52 11.11 14.7%

1 iter. w/overlap 382.34 11.11 27.8%

10 iter. w/overlap 396.94 11.10 23.4%

ordered swap using nonblocking ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 85.42 18.26 25.2%

1 iter. 370.33 17.25 31.3%

10 iter. 376.94 17.15 18.9%

cache inv. w/overlap 88.73 11.92 14.9%

1 iter. w/overlap 379.80 11.69 30.8%

10 iter. w/overlap 387.76 12.04 21.8%

synchronous

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 86.63 12.81 16.9%

1 iter. 355.01 12.92 34.0%

10 iter. 368.86 13.07 34.0%

ordered swap using nonblocking sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 84.99 23.44 18.8%

1 iter. 351.97 23.37 20.5%

10 iter. 369.54 23.65 15.8%

cache inv. w/overlap 84.73 23.57 17.5%

1 iter. w/overlap 366.63 24.24 22.4%

10 iter. w/overlap 374.71 24.50 18.5%

ordered swap using nonblocking receive with sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 84.62 23.76 19.3%

1 iter. 377.24 23.85 28.9%

10 iter. 376.44 23.94 20.0%

cache inv. w/overlap 83.66 23.93 21.0%

1 iter. w/overlap 368.61 24.50 27.9%

10 iter. w/overlap 377.23 24.30 18.5%

ordered swap using nonblocking sync. send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 84.53 24.50 19.2%

1 iter. 365.76 24.36 25.8%

10 iter. 372.45 24.48 16.0%

cache inv. w/overlap 84.81 25.54 18.0%

1 iter. w/overlap 362.91 25.48 25.9%

10 iter. w/overlap 377.62 26.29 18.0%

ordered simple swap using sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 90.34 22.77 13.6%

1 iter. 365.06 22.76 21.5%

10 iter. 369.63 22.97 19.4%

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.2217e-02 2.1697e-05 11.80 0.66 0.71 1.38

256 1.2395e-02 2.4209e-05 21.15 0.59 0.60 1.24

512 7.1562e-03 2.7954e-05 36.63 0.50 0.41 1.09

1024 4.7002e-03 3.6720e-05 55.77 0.37 0.27 0.83

2048 3.6493e-03 5.7020e-05 71.83 0.22 0.18 0.49

4096 3.0893e-03 9.6539e-05 84.86 0.13 0.10 0.28

8192 3.1282e-03 1.9551e-04 83.80 0.04 0.03 0.09

16384 3.0015e-03 3.7519e-04 87.34 0.03 0.03 0.05

32768 3.0262e-03 7.5654e-04 86.63 0.03 0.02 0.06

65536 3.1402e-03 1.5701e-03 83.48 0.02 0.02 0.06

131072 2.9016e-03 2.9016e-03 90.34 0.15 0.16 0.20

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 1 2 3 6

256 0 1 2 3 6

512 0 1 2 3 6

1024 0 2 1 3 6

2048 2 0 1 3 6

4096 0 1 2 3 6

8192 4 5 0 10 2

16384 0 4 2 5 10

32768 6 2 10 0 1

65536 10 3 0 5 1

131072 10 3 2 9 0

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 4

256 1 1 4

512 1 1 4

1024 1 2 5

2048 3 4 7

4096 1 3 8

8192 1 9 11

16384 1 10 11

32768 1 10 11

65536 4 10 11

131072 1 1 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.0568e-02 2.0086e-05 12.75 0.71 0.76 1.48

256 1.0816e-02 2.1124e-05 24.24 0.69 0.76 1.41

512 5.8535e-03 2.2865e-05 44.78 0.63 0.67 1.28

1024 3.4191e-03 2.6712e-05 76.67 0.50 0.37 1.11

2048 2.1351e-03 3.3361e-05 122.78 0.41 0.28 0.93

4096 1.5426e-03 4.8205e-05 169.94 0.27 0.18 0.61

8192 1.3422e-03 8.3890e-05 195.30 0.21 0.23 0.30

16384 1.1990e-03 1.4988e-04 218.63 0.09 0.11 0.16

32768 1.1473e-03 2.8684e-04 228.48 0.04 0.03 0.10

65536 8.3447e-04 4.1723e-04 314.15 0.05 0.05 0.13

131072 6.9489e-04 6.9489e-04 377.24 0.05 0.03 0.21

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 1 2 3 6

256 0 1 2 3 6

512 0 1 2 3 6

1024 0 1 2 3 6

2048 0 1 2 3 6

4096 0 2 1 3 6

8192 4 5 0 10 1

16384 5 4 10 0 1

32768 10 1 6 7 0

65536 4 1 3 9 0

131072 8 2 0 5 9

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 4

256 1 1 4

512 1 1 4

1024 1 1 4

2048 1 1 5

4096 1 3 7

8192 1 2 7

16384 1 2 11

32768 2 6 11

65536 1 6 11

131072 3 8 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.0635e-02 2.0152e-05 12.70 0.71 0.75 1.48

256 1.0817e-02 2.1128e-05 24.23 0.69 0.76 1.41

512 5.8305e-03 2.2775e-05 44.96 0.64 0.68 1.31

1024 3.3533e-03 2.6198e-05 78.18 0.52 0.42 1.14

2048 2.1166e-03 3.3072e-05 123.85 0.41 0.27 0.91

4096 1.4944e-03 4.6699e-05 175.42 0.29 0.19 0.64

8192 1.3248e-03 8.2803e-05 197.87 0.20 0.24 0.29

16384 1.1248e-03 1.4060e-04 233.06 0.11 0.12 0.15

32768 9.9817e-04 2.4954e-04 262.62 0.07 0.07 0.11

65536 7.9398e-04 3.9699e-04 330.17 0.04 0.05 0.07

131072 6.8884e-04 6.8884e-04 380.56 0.02 0.02 0.05

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 1 2 3 6

256 0 1 2 3 6

512 0 1 2 3 6

1024 0 1 2 3 6

2048 0 1 2 3 6

4096 0 1 2 3 6

8192 4 5 10 0 1

16384 4 5 0 1 10

32768 4 5 7 1 9

65536 4 5 10 0 1

131072 1 3 5 8 4

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 4

256 1 1 4

512 1 1 4

1024 1 1 4

2048 1 1 5

4096 1 3 7

8192 1 2 8

16384 2 2 11

32768 1 2 11

65536 2 6 11

131072 3 11 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.2078e-02 2.1561e-05 11.87 0.60 0.23 1.48

256 1.2404e-02 2.4227e-05 21.13 0.54 0.21 1.30

512 7.2183e-03 2.8196e-05 36.32 0.46 0.16 1.12

1024 4.7753e-03 3.7307e-05 54.90 0.35 0.12 0.86

2048 3.6071e-03 5.6361e-05 72.67 0.22 0.08 0.54

4096 3.1348e-03 9.7962e-05 83.62 0.12 0.05 0.28

8192 3.0303e-03 1.8939e-04 86.51 0.09 0.12 0.13

16384 2.9284e-03 3.6605e-04 89.52 0.05 0.06 0.08

32768 3.0386e-03 7.5965e-04 86.27 0.03 0.03 0.07

65536 3.0627e-03 1.5313e-03 85.59 0.03 0.03 0.07

131072 2.9304e-03 2.9304e-03 89.46 0.09 0.10 0.13

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 1 4 5

256 0 2 1 4 5

512 0 2 4 1 5

1024 0 2 1 4 5

2048 0 2 4 1 3

4096 0 2 3 5 4

8192 4 5 0 10 6

16384 4 5 0 10 2

32768 2 0 10 4 9

65536 4 6 10 0 8

131072 10 0 5 6 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 6

256 1 1 6

512 1 1 6

1024 1 2 7

2048 1 2 7

4096 1 5 10

8192 1 3 11

16384 2 4 11

32768 4 10 11

65536 1 10 11

131072 1 1 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.0807e-02 2.0320e-05 12.60 0.64 0.25 1.57

256 1.0983e-02 2.1452e-05 23.87 0.61 0.27 1.49

512 5.9099e-03 2.3086e-05 44.36 0.58 0.23 1.38

1024 3.4795e-03 2.7183e-05 75.34 0.48 0.18 1.17

2048 2.1618e-03 3.3778e-05 121.26 0.40 0.17 0.96

4096 1.5503e-03 4.8447e-05 169.09 0.26 0.09 0.65

8192 1.3261e-03 8.2879e-05 197.69 0.24 0.28 0.32

16384 1.0837e-03 1.3547e-04 241.89 0.19 0.22 0.27

32768 1.0726e-03 2.6814e-04 244.41 0.11 0.11 0.17

65536 8.0819e-04 4.0410e-04 324.36 0.06 0.07 0.10

131072 6.8564e-04 6.8564e-04 382.34 0.03 0.02 0.09

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 1 4 5

256 0 2 1 4 3

512 0 2 1 4 5

1024 0 2 1 4 5

2048 0 2 4 1 5

4096 0 2 1 4 5

8192 4 5 10 0 2

16384 4 5 10 0 8

32768 4 5 10 7 1

65536 4 5 10 7 2

131072 4 5 6 2 0

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 6

256 1 1 5

512 1 1 6

1024 1 1 6

2048 1 1 6

4096 1 4 7

8192 1 2 4

16384 1 1 9

32768 1 1 11

65536 1 2 11

131072 2 9 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.0772e-02 2.0285e-05 12.62 0.65 0.24 1.65

256 1.0898e-02 2.1286e-05 24.05 0.62 0.23 1.58

512 5.9012e-03 2.3052e-05 44.42 0.57 0.21 1.46

1024 3.3709e-03 2.6335e-05 77.77 0.50 0.19 1.28

2048 2.1342e-03 3.3347e-05 122.83 0.39 0.15 1.01

4096 1.4914e-03 4.6606e-05 175.77 0.29 0.12 0.72

8192 1.2384e-03 7.7397e-05 211.69 0.29 0.34 0.40

16384 1.0739e-03 1.3424e-04 244.10 0.16 0.19 0.23

32768 9.7766e-04 2.4441e-04 268.13 0.11 0.13 0.18

65536 7.6576e-04 3.8288e-04 342.33 0.08 0.09 0.11

131072 6.6040e-04 6.6040e-04 396.94 0.05 0.05 0.07

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 1 4 5

256 0 2 1 4 5

512 0 2 1 4 5

1024 0 2 1 4 5

2048 0 2 1 4 5

4096 0 2 1 4 5

8192 4 5 0 10 2

16384 5 4 0 10 1

32768 4 5 0 8 7

65536 4 5 8 0 7

131072 4 5 0 9 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 6

256 1 1 6

512 1 1 6

1024 1 1 6

2048 1 1 7

4096 1 1 7

8192 1 2 2

16384 2 2 11

32768 1 2 11

65536 1 2 11

131072 1 3 11

DISCUSSION

The SP3 Winterhawk II processing node consists of four processors. In these tests, we examine MPI communication performance between processors in a node when all processors in the node are communicating. The performance is reported for a single processor pair (logical processors 0 and 1), representing the latency and bandwdith that would typically appear in a highly synchronous communication-intensive operation, such as a collective communication command.