COMMTEST SP3-375 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM SP3-375 Winterhawk II SWAP Performance

(unordered swap of 2MB message using MPI within a node)

(performance measured per processor when all processors in node communicating)

Date/Person:	May 25, 2000 / P. Worley
Platform:	IBM SP3 at Oak Ridge National Laboratory (eagle.ccs.ornl.gov):
	160 4-way Winterhawk II SMP nodes (375MHz POWER3 with 8MB L2 cache)
Environment:	AIX 4.3.3; PSSP 3.1.1
Communication Library:	shared memory MPI within a node, MPI over the switch between nodes, using User Space protocol in both cases
SWAP size:	262144 REAL*8 floating point values each direction
Message size:	Largest - 262144 REAL*8 floating point values
	Smallest - 256 REAL*8 floating point values
Processors:	0 and 1
	2 and 3
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

unordered simple swap

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 156.54 12.22 33.9%

1 iter. 249.83 12.15 45.7%

10 iter. 335.95 12.22 45.6%

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 174.71 11.85 14.2%

1 iter. 286.54 13.18 19.6%

10 iter. 298.07 13.16 27.3%

cache inv. w/overlap 179.57 12.94 15.1%

1 iter. w/overlap 295.96 13.56 22.5%

10 iter. w/overlap 287.79 13.55 17.4%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 180.25 8.87 21.5%

1 iter. 287.76 11.10 26.0%

10 iter. 369.74 11.44 31.6%

cache inv. w/overlap 175.76 10.34 18.8%

1 iter. w/overlap 302.91 11.40 27.5%

10 iter. w/overlap 376.23 10.73 33.9%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 177.03 11.23 19.0%

1 iter. 295.62 14.06 24.3%

10 iter. 369.28 13.26 30.5%

cache inv. w/overlap 176.86 12.38 17.2%

1 iter. w/overlap 298.75 14.24 24.8%

10 iter. w/overlap 419.90 13.46 37.5%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 174.21 20.76 2.7%

1 iter. 284.62 23.28 5.0%

10 iter. 287.30 22.83 21.8%

cache inv. w/overlap 179.86 9.58 6.1%

1 iter. w/overlap 307.56 12.29 7.9%

10 iter. w/overlap 348.29 11.70 27.8%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 176.00 22.09 4.4%

1 iter. 305.56 24.77 9.6%

10 iter. 293.12 24.51 19.5%

cache inv. w/overlap 178.68 12.56 6.1%

1 iter. w/overlap 301.52 13.58 7.1%

10 iter. w/overlap 373.39 12.91 32.3%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 173.84 9.47 16.9%

1 iter. 288.48 12.40 23.0%

10 iter. 372.65 11.66 33.0%

unordered swap using nonblocking sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 181.73 32.78 9.9%

1 iter. 285.02 32.57 3.7%

10 iter. 295.64 32.27 20.5%

cache inv. w/overlap 170.79 33.87 6.7%

1 iter. w/overlap 286.23 34.95 5.0%

10 iter. w/overlap 300.86 31.91 20.0%

unordered swap using nonblocking receive with sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 176.05 32.99 3.7%

1 iter. 286.15 33.12 3.7%

10 iter. 372.32 32.87 28.6%

cache inv. w/overlap 177.25 35.69 5.4%

1 iter. w/overlap 291.26 35.78 3.9%

10 iter. w/overlap 397.29 33.00 29.8%

unordered swap using nonblocking sync. send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 175.85 36.44 3.0%

1 iter. 298.10 39.72 5.0%

10 iter. 398.23 36.87 30.0%

cache inv. w/overlap 175.77 40.96 10.7%

1 iter. w/overlap 313.73 42.44 9.2%

10 iter. w/overlap 398.60 40.28 27.5%

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 3.3259e-02 3.2480e-05 126.11 0.31 0.34 0.80

4096 2.8229e-02 5.5134e-05 148.58 0.18 0.20 0.46

8192 2.8848e-02 1.1269e-04 145.39 0.10 0.12 0.14

16384 2.6013e-02 2.0323e-04 161.24 0.09 0.10 0.17

32768 2.5620e-02 4.0032e-04 163.71 0.04 0.03 0.12

65536 2.4232e-02 7.5726e-04 173.09 0.06 0.05 0.14

131072 2.3995e-02 1.4997e-03 174.80 0.04 0.04 0.15

262144 2.4007e-02 3.0008e-03 174.71 0.03 0.03 0.12

524288 2.3334e-02 5.8335e-03 179.75 0.06 0.03 0.31

1048576 2.3831e-02 1.1916e-02 176.00 0.09 0.02 0.70

2097152 2.3080e-02 2.3080e-02 181.73 0.12 0.07 0.71

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 6 2 3 1 0

4096 1 3 6 2 0

8192 4 5 1 7 6

16384 5 4 1 7 9

32768 4 1 8 3 7

65536 4 5 1 6 2

131072 3 6 2 9 8

262144 1 3 9 5 7

524288 7 3 8 5 9

1048576 5 9 7 8 4

2097152 7 2 6 4 9

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 2 4 5

4096 3 4 7

8192 2 2 10

16384 2 2 10

32768 2 6 10

65536 1 5 10

131072 2 8 10

262144 2 9 10

524288 1 7 9

1048576 4 9 9

2097152 2 3 9

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 2.6378e-02 2.5760e-05 159.01 0.39 0.43 1.01

4096 2.0692e-02 4.0415e-05 202.70 0.23 0.25 0.58

8192 2.0519e-02 8.0150e-05 204.42 0.11 0.12 0.18

16384 1.6951e-02 1.3243e-04 247.43 0.12 0.13 0.20

32768 1.5672e-02 2.4487e-04 267.63 0.09 0.09 0.19

65536 1.5119e-02 4.7247e-04 277.42 0.04 0.05 0.11

131072 1.4737e-02 9.2104e-04 284.62 0.05 0.05 0.18

262144 1.4070e-02 1.7588e-03 298.10 0.10 0.09 0.43

524288 1.4188e-02 3.5470e-03 295.62 0.09 0.08 0.32

1048576 1.3727e-02 6.8634e-03 305.56 0.19 0.09 1.26

2097152 1.4658e-02 1.4658e-02 286.15 0.11 0.04 0.88

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 2 6 3 1 0

4096 2 6 3 1 0

8192 4 5 1 7 8

16384 4 5 7 1 8

32768 4 5 1 9 3

65536 5 7 9 6 4

131072 4 9 3 8 6

262144 9 3 2 8 6

524288 3 9 8 5 6

1048576 5 9 6 1 2

2097152 8 7 4 9 5

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 1 2 5

4096 3 4 6

8192 2 2 10

16384 1 2 10

32768 1 2 10

65536 1 6 10

131072 1 6 10

262144 1 3 9

524288 2 3 9

1048576 1 1 9

2097152 2 8 9

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 2.3528e-02 2.2976e-05 178.27 0.42 0.48 1.08

4096 1.7673e-02 3.4517e-05 237.33 0.27 0.31 0.70

8192 1.8384e-02 7.1811e-05 228.15 0.11 0.14 0.16

16384 1.6046e-02 1.2536e-04 261.39 0.09 0.11 0.12

32768 1.4249e-02 2.2265e-04 294.35 0.06 0.08 0.13

65536 1.5330e-02 4.7907e-04 273.60 0.10 0.11 0.22

131072 1.1255e-02 7.0345e-04 372.65 0.25 0.11 0.75

262144 1.1392e-02 1.4240e-03 368.18 0.28 0.37 0.60

524288 1.0532e-02 2.6331e-03 398.23 0.35 0.35 0.70

1048576 1.0730e-02 5.3648e-03 390.91 0.38 0.34 1.03

2097152 1.1641e-02 1.1641e-02 360.30 0.29 0.26 0.97

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 2 6 3 1 0

4096 2 6 3 1 0

8192 5 4 1 2 6

16384 4 5 9 3 6

32768 2 5 3 4 8

65536 0 3 1 7 9

131072 6 8 9 2 3

262144 3 2 8 6 9

524288 9 8 2 6 3

1048576 9 6 2 3 1

2097152 8 6 9 7 5

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 2 2 5

4096 2 4 5

8192 2 2 10

16384 1 2 10

32768 2 4 10

65536 1 2 10

131072 5 5 6

262144 2 2 5

524288 1 1 3

1048576 1 1 3

2097152 1 1 4

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 3.3324e-02 3.2543e-05 125.87 0.25 0.05 0.84

4096 2.8062e-02 5.4809e-05 149.46 0.14 0.03 0.44

8192 2.6106e-02 1.0198e-04 160.67 0.19 0.24 0.26

16384 2.4849e-02 1.9413e-04 168.79 0.12 0.14 0.21

32768 2.4275e-02 3.7929e-04 172.79 0.08 0.10 0.15

65536 2.3508e-02 7.3463e-04 178.42 0.08 0.08 0.18

131072 2.3879e-02 1.4924e-03 175.65 0.04 0.02 0.16

262144 2.3981e-02 2.9976e-03 174.90 0.04 0.03 0.17

524288 2.3578e-02 5.8946e-03 177.89 0.06 0.04 0.34

1048576 2.3320e-02 1.1660e-02 179.86 0.08 0.05 0.50

2097152 2.3474e-02 2.3474e-02 178.68 0.10 0.04 0.65

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 6 2 4 3 5

4096 2 5 3 1 4

8192 4 5 1 6 7

16384 5 4 7 1 9

32768 4 5 6 1 7

65536 5 4 1 6 7

131072 8 9 2 4 5

262144 5 3 6 9 1

524288 5 3 8 4 9

1048576 4 1 8 3 6

2097152 5 9 2 6 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 3 5 7

4096 3 6 7

8192 1 2 8

16384 2 2 10

32768 1 2 10

65536 1 2 10

131072 3 8 10

262144 1 9 10

524288 1 8 9

1048576 2 7 9

2097152 1 7 9

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 2.6771e-02 2.6143e-05 156.68 0.31 0.08 1.05

4096 2.0658e-02 4.0348e-05 203.03 0.19 0.06 0.60

8192 1.7622e-02 6.8836e-05 238.02 0.25 0.31 0.36

16384 1.5638e-02 1.2217e-04 268.22 0.18 0.22 0.27

32768 1.5234e-02 2.3803e-04 275.33 0.09 0.11 0.21

65536 1.3638e-02 4.2617e-04 307.56 0.12 0.15 0.23

131072 1.4172e-02 8.8575e-04 295.96 0.06 0.04 0.21

262144 1.3899e-02 1.7374e-03 301.77 0.07 0.05 0.27

524288 1.4037e-02 3.5092e-03 298.81 0.07 0.04 0.39

1048576 1.3847e-02 6.9233e-03 302.91 0.17 0.07 1.09

2097152 1.3369e-02 1.3369e-02 313.73 0.18 0.11 1.04

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 2 6 4 5 3

4096 6 4 2 3 5

8192 5 4 7 1 6

16384 5 4 1 7 9

32768 5 4 1 3 7

65536 4 5 1 7 6

131072 1 8 3 2 4

262144 9 2 3 8 5

524288 9 3 1 5 2

1048576 2 9 1 8 4

2097152 9 6 3 2 4

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 2 3 7

4096 2 5 7

8192 2 2 2

16384 1 2 9

32768 2 2 10

65536 1 2 10

131072 1 7 10

262144 2 6 9

524288 2 6 9

1048576 1 2 9

2097152 1 2 9

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 2.3445e-02 2.2896e-05 178.90 0.34 0.09 1.17

4096 1.7765e-02 3.4698e-05 236.09 0.21 0.05 0.70

8192 1.5943e-02 6.2277e-05 263.08 0.26 0.32 0.35

16384 1.5461e-02 1.2079e-04 271.28 0.11 0.13 0.16

32768 1.4598e-02 2.2809e-04 287.32 0.05 0.07 0.10

65536 1.1233e-02 3.5103e-04 373.39 0.36 0.46 0.58

131072 1.1148e-02 6.9676e-04 376.23 0.18 0.24 0.59

262144 1.0523e-02 1.3153e-03 398.60 0.32 0.39 0.69

524288 9.9888e-03 2.4972e-03 419.90 0.41 0.55 0.72

1048576 1.0963e-02 5.4814e-03 382.60 0.27 0.30 0.94

2097152 1.1143e-02 1.1143e-02 376.39 0.25 0.21 1.22

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 2 6 4 5 3

4096 4 6 2 3 5

8192 4 5 1 6 7

16384 4 5 3 9 8

32768 4 9 5 8 3

65536 5 4 0 9 3

131072 2 8 6 9 3

262144 9 8 3 6 2

524288 3 6 9 8 2

1048576 9 3 8 6 2

2097152 9 3 2 6 8

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 2 3 7

4096 2 6 7

8192 1 2 2

16384 2 2 10

32768 1 5 10

65536 1 1 3

131072 2 5 6

262144 2 3 4

524288 1 2 3

1048576 1 5 5

2097152 1 3 6

DISCUSSION

The SP3 Winterhawk II processing node consists of four processors. In these tests, we examine MPI communication performance between processors in a node when all processors in the node are communicating. The performance is reported for a single processor pair (logical processors 0 and 1), representing the latency and bandwdith that would typically appear in a highly synchronous communication-intensive operation, such as a collective communication command.