COMMTEST SP3-375 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM SP3-375 Winterhawk II SWAP Performance

(unordered swap of 128KB message using MPI within a node)

(performance measured per processor when all processors in node communicating)

Date/Person:	May 25, 2000 / P. Worley
Platform:	IBM SP3 at Oak Ridge National Laboratory (eagle.ccs.ornl.gov):
	160 4-way Winterhawk II SMP nodes (375MHz POWER3 with 8MB L2 cache)
Environment:	AIX 4.3.3; PSSP 3.1.1
Communication Library:	shared memory MPI within a node, MPI over the switch between nodes, using User Space protocol in both cases
SWAP size:	16384 REAL*8 floating point values each direction
Message size:	Largest - 16384 REAL*8 floating point values
	Smallest - 16 REAL*8 floating point values
Processors:	0 and 1
	2 and 3
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

unordered simple swap

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 82.92 13.42 21.1%

1 iter. 267.26 13.59 56.6%

10 iter. 252.29 13.77 25.3%

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 90.60 13.91 17.1%

1 iter. 258.05 14.05 46.2%

10 iter. 279.86 14.20 27.7%

cache inv. w/overlap 90.00 14.29 18.8%

1 iter. w/overlap 209.36 14.72 36.5%

10 iter. w/overlap 265.13 14.29 29.1%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 89.52 12.90 14.0%

1 iter. 269.72 13.18 50.1%

10 iter. 309.27 13.16 23.7%

cache inv. w/overlap 91.83 13.30 18.1%

1 iter. w/overlap 232.86 13.27 37.7%

10 iter. w/overlap 352.92 13.17 25.4%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 89.15 14.71 20.2%

1 iter. 281.14 14.88 50.7%

10 iter. 326.56 14.84 26.7%

cache inv. w/overlap 90.34 15.08 21.3%

1 iter. w/overlap 287.70 15.15 34.4%

10 iter. w/overlap 334.91 15.14 23.2%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 89.63 23.63 13.1%

1 iter. 230.19 24.55 40.8%

10 iter. 317.79 24.38 41.8%

cache inv. w/overlap 91.87 13.47 16.0%

1 iter. w/overlap 314.19 13.86 44.5%

10 iter. w/overlap 369.79 13.66 15.5%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 90.22 25.31 13.9%

1 iter. 278.62 26.15 50.0%

10 iter. 272.27 25.80 34.1%

cache inv. w/overlap 91.70 15.07 19.5%

1 iter. w/overlap 295.60 15.64 40.6%

10 iter. w/overlap 389.82 15.12 27.7%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 90.48 13.36 17.4%

1 iter. 311.72 13.32 37.1%

10 iter. 334.82 13.29 23.3%

unordered swap using nonblocking sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 89.84 34.28 11.8%

1 iter. 230.32 34.23 38.7%

10 iter. 313.97 34.07 39.2%

cache inv. w/overlap 89.16 34.74 14.6%

1 iter. w/overlap 289.32 35.52 39.1%

10 iter. w/overlap 243.64 33.81 28.0%

unordered swap using nonblocking receive with sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 89.49 35.05 12.9%

1 iter. 265.56 35.58 49.5%

10 iter. 323.15 34.96 24.5%

cache inv. w/overlap 89.86 36.95 13.5%

1 iter. w/overlap 299.92 35.49 43.0%

10 iter. w/overlap 334.29 34.66 33.2%

unordered swap using nonblocking sync. send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 90.05 38.38 13.9%

1 iter. 274.56 39.57 49.1%

10 iter. 353.49 38.38 37.5%

cache inv. w/overlap 91.64 41.85 18.0%

1 iter. w/overlap 274.80 40.51 26.7%

10 iter. w/overlap 341.13 40.60 23.7%

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.5387e-02 1.5026e-05 17.04 0.63 0.70 1.71

256 8.7804e-03 1.7149e-05 29.86 0.55 0.61 1.51

512 5.3818e-03 2.1023e-05 48.71 0.46 0.50 1.22

1024 3.9649e-03 3.0976e-05 66.12 0.29 0.34 0.74

2048 3.3281e-03 5.2001e-05 78.77 0.16 0.18 0.42

4096 3.0778e-03 9.6180e-05 85.17 0.09 0.09 0.22

8192 3.1209e-03 1.9505e-04 84.00 0.04 0.04 0.07

16384 3.0659e-03 3.8323e-04 85.50 0.01 0.01 0.04

32768 2.9455e-03 7.3638e-04 89.00 0.03 0.03 0.09

65536 2.9057e-03 1.4529e-03 90.22 0.03 0.01 0.17

131072 2.8935e-03 2.8935e-03 90.60 0.04 0.01 0.29

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 6 0 1 3

256 2 6 1 0 3

512 6 2 0 1 3

1024 6 1 3 0 2

2048 1 2 0 6 3

4096 6 2 3 1 0

8192 5 4 2 3 7

16384 7 6 5 4 2

32768 4 2 3 6 5

65536 5 1 6 2 8

131072 1 6 9 7 4

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 4 5

256 2 4 5

512 1 4 5

1024 2 5 5

2048 1 5 7

4096 1 5 10

8192 1 9 10

16384 4 10 10

32768 1 9 10

65536 5 9 10

131072 4 9 9

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.4174e-02 1.3842e-05 18.49 0.69 0.82 1.85

256 7.4256e-03 1.4503e-05 35.30 0.65 0.78 1.70

512 3.9862e-03 1.5571e-05 65.76 0.62 0.73 1.63

1024 2.2332e-03 1.7447e-05 117.39 0.57 0.67 1.48

2048 1.4634e-03 2.2866e-05 179.13 0.41 0.46 1.08

4096 1.0740e-03 3.3564e-05 244.07 0.28 0.28 0.67

8192 1.0898e-03 6.8112e-05 240.55 0.10 0.13 0.19

16384 9.4085e-04 1.1761e-04 278.62 0.15 0.15 0.53

32768 9.3245e-04 2.3311e-04 281.14 0.35 0.37 0.64

65536 8.4096e-04 4.2048e-04 311.72 0.73 0.85 1.36

131072 1.3588e-03 1.3588e-03 192.92 0.42 0.44 0.69

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 6 0 1 3

256 2 6 0 1 3

512 2 6 1 0 3

1024 2 6 3 0 1

2048 6 2 3 1 0

4096 6 2 0 1 3

8192 5 2 4 0 8

16384 5 9 2 8 6

32768 3 0 6 9 8

65536 6 1 9 7 2

131072 6 7 1 3 5

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 3 5

256 2 3 5

512 2 2 5

1024 1 2 5

2048 2 4 5

4096 1 2 5

8192 1 2 10

16384 1 4 9

32768 1 1 3

65536 1 1 2

131072 1 1 1

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.4086e-02 1.3756e-05 18.61 0.69 0.81 1.84

256 7.3509e-03 1.4357e-05 35.66 0.66 0.77 1.76

512 3.9272e-03 1.5341e-05 66.75 0.63 0.73 1.66

1024 2.2404e-03 1.7503e-05 117.01 0.55 0.64 1.44

2048 1.4320e-03 2.2375e-05 183.06 0.43 0.49 1.12

4096 1.0262e-03 3.2070e-05 255.44 0.33 0.36 0.79

8192 1.0494e-03 6.5586e-05 249.81 0.19 0.22 0.26

16384 1.0214e-03 1.2767e-04 256.65 0.05 0.03 0.18

32768 8.2489e-04 2.0622e-04 317.79 0.17 0.17 0.39

65536 9.3671e-04 4.6836e-04 279.86 0.21 0.22 0.48

131072 7.4160e-04 7.4160e-04 353.49 0.45 0.53 1.02

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 6 0 1 3

256 2 6 0 1 3

512 2 6 0 1 3

1024 2 6 3 1 0

2048 2 6 3 1 0

4096 2 1 6 3 0

8192 5 4 7 1 9

16384 5 9 0 1 6

32768 4 7 6 8 2

65536 1 6 5 4 3

131072 9 6 3 8 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 3 5

256 2 3 5

512 2 2 5

1024 2 2 5

2048 2 2 5

4096 1 1 5

8192 1 1 8

16384 1 7 10

32768 1 2 8

65536 1 2 8

131072 1 1 5

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.5502e-02 1.5138e-05 16.91 0.52 0.12 1.84

256 8.6901e-03 1.6973e-05 30.17 0.47 0.11 1.59

512 5.2987e-03 2.0698e-05 49.47 0.38 0.08 1.29

1024 3.9511e-03 3.0868e-05 66.35 0.24 0.02 0.87

2048 3.3401e-03 5.2189e-05 78.48 0.13 0.03 0.41

4096 3.0773e-03 9.6167e-05 85.19 0.07 0.02 0.21

8192 2.9976e-03 1.8735e-04 87.45 0.08 0.10 0.14

16384 2.9444e-03 3.6805e-04 89.03 0.04 0.05 0.09

32768 2.8718e-03 7.1795e-04 91.28 0.05 0.04 0.13

65536 2.8546e-03 1.4273e-03 91.83 0.05 0.04 0.20

131072 2.8533e-03 2.8533e-03 91.87 0.04 0.02 0.32

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 6 4 0 1

256 2 4 6 0 1

512 2 4 6 1 0

1024 6 3 4 2 5

2048 4 2 0 6 1

4096 4 3 0 5 2

8192 5 4 6 2 9

16384 5 4 2 8 3

32768 5 4 2 9 3

65536 2 5 3 4 9

131072 4 2 5 9 3

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 4 7

256 1 4 7

512 1 4 7

1024 2 7 7

2048 3 7 7

4096 3 6 10

8192 2 2 10

16384 1 5 10

32768 2 7 10

65536 2 7 10

131072 4 9 9

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.4313e-02 1.3977e-05 18.32 0.56 0.14 1.92

256 7.5188e-03 1.4685e-05 34.87 0.52 0.14 1.81

512 4.0494e-03 1.5818e-05 64.74 0.48 0.11 1.66

1024 2.2803e-03 1.7815e-05 114.96 0.45 0.13 1.49

2048 1.4609e-03 2.2826e-05 179.45 0.35 0.10 1.17

4096 1.0514e-03 3.2856e-05 249.33 0.29 0.16 0.77

8192 8.3436e-04 5.2148e-05 314.19 0.47 0.53 0.76

16384 8.7404e-04 1.0926e-04 299.92 0.32 0.43 0.56

32768 9.0670e-04 2.2668e-04 289.12 0.31 0.49 0.55

65536 1.1128e-03 5.5638e-04 235.58 0.24 0.25 0.43

131072 1.3240e-03 1.3240e-03 197.99 0.16 0.17 0.29

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 6 0 4 1

256 2 6 0 4 1

512 6 2 4 0 1

1024 6 2 4 0 3

2048 2 6 4 3 5

4096 4 6 2 3 0

8192 4 5 7 3 0

16384 8 7 9 3 4

32768 7 3 6 9 8

65536 3 5 4 9 1

131072 9 3 5 4 7

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 4 7

256 2 4 7

512 2 4 7

1024 2 3 7

2048 1 3 7

4096 1 2 6

8192 1 1 2

16384 1 2 4

32768 2 3 4

65536 1 1 6

131072 1 1 9

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.4246e-02 1.3912e-05 18.40 0.54 0.13 1.98

256 7.4654e-03 1.4581e-05 35.11 0.52 0.12 1.91

512 3.9777e-03 1.5538e-05 65.90 0.49 0.12 1.74

1024 2.2659e-03 1.7702e-05 115.69 0.43 0.10 1.48

2048 1.4446e-03 2.2572e-05 181.46 0.33 0.08 1.15

4096 1.0751e-03 3.3597e-05 243.83 0.22 0.06 0.72

8192 9.6553e-04 6.0346e-05 271.50 0.27 0.32 0.38

16384 7.6544e-04 9.5680e-05 342.48 0.34 0.40 0.46

32768 7.8759e-04 1.9690e-04 332.85 0.21 0.29 0.46

65536 6.7247e-04 3.3623e-04 389.82 0.48 0.52 0.90

131072 7.4278e-04 7.4278e-04 352.92 0.32 0.15 0.93

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 6 4 0 1

256 6 2 4 0 1

512 2 6 4 0 1

1024 6 2 4 0 1

2048 4 2 6 5 1

4096 6 5 4 3 2

8192 4 5 0 7 1

16384 4 5 2 3 6

32768 9 6 8 4 3

65536 5 4 2 1 3

131072 2 9 3 8 6

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 4 7

256 2 4 7

512 2 4 7

1024 2 3 7

2048 2 3 7

4096 2 5 7

8192 1 2 2

16384 1 1 2

32768 1 1 5

65536 1 1 2

131072 1 2 7

DISCUSSION

The SP3 Winterhawk II processing node consists of four processors. In these tests, we examine MPI communication performance between processors in a node when all processors in the node are communicating. The performance is reported for a single processor pair (logical processors 0 and 1), representing the latency and bandwdith that would typically appear in a highly synchronous communication-intensive operation, such as a collective communication command.