COMMTEST SP3-200 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM SP3-200 SWAP Performance

(unordered swap of 128KB message using MPI and US between nodes)

(performance measured per processor when all processors in node communicating)

Date/Person:	May 25, 2000 / P. Worley
Platform:	IBM SP3 at National Energy Research Scientific Computing Center (Gseaborg)
	256 2-way Winterhawk I SMP nodes (200 MHz POWER3 with 4MB L2 cache, equivalent to RS/6000 Model 260
Environment:	AIX 4.3.2; POE 2.4.0.12
Communication Library:	MPI over the switch using user space
SWAP size:	16384 REAL*8 floating point values each direction
Message size:	Largest - 16384 REAL*8 floating point values
	Smallest - 16 REAL*8 floating point values
Processors:	0 and 2
	1 and 3
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

unordered simple swap

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 73.43 58.40 31.2%

1 iter. 77.90 56.71 30.3%

10 iter. 81.50 57.74 34.3%

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 84.60 57.25 40.3%

1 iter. 86.06 53.36 37.7%

10 iter. 87.08 58.34 36.0%

cache inv. w/overlap 83.60 58.49 35.1%

1 iter. w/overlap 83.67 57.13 35.4%

10 iter. w/overlap 86.78 59.42 35.5%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 83.76 58.37 44.9%

1 iter. 74.19 57.54 36.6%

10 iter. 79.55 58.56 40.0%

cache inv. w/overlap 85.10 58.33 42.3%

1 iter. w/overlap 86.28 58.45 41.1%

10 iter. w/overlap 73.59 58.02 37.2%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 84.70 58.59 45.6%

1 iter. 85.61 60.50 45.7%

10 iter. 78.57 58.24 41.3%

cache inv. w/overlap 83.28 61.86 42.7%

1 iter. w/overlap 83.45 61.26 39.5%

10 iter. w/overlap 72.46 60.60 35.4%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 81.36 113.30 6.4%

1 iter. 86.54 114.64 7.4%

10 iter. 87.29 113.46 7.8%

cache inv. w/overlap 87.32 57.86 12.9%

1 iter. w/overlap 90.04 58.98 10.1%

10 iter. w/overlap 90.66 57.30 11.0%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 87.45 114.47 13.2%

1 iter. 84.86 117.24 6.2%

10 iter. 87.76 116.99 8.2%

cache inv. w/overlap 86.55 59.14 12.5%

1 iter. w/overlap 86.74 66.24 10.9%

10 iter. w/overlap 90.70 60.84 10.2%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 84.02 56.13 43.3%

1 iter. 83.72 56.21 42.8%

10 iter. 75.24 56.77 38.2%

unordered swap using nonblocking sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 83.21 139.20 18.6%

1 iter. 85.54 137.92 17.8%

10 iter. 85.74 137.80 19.0%

cache inv. w/overlap 86.09 143.64 19.8%

1 iter. w/overlap 83.21 143.30 15.9%

10 iter. w/overlap 86.27 146.83 15.4%

unordered swap using nonblocking receive with sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 83.66 140.24 28.1%

1 iter. 85.72 141.74 25.0%

10 iter. 77.29 141.73 22.9%

cache inv. w/overlap 83.29 139.21 25.6%

1 iter. w/overlap 85.18 140.85 24.0%

10 iter. w/overlap 78.82 148.02 20.3%

unordered swap using nonblocking sync. send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 83.61 148.28 27.3%

1 iter. 86.25 146.08 24.1%

10 iter. 85.00 148.81 23.4%

cache inv. w/overlap 83.57 150.08 24.1%

1 iter. w/overlap 73.75 152.87 17.7%

10 iter. w/overlap 72.07 148.62 16.4%

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 6.2103e-02 6.0648e-05 4.22 0.63 0.93 1.56

256 3.2201e-02 6.2892e-05 8.14 0.65 0.92 1.58

512 1.7325e-02 6.7677e-05 15.13 0.62 0.84 1.52

1024 1.1412e-02 8.9156e-05 22.97 0.55 0.66 1.41

2048 7.3073e-03 1.1418e-04 35.87 0.45 0.52 1.15

4096 5.2481e-03 1.6400e-04 49.95 0.37 0.38 1.07

8192 5.2828e-03 3.3017e-04 49.62 0.26 0.34 0.43

16384 4.3214e-03 5.4017e-04 60.66 0.15 0.18 0.27

32768 3.7028e-03 9.2569e-04 70.80 0.07 0.08 0.19

65536 3.3013e-03 1.6506e-03 79.41 0.06 0.06 0.14

131072 2.9978e-03 2.9978e-03 87.45 0.06 0.05 0.19

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 6 1 2 3

256 0 2 1 6 3

512 0 6 1 3 2

1024 1 2 0 6 3

2048 6 0 2 1 3

4096 1 2 6 0 3

8192 4 5 7 0 1

16384 4 5 1 7 0

32768 4 5 1 8 9

65536 4 5 3 7 1

131072 5 3 1 6 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 4 5 5

256 2 4 5

512 3 5 5

1024 2 4 5

2048 2 5 5

4096 4 5 5

8192 2 2 4

16384 1 2 9

32768 2 3 10

65536 1 4 10

131072 1 7 10

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 5.9085e-02 5.7700e-05 4.44 0.69 1.02 1.65

256 3.1739e-02 6.1989e-05 8.26 0.63 0.91 1.58

512 1.6870e-02 6.5898e-05 15.54 0.64 0.89 1.56

1024 1.1207e-02 8.7553e-05 23.39 0.55 0.69 1.37

2048 7.0576e-03 1.1027e-04 37.14 0.47 0.55 1.20

4096 5.0188e-03 1.5684e-04 52.23 0.38 0.44 1.02

8192 5.1401e-03 3.2126e-04 51.00 0.25 0.36 0.44

16384 4.0833e-03 5.1041e-04 64.20 0.16 0.18 0.28

32768 3.4669e-03 8.6673e-04 75.61 0.14 0.15 0.25

65536 3.2384e-03 1.6192e-03 80.95 0.08 0.09 0.19

131072 3.0291e-03 3.0291e-03 86.54 0.04 0.01 0.17

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 1 6 0 2 3

256 0 1 6 2 3

512 1 6 2 0 3

1024 6 2 1 0 3

2048 2 1 6 0 3

4096 6 2 1 3 0

8192 5 4 7 0 1

16384 5 4 1 7 0

32768 5 4 1 7 0

65536 1 5 4 7 0

131072 4 9 1 8 3

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 4 5

256 4 5 5

512 1 5 5

1024 1 4 5

2048 1 5 5

4096 2 5 5

8192 1 2 5

16384 1 2 7

32768 1 2 10

65536 3 4 10

131072 4 8 10

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 6.1137e-02 5.9704e-05 4.29 0.65 0.95 1.58

256 3.1848e-02 6.2204e-05 8.23 0.64 0.92 1.57

512 1.7039e-02 6.6559e-05 15.38 0.64 0.86 1.56

1024 1.1348e-02 8.8659e-05 23.10 0.54 0.66 1.37

2048 7.1740e-03 1.1209e-04 36.54 0.46 0.55 1.20

4096 5.0764e-03 1.5864e-04 51.64 0.38 0.42 1.02

8192 5.2004e-03 3.2502e-04 50.41 0.25 0.36 0.41

16384 4.0916e-03 5.1145e-04 64.07 0.21 0.31 0.37

32768 3.5466e-03 8.8666e-04 73.91 0.16 0.25 0.29

65536 3.2077e-03 1.6038e-03 81.72 0.11 0.17 0.21

131072 2.9871e-03 2.9871e-03 87.76 0.07 0.08 0.17

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 6 0 1 2 3

256 0 6 1 2 3

512 2 6 0 1 3

1024 6 1 2 0 3

2048 0 2 6 1 3

4096 6 1 2 0 3

8192 5 4 7 1 0

16384 4 5 7 1 0

32768 5 4 7 1 0

65536 4 5 7 1 0

131072 5 4 1 7 9

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 5 5

256 2 4 5

512 3 4 5

1024 2 4 5

2048 2 5 5

4096 3 5 5

8192 2 2 5

16384 1 2 5

32768 2 2 6

65536 2 4 10

131072 3 5 10

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 6.1628e-02 6.0184e-05 4.25 0.47 0.05 1.59

256 3.1209e-02 6.0955e-05 8.40 0.51 0.09 1.65

512 1.4942e-02 5.8366e-05 17.54 0.67 0.16 1.95

1024 1.1496e-02 8.9813e-05 22.80 0.40 0.05 1.34

2048 7.2736e-03 1.1365e-04 36.04 0.35 0.04 1.21

4096 5.2289e-03 1.6340e-04 50.13 0.29 0.04 0.95

8192 4.0550e-03 2.5344e-04 64.65 0.56 0.72 0.80

16384 3.6234e-03 4.5293e-04 72.35 0.31 0.35 0.56

32768 3.3838e-03 8.4596e-04 77.47 0.16 0.17 0.28

65536 3.1188e-03 1.5594e-03 84.05 0.09 0.10 0.15

131072 3.0021e-03 3.0021e-03 87.32 0.05 0.04 0.19

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 0 4 6 1

256 6 2 4 0 1

512 3 0 6 1 4

1024 0 6 2 1 4

2048 2 0 4 1 6

4096 6 4 1 2 5

8192 4 5 1 7 0

16384 4 5 7 1 0

32768 4 5 0 7 1

65536 5 4 8 6 1

131072 4 5 7 2 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 3 6 7

256 1 3 7

512 1 1 6

1024 3 5 7

2048 2 6 7

4096 3 6 7

8192 1 2 2

16384 2 2 2

32768 2 2 8

65536 1 2 10

131072 2 8 10

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 6.1147e-02 5.9714e-05 4.29 0.48 0.07 1.63

256 3.1870e-02 6.2246e-05 8.23 0.47 0.06 1.59

512 1.7108e-02 6.6828e-05 15.32 0.48 0.06 1.60

1024 1.1385e-02 8.8949e-05 23.02 0.40 0.06 1.31

2048 7.1167e-03 1.1120e-04 36.84 0.35 0.04 1.17

4096 5.0482e-03 1.5776e-04 51.93 0.30 0.06 0.98

8192 4.1272e-03 2.5795e-04 63.52 0.50 0.63 0.77

16384 3.5516e-03 4.4395e-04 73.81 0.33 0.37 0.55

32768 3.2910e-03 8.2274e-04 79.66 0.19 0.20 0.36

65536 3.0999e-03 1.5499e-03 84.57 0.13 0.15 0.31

131072 2.9113e-03 2.9113e-03 90.04 0.10 0.08 0.23

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 6 2 4 1

256 2 0 4 6 5

512 2 1 6 0 4

1024 4 2 0 1 6

2048 2 6 0 4 1

4096 2 6 1 0 4

8192 5 4 7 1 0

16384 5 4 1 7 0

32768 5 4 1 7 8

65536 4 5 1 8 7

131072 4 5 2 8 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 5 7

256 4 5 7

512 2 5 7

1024 1 5 7

2048 2 6 7

4096 1 3 7

8192 1 2 2

16384 2 2 2

32768 2 2 6

65536 2 2 8

131072 1 3 10

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 6.0196e-02 5.8785e-05 4.35 0.50 0.08 1.60

256 3.1405e-02 6.1338e-05 8.35 0.48 0.07 1.56

512 1.6789e-02 6.5582e-05 15.61 0.47 0.07 1.56

1024 1.1199e-02 8.7493e-05 23.41 0.40 0.06 1.29

2048 7.1428e-03 1.1161e-04 36.70 0.35 0.03 1.14

4096 5.0075e-03 1.5648e-04 52.35 0.29 0.03 1.00

8192 4.0382e-03 2.5239e-04 64.92 0.55 0.74 0.78

16384 3.4657e-03 4.3322e-04 75.64 0.40 0.57 0.66

32768 3.1925e-03 7.9813e-04 82.11 0.28 0.38 0.50

65536 3.0111e-03 1.5056e-03 87.06 0.20 0.27 0.40

131072 2.8904e-03 2.8904e-03 90.70 0.13 0.15 0.26

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 6 2 4 1

256 0 6 2 4 1

512 0 6 4 2 1

1024 6 4 0 2 1

2048 0 6 2 5 1

4096 6 1 4 2 0

8192 4 5 1 7 0

16384 5 4 7 1 0

32768 4 5 1 7 0

65536 5 4 1 7 0

131072 5 4 1 7 0

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 4 7

256 1 5 7

512 2 5 7

1024 1 5 7

2048 1 6 7

4096 2 7 7

8192 2 2 2

16384 2 2 2

32768 2 2 5

65536 2 2 5

131072 2 3 8

DISCUSSION

The SP3 Winterhawk I processing node consists of two processors. MPI between processors within a node uses a shared memory implementation for communication, in contrast to communication between nodes which uses a network interface card and an external switch.