COMMTEST SP3-200 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM SP3-200 SWAP Performance

(ordered swap of 128KB message using MPI and US between nodes)

(performance measured per processor when all processors in node communicating)

May 25, 2000 / P. Worley

Date/Person:
Platform:	IBM SP3 at National Energy Research Scientific Computing Center (Gseaborg)
	256 2-way Winterhawk I SMP nodes (200 MHz POWER3 with 4MB L2 cache, equivalent to RS/6000 Model 260
Environment:	AIX 4.3.2; POE 2.4.0.12
Communication Library:	MPI over the switch using user space
SWAP size:	16384 REAL*8 floating point values each direction
Message size:	Largest - 16384 REAL*8 floating point values
	Smallest - 16 REAL*8 floating point values
Processors:	0 and 2
	1 and 3
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

ordered simple swap

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 67.69 47.49 41.9%

1 iter. 67.35 44.03 42.0%

10 iter. 68.69 46.80 42.3%

ordered swap using nonblocking send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 66.53 48.07 41.6%

1 iter. 68.29 50.48 41.7%

10 iter. 67.66 48.97 41.6%

cache inv. w/overlap 64.28 47.92 42.0%

1 iter. w/overlap 67.07 48.02 42.0%

10 iter. w/overlap 68.23 50.50 41.2%

ordered swap using nonblocking receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 64.75 46.28 40.4%

1 iter. 67.65 49.40 41.2%

10 iter. 65.37 50.59 39.8%

cache inv. w/overlap 64.83 46.50 41.6%

1 iter. w/overlap 67.57 48.91 41.4%

10 iter. w/overlap 67.84 49.72 40.7%

ordered swap using nonblocking send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 65.89 51.02 41.0%

1 iter. 67.90 47.45 41.2%

10 iter. 67.72 52.09 41.4%

cache inv. w/overlap 65.76 53.10 41.7%

1 iter. w/overlap 68.31 54.81 41.5%

10 iter. w/overlap 68.12 51.13 41.7%

ordered swap using ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 66.47 54.38 22.6%

1 iter. 68.85 52.39 23.7%

10 iter. 70.80 57.63 21.8%

cache inv. w/overlap 68.34 48.41 23.7%

1 iter. w/overlap 69.71 51.02 21.0%

10 iter. w/overlap 70.84 51.56 23.2%

ordered swap using nonblocking ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 66.32 56.45 22.2%

1 iter. 70.40 56.18 24.0%

10 iter. 70.83 57.80 22.2%

cache inv. w/overlap 68.86 52.62 22.3%

1 iter. w/overlap 69.93 54.73 20.6%

10 iter. w/overlap 70.95 51.28 22.4%

synchronous

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 65.11 51.34 39.8%

1 iter. 67.19 50.16 40.8%

10 iter. 67.45 51.25 40.4%

ordered swap using nonblocking sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 54.67 121.61 10.3%

1 iter. 67.23 119.99 16.1%

10 iter. 68.30 123.43 16.2%

cache inv. w/overlap 65.31 127.05 14.3%

1 iter. w/overlap 67.42 123.94 16.2%

10 iter. w/overlap 67.71 124.17 16.1%

ordered swap using nonblocking receive with sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 65.47 122.68 17.0%

1 iter. 68.95 124.12 15.7%

10 iter. 67.51 124.94 15.3%

cache inv. w/overlap 65.42 123.13 18.4%

1 iter. w/overlap 67.92 125.43 16.3%

10 iter. w/overlap 68.52 125.73 15.1%

ordered swap using nonblocking sync. send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 65.93 126.02 19.7%

1 iter. 67.68 125.52 16.4%

10 iter. 67.92 125.07 16.9%

cache inv. w/overlap 66.01 130.96 15.7%

1 iter. w/overlap 67.37 127.57 15.3%

10 iter. w/overlap 67.84 125.19 15.7%

ordered simple swap using sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 66.22 116.06 18.6%

1 iter. 68.08 122.13 16.6%

10 iter. 68.32 121.50 15.4%

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.0149e-01 9.9115e-05 2.58 0.60 0.16 1.57

256 5.2491e-02 1.0252e-04 4.99 0.59 0.17 1.52

512 2.7755e-02 1.0842e-04 9.45 0.60 0.24 1.51

1024 1.8266e-02 1.4271e-04 14.35 0.48 0.20 1.20

2048 1.2364e-02 1.9319e-04 21.20 0.33 0.13 0.83

4096 8.8817e-03 2.7755e-04 29.52 0.22 0.06 0.58

8192 7.2818e-03 4.5511e-04 36.00 0.25 0.29 0.37

16384 5.8156e-03 7.2695e-04 45.08 0.15 0.18 0.22

32768 4.8370e-03 1.2093e-03 54.20 0.09 0.10 0.13

65536 4.2762e-03 2.1381e-03 61.30 0.06 0.07 0.12

131072 3.8729e-03 3.8729e-03 67.69 0.05 0.03 0.26

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 1 6 3

256 6 0 3 1 2

512 0 2 6 3 1

1024 1 0 2 3 6

2048 0 1 2 3 6

4096 0 3 2 1 6

8192 4 5 2 0 1

16384 5 4 10 2 6

32768 4 5 2 0 3

65536 4 5 0 6 10

131072 0 1 4 5 10

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 5 7

256 2 5 7

512 1 2 6

1024 1 3 7

2048 3 5 7

4096 3 5 7

8192 1 2 2

16384 1 2 11

32768 1 2 11

65536 2 2 11

131072 1 10 10

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 9.7527e-02 9.5241e-05 2.69 0.65 0.18 1.68

256 5.1233e-02 1.0006e-04 5.12 0.62 0.20 1.60

512 2.7065e-02 1.0572e-04 9.69 0.62 0.23 1.56

1024 1.8286e-02 1.4286e-04 14.34 0.45 0.16 1.15

2048 1.1804e-02 1.8443e-04 22.21 0.36 0.15 0.90

4096 8.4728e-03 2.6477e-04 30.94 0.26 0.11 0.68

8192 7.0832e-03 4.4270e-04 37.01 0.25 0.30 0.32

16384 5.6148e-03 7.0185e-04 46.69 0.15 0.17 0.20

32768 4.6459e-03 1.1615e-03 56.43 0.10 0.12 0.14

65536 4.2055e-03 2.1028e-03 62.33 0.05 0.06 0.09

131072 3.7237e-03 3.7237e-03 70.40 0.03 0.04 0.05

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 3 6 1

256 2 0 1 6 3

512 0 2 1 3 6

1024 0 2 1 3 6

2048 0 2 1 3 6

4096 2 1 0 6 3

8192 4 5 0 3 8

16384 5 4 0 10 2

32768 5 4 10 7 0

65536 4 5 3 0 2

131072 5 8 4 1 10

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 2 7

256 1 4 7

512 1 3 6

1024 1 4 7

2048 1 3 7

4096 2 5 7

8192 1 2 2

16384 2 2 11

32768 1 2 11

65536 2 5 11

131072 1 11 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 9.8600e-02 9.6289e-05 2.66 0.66 0.21 1.64

256 5.0677e-02 9.8979e-05 5.17 0.65 0.20 1.62

512 2.6693e-02 1.0427e-04 9.82 0.66 0.28 1.60

1024 1.8060e-02 1.4109e-04 14.52 0.48 0.18 1.21

2048 1.1792e-02 1.8425e-04 22.23 0.37 0.15 0.89

4096 8.6070e-03 2.6897e-04 30.46 0.25 0.10 0.61

8192 7.0871e-03 4.4294e-04 36.99 0.26 0.31 0.33

16384 5.5571e-03 6.9464e-04 47.17 0.16 0.19 0.22

32768 4.6316e-03 1.1579e-03 56.60 0.10 0.13 0.15

65536 4.1576e-03 2.0788e-03 63.05 0.06 0.07 0.09

131072 3.7013e-03 3.7013e-03 70.83 0.04 0.04 0.08

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 1 2 6 3

256 0 6 2 1 3

512 0 1 2 6 3

1024 0 1 2 3 6

2048 0 2 1 3 6

4096 0 2 6 1 3

8192 4 5 10 0 6

16384 4 5 0 10 1

32768 5 4 10 7 9

65536 4 5 7 8 0

131072 5 4 0 10 7

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 6

256 1 1 6

512 1 1 5

1024 1 2 7

2048 1 2 7

4096 1 4 7

8192 2 2 2

16384 1 2 11

32768 2 2 11

65536 1 2 11

131072 2 9 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 9.7886e-02 9.5592e-05 2.68 0.66 0.14 1.76

256 4.9410e-02 9.6503e-05 5.31 0.69 0.17 1.75

512 2.7590e-02 1.0777e-04 9.50 0.60 0.17 1.58

1024 1.8151e-02 1.4181e-04 14.44 0.49 0.15 1.27

2048 1.2184e-02 1.9038e-04 21.51 0.37 0.11 0.93

4096 8.6202e-03 2.6938e-04 30.41 0.26 0.06 0.70

8192 6.9515e-03 4.3447e-04 37.71 0.30 0.35 0.40

16384 5.6932e-03 7.1165e-04 46.05 0.16 0.20 0.21

32768 4.7677e-03 1.1919e-03 54.98 0.10 0.12 0.14

65536 4.1916e-03 2.0958e-03 62.54 0.08 0.09 0.10

131072 3.8067e-03 3.8067e-03 68.86 0.04 0.05 0.07

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 4 1 6

256 0 6 4 2 3

512 0 1 2 4 6

1024 0 2 6 1 4

2048 0 2 4 6 3

4096 0 1 6 2 4

8192 4 5 6 10 7

16384 5 4 10 0 6

32768 4 5 0 10 9

65536 4 5 7 6 2

131072 5 4 6 0 9

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 7

256 1 1 7

512 1 1 7

1024 1 1 7

2048 1 1 7

4096 1 4 7

8192 1 2 2

16384 2 2 11

32768 2 2 11

65536 1 2 11

131072 2 6 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.0006e-01 9.7715e-05 2.62 0.62 0.13 1.66

256 5.2636e-02 1.0281e-04 4.98 0.58 0.09 1.58

512 2.7657e-02 1.0804e-04 9.48 0.60 0.13 1.61

1024 1.8208e-02 1.4225e-04 14.40 0.47 0.12 1.22

2048 1.1878e-02 1.8560e-04 22.07 0.37 0.10 0.91

4096 8.6070e-03 2.6897e-04 30.46 0.24 0.06 0.63

8192 6.6744e-03 4.1715e-04 39.28 0.32 0.40 0.43

16384 5.4185e-03 6.7731e-04 48.38 0.19 0.22 0.27

32768 4.5853e-03 1.1463e-03 57.17 0.11 0.13 0.15

65536 4.1016e-03 2.0508e-03 63.91 0.08 0.09 0.11

131072 3.7488e-03 3.7488e-03 69.93 0.03 0.03 0.04

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 6 1 4

256 0 6 2 4 1

512 0 2 4 1 6

1024 0 2 6 1 3

2048 0 2 1 6 4

4096 0 2 6 4 1

8192 4 5 0 10 2

16384 4 5 7 10 8

32768 4 5 2 0 6

65536 5 4 10 9 7

131072 5 4 3 10 8

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 3 7

256 1 3 7

512 1 1 7

1024 1 2 7

2048 1 1 7

4096 2 5 7

8192 1 2 2

16384 2 2 10

32768 2 2 11

65536 1 2 11

131072 2 11 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 9.9969e-02 9.7626e-05 2.62 0.62 0.09 1.62

256 5.1940e-02 1.0145e-04 5.05 0.59 0.08 1.58

512 2.7203e-02 1.0626e-04 9.64 0.61 0.13 1.55

1024 1.8493e-02 1.4447e-04 14.18 0.43 0.09 1.14

2048 1.1737e-02 1.8339e-04 22.33 0.37 0.09 0.94

4096 8.5009e-03 2.6565e-04 30.84 0.25 0.05 0.65

8192 6.8731e-03 4.2957e-04 38.14 0.29 0.34 0.37

16384 5.3462e-03 6.6828e-04 49.03 0.20 0.24 0.29

32768 4.5721e-03 1.1430e-03 57.34 0.12 0.14 0.20

65536 4.1025e-03 2.0512e-03 63.90 0.08 0.09 0.11

131072 3.6947e-03 3.6947e-03 70.95 0.04 0.04 0.07

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 6 2 1 3

256 0 6 2 1 3

512 0 2 6 1 4

1024 0 1 2 6 5

2048 0 2 4 1 6

4096 0 2 1 5 4

8192 5 4 0 10 2

16384 4 5 6 10 9

32768 5 4 7 1 10

65536 4 5 10 6 3

131072 5 4 8 6 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 7

256 1 3 7

512 1 2 7

1024 1 3 7

2048 1 2 7

4096 1 6 7

8192 1 2 2

16384 1 2 9

32768 1 2 11

65536 2 2 11

131072 2 10 11

DISCUSSION

The SP3 Winterhawk I processing node consists of two processors. MPI between processors within a node uses a shared memory implementation for communication, in contrast to communication between nodes which uses a network interface card and an external switch.