COMMTEST SP3-200 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM SP3-200 SWAP Performance

(unordered swap of 8KB message using MPI and US between nodes)

(performance measured per processor when all processors in node communicating)

Date/Person:	May 25, 2000 / P. Worley
Platform:	IBM SP3 at National Energy Research Scientific Computing Center (Gseaborg)
	256 2-way Winterhawk I SMP nodes (200 MHz POWER3 with 4MB L2 cache, equivalent to RS/6000 Model 260
Environment:	AIX 4.3.2; POE 2.4.0.12
Communication Library:	MPI over the switch using user space
SWAP size:	1024 REAL*8 floating point values each direction
Message size:	Largest - 1024 REAL*8 floating point values
	Smallest - 1 REAL*8 floating point values
Processors:	0 and 2
	1 and 3
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

unordered simple swap

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 40.44 57.43 28.4%

1 iter. 47.83 57.84 33.8%

10 iter. 49.59 54.73 33.1%

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 45.51 57.34 31.9%

1 iter. 46.32 55.35 33.7%

10 iter. 49.70 56.93 34.5%

cache inv. w/overlap 42.06 58.63 20.6%

1 iter. w/overlap 46.35 59.64 33.7%

10 iter. w/overlap 52.25 57.01 36.4%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 40.17 54.65 26.8%

1 iter. 47.23 58.24 33.6%

10 iter. 50.96 55.71 34.7%

cache inv. w/overlap 48.17 57.81 34.0%

1 iter. w/overlap 45.03 59.05 32.5%

10 iter. w/overlap 49.76 56.36 34.2%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 41.34 57.86 27.0%

1 iter. 47.08 62.29 35.8%

10 iter. 48.63 57.66 34.2%

cache inv. w/overlap 40.73 58.62 29.1%

1 iter. w/overlap 55.42 58.25 39.4%

10 iter. w/overlap 51.50 58.78 37.0%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 44.67 112.22 30.6%

1 iter. 44.71 109.50 31.4%

10 iter. 51.01 108.57 33.8%

cache inv. w/overlap 48.07 58.17 19.5%

1 iter. w/overlap 58.12 61.64 29.3%

10 iter. w/overlap 64.36 56.61 22.2%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 43.59 114.24 30.4%

1 iter. 49.44 113.02 34.1%

10 iter. 49.87 116.23 35.4%

cache inv. w/overlap 45.08 58.36 28.3%

1 iter. w/overlap 70.94 62.20 26.9%

10 iter. w/overlap 62.68 58.74 26.7%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 42.70 56.63 29.5%

1 iter. 45.61 59.77 33.3%

10 iter. 49.88 57.47 35.0%

unordered swap using nonblocking sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 34.95 138.61 35.4%

1 iter. 32.60 139.24 55.4%

10 iter. 42.83 137.18 35.9%

cache inv. w/overlap 37.08 141.78 32.1%

1 iter. w/overlap 36.31 142.33 31.5%

10 iter. w/overlap 44.96 143.00 39.2%

unordered swap using nonblocking receive with sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 42.31 140.13 36.2%

1 iter. 31.19 140.83 26.8%

10 iter. 37.71 141.74 32.6%

cache inv. w/overlap 39.52 137.07 33.1%

1 iter. w/overlap 33.94 135.78 29.4%

10 iter. w/overlap 36.55 143.57 32.0%

unordered swap using nonblocking sync. send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 38.37 145.85 34.2%

1 iter. 31.15 149.28 35.5%

10 iter. 35.15 146.25 31.4%

cache inv. w/overlap 36.16 148.19 32.7%

1 iter. w/overlap 40.06 146.87 35.9%

10 iter. w/overlap 36.28 145.67 32.3%

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 5.7248e-02 5.5906e-05 0.29 0.68 1.02 1.63

16 2.8761e-02 5.6174e-05 0.57 0.69 1.02 1.63

32 1.4812e-02 5.7859e-05 1.11 0.64 0.94 1.51

64 7.1604e-03 5.5941e-05 2.29 0.71 1.04 1.66

128 3.9007e-03 6.0949e-05 4.20 0.65 0.91 1.54

256 2.0290e-03 6.3408e-05 8.07 0.67 0.92 1.65

512 1.0713e-03 6.6955e-05 15.29 0.70 0.93 1.71

1024 7.3968e-04 9.2460e-05 22.15 0.54 0.64 1.32

2048 4.9564e-04 1.2391e-04 33.06 0.44 0.54 1.07

4096 3.6000e-04 1.8000e-04 45.51 0.33 0.35 0.97

8192 3.6681e-04 3.6681e-04 44.67 0.15 0.16 0.41

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 0 1 3

16 6 0 2 1 3

32 3 6 2 0 1

64 2 6 0 1 3

128 6 1 0 2 3

256 1 0 2 6 3

512 1 2 6 0 3

1024 0 6 1 3 2

2048 6 0 2 3 1

4096 1 6 3 0 2

8192 4 5 8 3 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 4 5

16 2 4 5

32 1 5 5

64 1 2 5

128 3 4 5

256 3 3 5

512 1 1 5

1024 1 2 5

2048 1 3 5

4096 1 1 5

8192 1 2 8

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 5.8023e-02 5.6663e-05 0.28 0.66 0.98 1.61

16 2.7964e-02 5.4617e-05 0.59 0.72 1.09 1.68

32 1.3953e-02 5.4504e-05 1.17 0.74 1.11 1.71

64 7.0880e-03 5.5375e-05 2.31 0.72 1.05 1.70

128 3.4162e-03 5.3378e-05 4.80 0.82 1.17 1.97

256 1.9694e-03 6.1543e-05 8.32 0.68 0.97 1.63

512 1.0612e-03 6.6323e-05 15.44 0.64 0.85 1.52

1024 7.2543e-04 9.0678e-05 22.59 0.54 0.65 1.38

2048 4.3011e-04 1.0753e-04 38.09 0.58 0.70 1.35

4096 3.4255e-04 1.7128e-04 47.83 0.29 0.30 0.92

8192 3.3138e-04 3.3138e-04 49.44 0.38 0.48 0.59

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 1 6 2 3

16 6 0 2 1 3

32 6 2 3 0 1

64 0 6 3 1 2

128 1 6 2 0 3

256 0 1 6 2 3

512 1 0 2 6 3

1024 6 0 1 3 2

2048 1 2 0 6 3

4096 0 2 3 1 6

8192 5 4 3 6 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 4 4 5

16 1 3 5

32 1 2 5

64 2 3 5

128 1 2 5

256 2 4 5

512 1 3 5

1024 2 4 5

2048 1 1 5

4096 1 5 5

8192 1 1 2

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 5.6916e-02 5.5582e-05 0.29 0.69 1.01 1.64

16 2.8435e-02 5.5537e-05 0.58 0.70 1.06 1.64

32 1.4411e-02 5.6293e-05 1.14 0.68 1.06 1.61

64 7.0160e-03 5.4812e-05 2.34 0.72 1.08 1.70

128 3.7199e-03 5.8123e-05 4.40 0.68 0.99 1.63

256 1.9169e-03 5.9903e-05 8.55 0.68 0.97 1.64

512 1.0533e-03 6.5828e-05 15.56 0.65 0.88 1.57

1024 7.0924e-04 8.8656e-05 23.10 0.53 0.63 1.34

2048 4.4081e-04 1.1020e-04 37.17 0.47 0.58 1.20

4096 3.2150e-04 1.6075e-04 50.96 0.38 0.43 0.98

8192 3.2118e-04 3.2118e-04 51.01 0.25 0.35 0.45

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 2 6 1 3

16 6 2 0 1 3

32 0 6 2 1 3

64 2 6 1 0 3

128 0 1 6 2 3

256 1 0 6 2 3

512 6 0 1 2 3

1024 1 2 6 0 3

2048 0 1 2 6 3

4096 2 6 1 0 3

8192 4 5 1 7 0

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 5 5

16 2 4 5

32 2 5 5

64 2 3 5

128 2 4 5

256 1 4 5

512 3 4 5

1024 3 5 5

2048 3 5 5

4096 1 5 5

8192 1 2 5

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 5.7130e-02 5.5791e-05 0.29 0.49 0.05 1.66

16 2.9262e-02 5.7153e-05 0.56 0.47 0.03 1.60

32 1.3919e-02 5.4372e-05 1.18 0.53 0.10 1.77

64 7.3238e-03 5.7217e-05 2.24 0.49 0.05 1.66

128 3.7493e-03 5.8582e-05 4.37 0.51 0.07 1.65

256 2.0067e-03 6.2709e-05 8.16 0.50 0.09 1.57

512 1.1382e-03 7.1137e-05 14.39 0.45 0.08 1.46

1024 7.8585e-04 9.8231e-05 20.85 0.36 0.05 1.14

2048 5.0394e-04 1.2598e-04 32.51 0.31 0.07 1.10

4096 3.4011e-04 1.7005e-04 48.17 0.36 0.23 0.91

8192 3.4085e-04 3.4085e-04 48.07 0.24 0.26 0.47

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 6 2 3 4

16 6 3 2 0 4

32 4 6 1 0 2

64 6 1 0 2 4

128 1 6 2 0 4

256 0 2 1 4 6

512 0 6 1 2 4

1024 6 0 2 1 4

2048 0 2 3 1 4

4096 2 5 4 3 6

8192 4 5 1 2 8

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 5 7

16 4 6 7

32 1 4 7

64 1 5 7

128 1 4 7

256 2 4 7

512 3 4 7

1024 2 5 7

2048 2 5 7

4096 1 1 6

8192 1 1 5

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 5.7626e-02 5.6276e-05 0.28 0.49 0.06 1.61

16 2.8653e-02 5.5963e-05 0.57 0.49 0.03 1.63

32 1.4663e-02 5.7279e-05 1.12 0.48 0.04 1.65

64 6.7302e-03 5.2580e-05 2.43 0.60 0.13 1.96

128 3.7637e-03 5.8808e-05 4.35 0.50 0.05 1.67

256 1.9816e-03 6.1925e-05 8.27 0.50 0.08 1.64

512 1.0565e-03 6.6031e-05 15.51 0.52 0.11 1.66

1024 7.1883e-04 8.9854e-05 22.79 0.41 0.06 1.31

2048 3.8798e-04 9.6995e-05 42.23 0.56 0.25 1.49

4096 2.9564e-04 1.4782e-04 55.42 0.40 0.20 1.09

8192 2.3096e-04 2.3096e-04 70.94 0.76 0.94 1.09

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 6 2 1 4

16 2 4 1 5 0

32 0 6 2 5 1

64 1 2 6 0 4

128 0 6 4 2 1

256 0 4 5 2 6

512 0 6 1 2 4

1024 4 6 2 0 3

2048 6 2 4 3 0

4096 3 4 5 0 6

8192 5 4 3 9 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 5 7

16 2 6 7

32 1 7 7

64 1 2 7

128 1 5 7

256 2 5 7

512 1 3 7

1024 1 5 7

2048 1 1 7

4096 1 1 7

8192 1 1 2

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 5.7066e-02 5.5729e-05 0.29 0.49 0.06 1.62

16 2.8680e-02 5.6016e-05 0.57 0.49 0.06 1.61

32 1.2690e-02 4.9572e-05 1.29 0.66 0.16 1.95

64 7.2144e-03 5.6362e-05 2.27 0.50 0.07 1.63

128 3.8168e-03 5.9637e-05 4.29 0.46 0.06 1.53

256 1.9241e-03 6.0128e-05 8.52 0.50 0.09 1.62

512 1.0334e-03 6.4590e-05 15.85 0.48 0.09 1.59

1024 6.8114e-04 8.5142e-05 24.05 0.42 0.06 1.31

2048 4.4103e-04 1.1026e-04 37.15 0.35 0.04 1.14

4096 2.9909e-04 1.4954e-04 54.78 0.34 0.09 1.09

8192 2.5459e-04 2.5459e-04 64.36 0.53 0.69 0.81

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 6 2 4 1

16 0 2 4 6 1

32 3 6 0 2 4

64 0 6 2 1 4

128 6 2 0 1 4

256 0 6 2 4 1

512 0 6 2 1 4

1024 6 0 1 4 2

2048 4 3 6 1 2

4096 5 4 1 3 0

8192 4 5 7 1 0

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 5 7

16 2 5 7

32 1 1 7

64 2 5 7

128 2 5 7

256 1 3 7

512 1 3 7

1024 1 4 7

2048 1 7 7

4096 1 3 7

8192 1 2 2

DISCUSSION

The SP3 Winterhawk I processing node consists of two processors. MPI between processors within a node uses a shared memory implementation for communication, in contrast to communication between nodes which uses a network interface card and an external switch.