COMMTEST SP3-375 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM SP3-375 Winterhawk II SWAP Performance

(ordered swap of 8KB message using MPI within a node)

(performance measured per processor when all processors in node communicating)

Date/Person:	May 25, 2000 / P. Worley
Platform:	IBM SP3 at Oak Ridge National Laboratory (eagle.ccs.ornl.gov):
	160 4-way Winterhawk II SMP nodes (375MHz POWER3 with 8MB L2 cache)
Environment:	AIX 4.3.3; PSSP 3.1.1
Communication Library:	shared memory MPI within a node, MPI over the switch between nodes, using User Space protocol in both cases
SWAP size:	1024 REAL*8 floating point values each direction
Message size:	Largest - 1024 REAL*8 floating point values
	Smallest - 1 REAL*8 floating point values
Processors:	0 and 1
	2 and 3
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

ordered simple swap

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 55.72 10.46 14.2%

1 iter. 163.67 9.46 37.8%

10 iter. 175.83 9.46 40.6%

ordered swap using nonblocking send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 46.68 10.19 15.5%

1 iter. 155.72 10.25 39.0%

10 iter. 165.49 10.24 41.4%

cache inv. w/overlap 41.45 10.91 11.0%

1 iter. w/overlap 150.25 10.78 39.5%

10 iter. w/overlap 163.56 10.94 43.7%

ordered swap using nonblocking receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 48.63 10.46 12.4%

1 iter. 154.63 10.37 39.1%

10 iter. 168.89 10.39 42.8%

cache inv. w/overlap 48.34 10.83 17.1%

1 iter. w/overlap 153.24 10.60 39.7%

10 iter. w/overlap 166.98 10.72 43.7%

ordered swap using nonblocking send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 47.34 11.04 14.9%

1 iter. 150.11 11.24 41.2%

10 iter. 155.66 11.10 42.2%

cache inv. w/overlap 45.48 11.76 14.6%

1 iter. w/overlap 146.82 12.12 43.4%

10 iter. w/overlap 158.66 12.01 46.5%

ordered swap using ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 40.96 16.46 16.5%

1 iter. 167.04 16.28 42.3%

10 iter. 194.19 16.21 38.4%

cache inv. w/overlap 48.64 11.11 16.5%

1 iter. w/overlap 187.59 11.03 25.3%

10 iter. w/overlap 205.37 10.97 27.5%

ordered swap using nonblocking ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 41.20 16.99 17.7%

1 iter. 171.89 16.75 35.2%

10 iter. 181.12 16.81 37.7%

cache inv. w/overlap 47.75 11.82 13.8%

1 iter. w/overlap 185.55 12.13 27.5%

10 iter. w/overlap 204.69 11.99 30.0%

synchronous

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 49.95 13.20 16.1%

1 iter. 145.70 12.92 46.0%

10 iter. 152.47 12.90 48.0%

ordered swap using nonblocking sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 41.66 23.50 23.9%

1 iter. 141.16 23.57 40.6%

10 iter. 159.61 23.44 45.7%

cache inv. w/overlap 38.16 23.87 15.2%

1 iter. w/overlap 143.00 24.31 42.4%

10 iter. w/overlap 147.17 24.33 43.7%

ordered swap using nonblocking receive with sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 42.50 23.94 19.6%

1 iter. 147.80 23.64 42.6%

10 iter. 158.79 24.25 47.0%

cache inv. w/overlap 43.20 24.02 16.6%

1 iter. w/overlap 143.26 24.50 42.8%

10 iter. w/overlap 154.63 23.97 45.2%

ordered swap using nonblocking sync. send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 42.37 24.36 18.9%

1 iter. 145.57 24.26 43.1%

10 iter. 155.42 24.32 46.1%

cache inv. w/overlap 41.87 25.41 18.7%

1 iter. w/overlap 134.27 25.54 41.9%

10 iter. w/overlap 151.84 26.31 48.8%

ordered simple swap using sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 48.07 23.35 16.7%

1 iter. 136.90 22.81 40.4%

10 iter. 159.15 22.70 44.1%

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 2.0842e-02 2.0354e-05 0.79 0.66 0.65 1.43

16 1.0134e-02 1.9792e-05 1.62 0.73 0.74 1.53

32 5.2329e-03 2.0441e-05 3.13 0.72 0.76 1.47

64 2.7585e-03 2.1551e-05 5.94 0.69 0.71 1.42

128 1.5439e-03 2.4124e-05 10.61 0.64 0.71 1.29

256 9.2745e-04 2.8983e-05 17.67 0.53 0.58 1.08

512 5.6117e-04 3.5073e-05 29.20 0.50 0.51 0.98

1024 4.0500e-04 5.0625e-05 40.45 0.41 0.45 0.79

2048 3.3128e-04 8.2819e-05 49.46 0.32 0.37 0.56

4096 2.9404e-04 1.4702e-04 55.72 0.25 0.27 0.39

8192 3.2245e-04 3.2245e-04 50.81 0.18 0.20 0.34

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 1 2 3 6

16 0 1 2 3 6

32 0 2 1 3 6

64 0 1 2 3 6

128 0 2 1 3 6

256 0 1 2 3 6

512 0 2 1 3 6

1024 0 1 2 3 6

2048 0 1 2 6 3

4096 0 6 2 3 1

8192 0 10 6 2 8

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 3 4

16 1 1 4

32 1 1 4

64 1 1 4

128 1 1 4

256 1 1 4

512 1 1 4

1024 1 1 5

2048 1 1 5

4096 1 1 5

8192 1 1 8

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.9413e-02 1.8958e-05 0.84 0.74 0.73 1.57

16 9.7220e-03 1.8988e-05 1.69 0.74 0.73 1.57

32 4.9313e-03 1.9263e-05 3.32 0.74 0.73 1.55

64 2.5183e-03 1.9674e-05 6.51 0.72 0.69 1.51

128 1.3012e-03 2.0331e-05 12.59 0.70 0.73 1.47

256 6.6809e-04 2.0878e-05 24.52 0.72 0.81 1.45

512 3.6457e-04 2.2786e-05 44.94 0.66 0.66 1.34

1024 2.1372e-04 2.6715e-05 76.66 0.53 0.39 1.17

2048 1.3968e-04 3.4920e-05 117.30 0.38 0.27 0.84

4096 1.0011e-04 5.0053e-05 163.67 0.26 0.15 0.60

8192 9.5319e-05 9.5319e-05 171.89 0.18 0.18 0.34

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 1 2 3 6

16 0 1 2 3 6

32 0 1 2 3 6

64 0 1 2 3 6

128 0 1 2 3 6

256 0 1 2 3 6

512 0 1 2 3 6

1024 0 2 1 3 6

2048 0 1 2 3 6

4096 0 1 2 3 6

8192 5 4 0 8 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 1 4

16 1 1 4

32 1 1 4

64 1 1 4

128 1 1 4

256 1 1 4

512 1 1 4

1024 1 1 4

2048 1 3 5

4096 1 1 7

8192 1 2 8

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.9436e-02 1.8980e-05 0.84 0.74 0.72 1.56

16 9.7507e-03 1.9044e-05 1.68 0.74 0.73 1.56

32 4.9145e-03 1.9197e-05 3.33 0.74 0.74 1.56

64 2.5069e-03 1.9585e-05 6.54 0.72 0.69 1.51

128 1.2847e-03 2.0074e-05 12.75 0.72 0.77 1.51

256 6.7762e-04 2.1176e-05 24.18 0.69 0.76 1.40

512 3.6222e-04 2.2639e-05 45.23 0.64 0.68 1.33

1024 2.0900e-04 2.6125e-05 78.39 0.53 0.41 1.15

2048 1.3357e-04 3.3394e-05 122.66 0.40 0.28 0.91

4096 9.3181e-05 4.6590e-05 175.83 0.31 0.20 0.67

8192 8.4372e-05 8.4372e-05 194.19 0.21 0.22 0.28

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 1 2 3 6

16 0 1 2 3 6

32 0 1 2 3 6

64 0 1 2 3 6

128 0 1 2 3 6

256 0 1 2 3 6

512 0 1 2 3 6

1024 0 2 1 3 6

2048 0 2 1 3 6

4096 0 2 1 3 6

8192 4 5 0 7 10

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 1 4

16 1 1 4

32 1 1 4

64 1 1 4

128 1 1 4

256 1 1 4

512 1 1 4

1024 1 1 4

2048 1 1 5

4096 1 2 7

8192 1 1 7

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 2.0234e-02 1.9759e-05 0.81 0.65 0.23 1.60

16 1.0277e-02 2.0072e-05 1.59 0.65 0.24 1.58

32 5.3338e-03 2.0835e-05 3.07 0.63 0.24 1.54

64 2.8368e-03 2.2163e-05 5.78 0.58 0.21 1.44

128 1.5218e-03 2.3778e-05 10.77 0.61 0.25 1.45

256 9.1468e-04 2.8584e-05 17.91 0.52 0.25 1.18

512 5.6394e-04 3.5246e-05 29.05 0.47 0.24 1.08

1024 3.9372e-04 4.9215e-05 41.61 0.46 0.34 0.94

2048 3.3138e-04 8.2846e-05 49.44 0.31 0.30 0.62

4096 3.0149e-04 1.5074e-04 54.34 0.23 0.19 0.53

8192 3.3372e-04 3.3372e-04 49.09 0.14 0.14 0.35

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 2 1 4 5

16 0 2 1 4 5

32 0 2 1 4 5

64 0 2 1 4 5

128 0 2 1 4 5

256 0 2 4 1 5

512 0 4 2 1 5

1024 0 2 4 5 3

2048 0 4 5 3 6

4096 0 6 4 2 5

8192 6 0 5 10 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 1 6

16 1 1 6

32 1 1 6

64 1 1 6

128 1 1 6

256 1 1 6

512 1 1 6

1024 1 1 3

2048 1 1 5

4096 1 1 6

8192 2 3 8

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.9778e-02 1.9314e-05 0.83 0.68 0.25 1.64

16 9.9053e-03 1.9346e-05 1.65 0.68 0.25 1.64

32 4.9832e-03 1.9466e-05 3.29 0.68 0.27 1.63

64 2.5434e-03 1.9870e-05 6.44 0.67 0.27 1.62

128 1.3057e-03 2.0402e-05 12.55 0.66 0.27 1.55

256 6.9670e-04 2.1772e-05 23.52 0.61 0.26 1.45

512 3.7447e-04 2.3404e-05 43.75 0.58 0.26 1.37

1024 2.1638e-04 2.7048e-05 75.72 0.52 0.23 1.20

2048 1.4596e-04 3.6489e-05 112.25 0.34 0.13 0.86

4096 1.0170e-04 5.0851e-05 161.10 0.27 0.12 0.65

8192 8.7340e-05 8.7340e-05 187.59 0.26 0.30 0.41

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 2 1 4 5

16 0 2 1 4 5

32 0 2 1 4 3

64 0 2 1 4 5

128 0 2 1 4 5

256 0 2 1 4 3

512 0 2 4 1 5

1024 0 2 1 4 3

2048 0 2 4 1 5

4096 0 2 4 1 3

8192 4 5 10 2 0

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 1 5

16 1 1 5

32 1 1 4

64 1 1 4

128 1 1 4

256 1 1 5

512 1 1 5

1024 1 1 6

2048 1 2 7

4096 1 1 7

8192 1 2 2

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.9648e-02 1.9187e-05 0.83 0.68 0.25 1.74

16 9.8365e-03 1.9212e-05 1.67 0.68 0.25 1.73

32 4.9737e-03 1.9428e-05 3.29 0.68 0.26 1.72

64 2.5279e-03 1.9749e-05 6.48 0.67 0.25 1.69

128 1.3010e-03 2.0328e-05 12.59 0.65 0.25 1.64

256 6.8619e-04 2.1443e-05 23.88 0.61 0.23 1.54

512 3.6943e-04 2.3089e-05 44.35 0.57 0.21 1.43

1024 2.1193e-04 2.6491e-05 77.31 0.50 0.19 1.26

2048 1.3291e-04 3.3229e-05 123.27 0.40 0.16 0.99

4096 9.2383e-05 4.6191e-05 177.35 0.29 0.13 0.71

8192 7.9777e-05 7.9777e-05 205.37 0.27 0.31 0.40

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 2 1 4 5

16 0 2 1 4 5

32 0 2 1 4 5

64 0 2 1 4 5

128 0 2 1 4 5

256 0 2 1 4 5

512 0 2 4 1 5

1024 0 2 1 4 5

2048 0 2 4 1 5

4096 0 2 4 1 3

8192 4 5 10 0 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 1 5

16 1 1 5

32 1 1 4

64 1 1 5

128 1 1 5

256 1 1 6

512 1 1 6

1024 1 1 6

2048 1 1 6

4096 1 1 7

8192 2 2 2

DISCUSSION

The SP3 Winterhawk II processing node consists of four processors. In these tests, we examine MPI communication performance between processors in a node when all processors in the node are communicating. The performance is reported for a single processor pair (logical processors 0 and 1), representing the latency and bandwdith that would typically appear in a highly synchronous communication-intensive operation, such as a collective communication command.