COMMTEST IBM p690 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM p690 SWAP Performance

(unordered swap of 8KB message using MPI within a node)

(performance measured per processor when all processors in node participating in a ring shift)

Date/Person:	April 4, 2002 / P. Worley
Platform:	IBM p690 at Oak Ridge National Laboratory (cheetah.ccs.ornl.gov):
	p690 32-way Turbo SMP node (1.3 GHz POWER4)
Environment:	AIX 5.1L
Communication Library:	MPI
SWAP size:	1024 REAL*8 floating point values each direction
Message size:	Largest - 1024 REAL*8 floating point values
	Smallest - 1 REAL*8 floating point values
Processors:	receive from i-1, send to i+1, i=0,1,...,31
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 142.06 14.86 12.9%

1 iter. 236.95 13.74 19.9%

10 iter. 451.78 13.76 43.2%

cache inv. w/overlap 142.88 14.85 13.0%

1 iter. w/overlap 209.66 14.20 18.2%

10 iter. w/overlap 454.12 14.84 41.1%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 153.41 12.13 11.4%

1 iter. 231.24 11.95 16.9%

10 iter. 464.87 12.40 43.1%

cache inv. w/overlap 143.72 12.06 20.2%

1 iter. w/overlap 245.71 12.72 19.1%

10 iter. w/overlap 488.26 13.33 42.5%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 150.87 14.85 13.7%

1 iter. 215.65 14.44 19.0%

10 iter. 431.54 14.88 43.5%

cache inv. w/overlap 148.55 15.29 14.3%

1 iter. w/overlap 279.53 14.93 25.5%

10 iter. w/overlap 454.12 16.21 44.9%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 134.43 22.57 23.8%

1 iter. 261.73 22.89 36.6%

10 iter. 516.91 22.71 71.6%

cache inv. w/overlap 191.10 20.24 23.6%

1 iter. w/overlap 245.76 21.36 32.0%

10 iter. w/overlap 646.53 23.10 91.2%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 131.27 28.01 34.6%

1 iter. 229.73 26.72 37.5%

10 iter. 483.23 26.15 77.1%

cache inv. w/overlap 170.38 22.72 23.6%

1 iter. w/overlap 245.37 22.42 33.6%

10 iter. w/overlap 614.77 25.75 96.6%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 145.04 12.60 12.8%

1 iter. 251.80 11.73 18.0%

10 iter. 489.80 12.64 43.9%

unordered swap using nonblocking sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 157.62 27.27 26.2%

1 iter. 251.08 27.01 41.4%

10 iter. 460.54 26.31 74.0%

cache inv. w/overlap 136.46 26.65 22.2%

1 iter. w/overlap 241.94 26.67 39.4%

10 iter. w/overlap 450.23 26.29 72.2%

unordered swap using nonblocking receive with sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 140.10 25.18 21.5%

1 iter. 252.94 24.78 38.3%

10 iter. 466.51 24.93 71.0%

cache inv. w/overlap 156.61 25.00 23.9%

1 iter. w/overlap 227.26 23.82 33.0%

10 iter. w/overlap 487.62 24.73 73.6%

unordered swap using nonblocking sync. send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 141.83 27.49 23.8%

1 iter. 235.99 26.60 38.3%

10 iter. 446.97 27.34 74.6%

cache inv. w/overlap 138.64 28.82 26.4%

1 iter. w/overlap 235.67 28.23 40.6%

10 iter. w/overlap 461.40 27.52 77.5%

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.2738e-02 1.2439e-05 1.29 0.65 0.87 1.22

16 6.5285e-03 1.2751e-05 2.51 0.62 0.87 1.17

32 3.2109e-03 1.2542e-05 5.10 0.64 0.80 1.25

64 1.6690e-03 1.3039e-05 9.82 0.59 0.75 1.18

128 8.7453e-04 1.3665e-05 18.73 0.57 0.73 1.09

256 4.7624e-04 1.4882e-05 34.40 0.55 0.67 1.12

512 2.8795e-04 1.7997e-05 56.90 0.46 0.53 1.07

1024 1.9168e-04 2.3960e-05 85.48 0.31 0.44 0.58

2048 1.4483e-04 3.6207e-05 113.13 0.21 0.21 0.36

4096 1.3487e-04 6.7433e-05 121.48 0.03 0.03 0.09

8192 1.0632e-04 1.0632e-04 154.10 0.06 0.04 0.14

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 1 3 4

16 2 6 1 3 4

32 2 6 1 3 4

64 6 2 1 3 4

128 2 6 1 3 4

256 2 6 3 1 4

512 2 6 1 3 4

1024 6 2 1 3 4

2048 6 2 3 1 5

4096 2 4 5 1 9

8192 6 2 5 9 4

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 2 4

16 1 2 4

32 1 2 3

64 2 2 4

128 1 2 4

256 1 2 4

512 1 1 4

1024 1 2 4

2048 1 1 5

4096 3 6 9

8192 2 5 9

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.2451e-02 1.2160e-05 1.32 0.64 0.85 1.21

16 6.3733e-03 1.2448e-05 2.57 0.60 0.78 1.18

32 3.1620e-03 1.2352e-05 5.18 0.62 0.81 1.20

64 1.5919e-03 1.2436e-05 10.29 0.60 0.81 1.19

128 8.2209e-04 1.2845e-05 19.93 0.58 0.78 1.12

256 4.3637e-04 1.3637e-05 37.55 0.55 0.72 1.06

512 2.4829e-04 1.5518e-05 65.99 0.49 0.67 0.99

1024 1.5175e-04 1.8968e-05 107.97 0.39 0.49 0.73

2048 1.0395e-04 2.5987e-05 157.62 0.31 0.42 0.53

4096 9.7027e-05 4.8513e-05 168.86 0.05 0.05 0.11

8192 7.2360e-05 7.2360e-05 226.42 0.09 0.08 0.19

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 6 2 1 3 4

16 2 6 1 3 4

32 6 2 1 3 4

64 6 2 1 3 4

128 6 2 1 3 4

256 6 2 1 3 4

512 6 2 1 3 4

1024 2 6 3 1 4

2048 6 2 1 3 4

4096 5 1 6 4 2

8192 6 7 1 9 8

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 2 4

16 1 2 4

32 1 2 4

64 2 2 4

128 2 2 4

256 2 2 4

512 2 2 4

1024 1 2 4

2048 1 2 4

4096 2 5 9

8192 1 2 9

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.2669e-02 1.2372e-05 1.29 0.62 0.83 1.21

16 6.3185e-03 1.2341e-05 2.59 0.62 0.83 1.21

32 3.1243e-03 1.2204e-05 5.24 0.64 0.86 1.24

64 1.5708e-03 1.2272e-05 10.43 0.63 0.83 1.21

128 7.8167e-04 1.2214e-05 20.96 0.63 0.88 1.27

256 3.9989e-04 1.2497e-05 40.97 0.61 0.80 1.17

512 2.0345e-04 1.2716e-05 80.53 0.61 0.80 1.21

1024 1.0699e-04 1.3374e-05 153.13 0.57 0.74 1.15

2048 5.8384e-05 1.4596e-05 280.62 0.52 0.67 1.00

4096 5.3680e-05 2.6840e-05 305.22 0.11 0.12 0.20

8192 3.3527e-05 3.3527e-05 488.69 0.09 0.08 0.18

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 1 3 4

16 2 6 1 3 4

32 2 6 1 3 4

64 6 2 1 3 4

128 6 2 1 3 4

256 6 2 1 3 4

512 6 2 1 3 4

1024 2 6 1 3 4

2048 6 2 1 3 4

4096 4 6 2 8 5

8192 4 6 5 2 8

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 2 4

16 1 2 4

32 1 2 4

64 1 2 4

128 1 2 4

256 2 2 4

512 1 2 4

1024 1 2 4

2048 1 2 4

4096 1 1 9

8192 1 2 9

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.3096e-02 1.2789e-05 1.25 0.57 0.64 1.24

16 6.8617e-03 1.3402e-05 2.39 0.51 0.63 1.12

32 3.2371e-03 1.2645e-05 5.06 0.61 0.79 1.20

64 1.6994e-03 1.3277e-05 9.64 0.55 0.64 1.26

128 8.6619e-04 1.3534e-05 18.92 0.56 0.68 1.14

256 4.6703e-04 1.4595e-05 35.08 0.51 0.56 1.04

512 2.9107e-04 1.8192e-05 56.29 0.38 0.43 0.77

1024 1.8523e-04 2.3153e-05 88.45 0.32 0.28 0.60

2048 1.3496e-04 3.3740e-05 121.40 0.22 0.25 0.46

4096 1.1768e-04 5.8840e-05 139.23 0.16 0.18 0.26

8192 8.6880e-05 8.6880e-05 188.58 0.21 0.24 0.32

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 6 1 3 4

16 6 2 1 3 4

32 2 6 1 3 4

64 2 6 1 3 4

128 6 2 1 3 4

256 2 6 3 1 4

512 2 6 1 3 4

1024 2 6 3 1 4

2048 2 6 3 1 4

4096 4 5 2 3 9

8192 4 2 3 5 6

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 2 4

16 2 2 4

32 1 1 3

64 1 2 4

128 2 2 3

256 1 2 4

512 1 2 4

1024 1 1 4

2048 1 2 5

4096 1 1 8

8192 1 1 5

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.2440e-02 1.2149e-05 1.32 0.61 0.73 1.30

16 6.1917e-03 1.2093e-05 2.65 0.61 0.70 1.30

32 3.1011e-03 1.2114e-05 5.28 0.60 0.71 1.28

64 1.5980e-03 1.2485e-05 10.25 0.58 0.68 1.28

128 8.1716e-04 1.2768e-05 20.05 0.56 0.67 1.18

256 4.3884e-04 1.3714e-05 37.33 0.52 0.58 1.12

512 2.4168e-04 1.5105e-05 67.79 0.47 0.60 0.92

1024 1.5251e-04 1.9063e-05 107.43 0.38 0.51 0.76

2048 1.0907e-04 2.7267e-05 150.22 0.23 0.28 0.48

4096 9.0360e-05 4.5180e-05 181.32 0.12 0.13 0.23

8192 6.8293e-05 6.8293e-05 239.91 0.13 0.10 0.30

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 6 2 1 3 4

16 6 2 1 3 4

32 6 2 1 3 4

64 6 2 1 3 4

128 6 2 1 3 4

256 6 2 1 3 4

512 2 6 1 3 4

1024 6 2 1 3 4

2048 2 3 6 1 4

4096 8 4 5 3 9

8192 5 9 7 8 3

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 2 4

16 1 2 4

32 1 2 4

64 1 2 4

128 2 2 4

256 1 2 4

512 1 2 4

1024 1 2 4

2048 1 2 4

4096 1 1 9

8192 1 1 7

Protocol Sensitivity Summary for Bidirectional Swap of 8192 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 1.2783e-02 1.2483e-05 1.28 0.64 0.85 1.19

16 6.3372e-03 1.2377e-05 2.59 0.65 0.87 1.20

32 3.2231e-03 1.2590e-05 5.08 0.62 0.84 1.17

64 1.5665e-03 1.2238e-05 10.46 0.66 0.90 1.21

128 7.7823e-04 1.2160e-05 21.05 0.66 0.90 1.23

256 4.0135e-04 1.2542e-05 40.82 0.63 0.83 1.20

512 2.0828e-04 1.3018e-05 78.66 0.58 0.78 1.10

1024 1.1112e-04 1.3890e-05 147.44 0.51 0.67 1.01

2048 5.9409e-05 1.4852e-05 275.78 0.47 0.57 0.93

4096 4.4747e-05 2.2373e-05 366.15 0.28 0.31 0.48

8192 2.5589e-05 2.5589e-05 640.27 0.33 0.39 0.55

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 6 2 1 3 4

16 6 2 1 3 4

32 6 2 1 3 4

64 6 2 1 3 4

128 6 2 1 3 4

256 6 2 1 3 4

512 6 2 1 3 4

1024 6 2 1 3 4

2048 6 2 1 3 4

4096 4 5 6 2 8

8192 4 5 6 2 3

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 1 3

16 1 1 3

32 1 1 3

64 1 1 3

128 1 1 3

256 1 2 3

512 1 2 4

1024 1 2 4

2048 1 2 4

4096 1 1 2

8192 1 1 2

DISCUSSION

The p690 processing node consists of 32 processors. In these tests, we examine MPI communication performance between processors in a node communicating in a ring pattern. The performance is reported for the processor with the maximum execution time, representing the worst case latency and bandwidth.