COMMTEST IBM p690 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM p690 SWAP Performance

(ordered swap of 128KB message using MPI within a node)

(performance measured per processor when all processors in node participating in a ring shift)

Date/Person:	April 4, 2002 / P. Worley
Platform:	IBM p690 at Oak Ridge National Laboratory (cheetah.ccs.ornl.gov):
	p690 32-way Turbo SMP node (1.3 GHz POWER4)
Environment:	AIX 5.1L
Communication Library:	MPI
SWAP size:	16384 REAL*8 floating point values each direction
Message size:	Largest - 16384 REAL*8 floating point values
	Smallest - 16 REAL*8 floating point values
Processors:	receive from i-1, send to i+1, i=0,1,...,31
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

ordered simple swap

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 326.24 11.21 29.7%

1 iter. 711.09 10.79 37.1%

10 iter. 939.45 11.29 34.3%

ordered swap using nonblocking send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 312.31 12.48 26.0%

1 iter. 679.27 12.10 36.0%

10 iter. 926.93 12.86 31.9%

cache inv. w/overlap 316.46 13.79 23.8%

1 iter. w/overlap 657.79 13.75 26.3%

10 iter. w/overlap 912.50 12.73 30.7%

ordered swap using nonblocking receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 316.47 11.83 28.3%

1 iter. 676.79 11.22 37.1%

10 iter. 939.75 12.17 32.9%

cache inv. w/overlap 312.28 11.33 27.2%

1 iter. w/overlap 682.88 11.28 36.8%

10 iter. w/overlap 931.84 11.64 35.2%

ordered swap using nonblocking send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 302.48 13.54 22.5%

1 iter. 665.54 12.70 34.4%

10 iter. 921.74 13.88 29.1%

cache inv. w/overlap 323.67 12.69 30.3%

1 iter. w/overlap 658.19 12.58 35.8%

10 iter. w/overlap 909.47 13.30 33.1%

ordered swap using ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 318.98 16.23 7.3%

1 iter. 673.50 15.08 7.8%

10 iter. 982.37 15.63 26.0%

cache inv. w/overlap 344.85 15.76 13.1%

1 iter. w/overlap 683.47 15.29 8.0%

10 iter. w/overlap 981.77 16.72 25.1%

ordered swap using nonblocking ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 324.31 15.99 10.2%

1 iter. 686.53 16.80 8.8%

10 iter. 971.43 16.54 26.9%

cache inv. w/overlap 333.86 17.23 10.3%

1 iter. w/overlap 722.98 17.07 9.6%

10 iter. w/overlap 959.78 18.37 26.9%

synchronous

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 309.45 15.13 25.5%

1 iter. 678.29 14.67 32.3%

10 iter. 894.21 14.45 32.6%

ordered swap using nonblocking sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 301.47 22.56 5.2%

1 iter. 702.05 22.42 12.0%

10 iter. 909.57 22.49 31.2%

cache inv. w/overlap 326.21 22.56 8.5%

1 iter. w/overlap 675.12 22.20 11.4%

10 iter. w/overlap 920.67 22.70 31.9%

ordered swap using nonblocking receive with sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 322.40 22.16 5.9%

1 iter. 686.00 22.25 11.6%

10 iter. 930.23 21.99 31.2%

cache inv. w/overlap 317.23 22.10 5.7%

1 iter. w/overlap 670.74 22.17 11.3%

10 iter. w/overlap 933.56 21.73 30.9%

ordered swap using nonblocking sync. send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 300.99 23.35 7.2%

1 iter. 654.33 23.01 11.5%

10 iter. 922.09 23.16 32.6%

cache inv. w/overlap 315.57 24.35 5.9%

1 iter. w/overlap 679.95 24.09 12.5%

10 iter. w/overlap 915.87 23.73 33.2%

ordered simple swap using sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 320.46 21.71 5.3%

1 iter. 662.94 21.49 10.9%

10 iter. 959.79 21.04 30.8%

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.3728e-02 2.3172e-05 11.05 0.49 0.40 1.05

256 1.2247e-02 2.3921e-05 21.40 0.47 0.35 1.02

512 6.6067e-03 2.5807e-05 39.68 0.43 0.33 0.92

1024 3.6392e-03 2.8431e-05 72.03 0.41 0.27 0.89

2048 2.2637e-03 3.5371e-05 115.80 0.29 0.20 0.66

4096 1.7568e-03 5.4899e-05 149.22 0.23 0.26 0.39

8192 1.3141e-03 8.2128e-05 199.49 0.14 0.17 0.23

16384 1.0981e-03 1.3726e-04 238.72 0.10 0.09 0.16

32768 1.0172e-03 2.5430e-04 257.71 0.05 0.05 0.10

65536 8.8040e-04 4.4020e-04 297.76 0.03 0.03 0.07

131072 7.9817e-04 7.9817e-04 328.43 0.03 0.03 0.05

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 1 3 6

256 0 2 1 3 6

512 0 2 1 3 6

1024 0 2 3 1 6

2048 0 2 1 3 6

4096 4 5 10 0 7

8192 4 5 0 10 2

16384 4 0 5 2 10

32768 4 10 5 8 0

65536 6 9 5 4 10

131072 10 1 5 6 0

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 2 4

256 1 2 5

512 1 2 5

1024 1 2 5

2048 1 3 6

4096 1 2 4

8192 1 1 11

16384 1 1 11

32768 2 5 11

65536 1 9 11

131072 1 10 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.2828e-02 2.2293e-05 11.48 0.51 0.40 1.09

256 1.1781e-02 2.3010e-05 22.25 0.49 0.41 1.05

512 5.9637e-03 2.3296e-05 43.96 0.50 0.38 1.03

1024 3.1432e-03 2.4556e-05 83.40 0.47 0.37 0.98

2048 1.7328e-03 2.7075e-05 151.29 0.43 0.33 0.92

4096 1.3203e-03 4.1260e-05 198.55 0.30 0.33 0.48

8192 8.5805e-04 5.3628e-05 305.51 0.22 0.24 0.33

16384 6.5460e-04 8.1825e-05 400.46 0.12 0.14 0.21

32768 5.3947e-04 1.3487e-04 485.93 0.06 0.07 0.08

65536 4.2992e-04 2.1496e-04 609.75 0.05 0.05 0.09

131072 3.8457e-04 3.8457e-04 681.65 0.04 0.04 0.10

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 1 3 6

256 0 2 1 3 6

512 0 2 1 3 6

1024 0 2 1 3 6

2048 0 2 1 3 6

4096 4 5 0 10 2

8192 4 5 0 10 7

16384 5 4 0 8 9

32768 4 5 8 9 0

65536 0 4 5 9 10

131072 10 1 0 2 7

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 2 4

256 1 2 4

512 1 1 4

1024 1 2 5

2048 1 1 5

4096 1 1 2

8192 1 2 6

16384 1 2 11

32768 2 3 11

65536 1 6 11

131072 2 7 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.3581e-02 2.3028e-05 11.12 0.47 0.36 1.02

256 1.2020e-02 2.3477e-05 21.81 0.45 0.34 0.99

512 6.0047e-03 2.3456e-05 43.66 0.47 0.37 1.02

1024 3.0249e-03 2.3632e-05 86.66 0.48 0.38 1.03

2048 1.5956e-03 2.4931e-05 164.30 0.46 0.35 0.97

4096 1.1540e-03 3.6064e-05 227.15 0.33 0.38 0.57

8192 6.7459e-04 4.2162e-05 388.60 0.27 0.31 0.42

16384 4.1735e-04 5.2168e-05 628.12 0.24 0.27 0.40

32768 3.1098e-04 7.7745e-05 842.96 0.15 0.18 0.22

65536 2.6685e-04 1.3342e-04 982.37 0.05 0.06 0.10

131072 3.1200e-04 3.1200e-04 840.20 0.04 0.04 0.08

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 1 3 6

256 0 2 1 3 6

512 0 2 1 3 6

1024 0 2 1 3 6

2048 0 2 1 3 6

4096 4 5 10 0 2

8192 4 5 10 0 2

16384 4 5 0 10 2

32768 4 5 10 0 7

65536 4 5 10 2 0

131072 7 9 2 4 5

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 2 4

256 2 2 5

512 1 2 4

1024 1 2 4

2048 1 2 4

4096 1 1 2

8192 1 1 3

16384 1 2 4

32768 1 2 11

65536 1 5 11

131072 3 6 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.4026e-02 2.3463e-05 10.91 0.48 0.38 1.11

256 1.2415e-02 2.4248e-05 21.12 0.45 0.37 1.07

512 6.8543e-03 2.6775e-05 38.25 0.40 0.31 0.95

1024 3.6248e-03 2.8319e-05 72.32 0.41 0.33 0.91

2048 2.1678e-03 3.3873e-05 120.92 0.35 0.29 0.78

4096 1.7941e-03 5.6067e-05 146.11 0.21 0.23 0.35

8192 1.3434e-03 8.3965e-05 195.13 0.11 0.12 0.17

16384 1.1307e-03 1.4134e-04 231.84 0.05 0.06 0.11

32768 1.0147e-03 2.5368e-04 258.34 0.05 0.05 0.10

65536 8.7425e-04 4.3713e-04 299.85 0.04 0.02 0.12

131072 7.9285e-04 7.9285e-04 330.63 0.02 0.02 0.06

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 0 3 1 6

256 0 2 3 1 6

512 2 0 3 1 6

1024 0 2 1 3 6

2048 0 2 1 3 6

4096 4 5 0 10 2

8192 4 5 0 10 3

16384 5 0 8 4 2

32768 4 5 0 10 8

65536 4 5 8 2 1

131072 5 3 9 4 7

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 2 4

256 2 2 4

512 1 2 5

1024 1 2 5

2048 1 1 5

4096 1 2 6

8192 1 1 11

16384 1 5 11

32768 1 5 11

65536 2 9 11

131072 3 9 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.3173e-02 2.2630e-05 11.31 0.50 0.39 1.15

256 1.1517e-02 2.2494e-05 22.76 0.52 0.44 1.19

512 5.9314e-03 2.3169e-05 44.20 0.52 0.42 1.15

1024 3.2190e-03 2.5149e-05 81.44 0.44 0.36 1.06

2048 1.7199e-03 2.6873e-05 152.42 0.44 0.38 0.99

4096 1.3516e-03 4.2237e-05 193.95 0.27 0.29 0.44

8192 8.6739e-04 5.4212e-05 302.22 0.21 0.24 0.35

16384 6.4457e-04 8.0572e-05 406.69 0.13 0.14 0.26

32768 5.4505e-04 1.3626e-04 480.95 0.05 0.07 0.08

65536 4.3465e-04 2.1733e-04 603.11 0.04 0.04 0.08

131072 3.8525e-04 3.8525e-04 680.45 0.06 0.06 0.15

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 3 1 6

256 0 2 3 1 6

512 0 2 1 3 6

1024 0 2 3 1 6

2048 0 2 1 3 6

4096 4 5 0 1 8

8192 4 5 0 7 2

16384 4 5 0 2 7

32768 4 5 10 2 9

65536 5 4 0 3 9

131072 4 5 2 0 10

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 2 4

256 1 2 4

512 1 1 4

1024 1 2 5

2048 1 1 5

4096 1 2 2

8192 1 2 7

16384 2 2 10

32768 1 4 11

65536 2 6 11

131072 1 5 11

Protocol Sensitivity Summary for Unidirectional Swap of 131072 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 2.3664e-02 2.3110e-05 11.08 0.49 0.46 1.06

256 1.1921e-02 2.3284e-05 21.99 0.49 0.47 1.06

512 6.0306e-03 2.3557e-05 43.47 0.49 0.47 1.06

1024 3.0523e-03 2.3846e-05 85.88 0.50 0.48 1.05

2048 1.5833e-03 2.4740e-05 165.56 0.49 0.47 1.03

4096 1.2774e-03 3.9918e-05 205.22 0.23 0.24 0.43

8192 7.1877e-04 4.4923e-05 364.71 0.20 0.22 0.34

16384 4.4629e-04 5.5787e-05 587.38 0.17 0.18 0.25

32768 3.3072e-04 8.2680e-05 792.65 0.11 0.12 0.21

65536 2.6701e-04 1.3351e-04 981.77 0.06 0.05 0.09

131072 3.1058e-04 3.1058e-04 844.06 0.05 0.04 0.13

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 0 2 1 3 6

256 0 2 1 3 6

512 0 2 1 3 6

1024 0 2 1 3 6

2048 0 2 1 3 6

4096 4 5 0 10 2

8192 4 5 0 10 2

16384 4 5 10 0 2

32768 4 5 10 0 8

65536 4 5 10 0 8

131072 10 1 3 8 7

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 2 4

256 1 2 4

512 1 2 4

1024 1 2 4

2048 1 1 4

4096 1 1 6

8192 1 1 8

16384 1 1 10

32768 1 2 11

65536 1 3 11

131072 1 8 11

DISCUSSION

The p690 processing node consists of 32 processors. In these tests, we examine MPI communication performance between processors in a node communicating in a ring pattern. The performance is reported for the processor with the maximum execution time, representing the worst case latency and bandwidth.