COMMTEST IBM p690 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM p690 SWAP Performance

(unordered swap of 128KB message using MPI within a node)

(performance measured per processor when all processors in node participating in a ring shift)

Date/Person:	April 4, 2002 / P. Worley
Platform:	IBM p690 at Oak Ridge National Laboratory (cheetah.ccs.ornl.gov):
	p690 32-way Turbo SMP node (1.3 GHz POWER4)
Environment:	AIX 5.1L
Communication Library:	MPI
SWAP size:	16384 REAL*8 floating point values each direction
Message size:	Largest - 16384 REAL*8 floating point values
	Smallest - 16 REAL*8 floating point values
Processors:	receive from i-1, send to i+1, i=0,1,...,31
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 329.58 14.24 16.7%

1 iter. 739.21 13.81 32.6%

10 iter. 1153.00 13.78 28.8%

cache inv. w/overlap 338.80 14.52 13.5%

1 iter. w/overlap 685.98 14.26 23.0%

10 iter. w/overlap 1153.19 14.71 28.5%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 340.73 12.40 15.3%

1 iter. 728.12 12.16 29.8%

10 iter. 1157.07 12.20 33.8%

cache inv. w/overlap 328.50 12.71 17.1%

1 iter. w/overlap 725.28 13.60 23.6%

10 iter. w/overlap 1168.15 13.35 26.8%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 326.04 14.76 13.7%

1 iter. 704.87 14.60 24.5%

10 iter. 1165.15 14.91 27.5%

cache inv. w/overlap 340.78 15.13 14.7%

1 iter. w/overlap 745.57 14.54 30.5%

10 iter. w/overlap 1138.32 16.07 22.5%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 358.12 23.65 18.2%

1 iter. 773.38 21.51 8.3%

10 iter. 1259.64 22.62 40.7%

cache inv. w/overlap 369.72 21.42 13.3%

1 iter. w/overlap 825.81 20.64 6.5%

10 iter. w/overlap 1211.54 23.20 36.9%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 357.42 26.21 17.0%

1 iter. 790.83 26.62 11.3%

10 iter. 1265.83 26.26 40.9%

cache inv. w/overlap 360.13 23.24 14.7%

1 iter. w/overlap 775.76 22.84 10.7%

10 iter. w/overlap 1269.86 26.10 43.0%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 341.92 11.80 13.9%

1 iter. 747.19 13.44 23.7%

10 iter. 1197.06 12.54 29.7%

unordered swap using nonblocking sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 330.10 27.63 17.5%

1 iter. 709.80 25.70 15.0%

10 iter. 1173.70 26.19 41.7%

cache inv. w/overlap 328.66 27.37 12.4%

1 iter. w/overlap 707.81 26.13 14.2%

10 iter. w/overlap 1130.38 26.25 45.2%

unordered swap using nonblocking receive with sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 339.04 25.05 15.4%

1 iter. 740.63 24.78 9.8%

10 iter. 1130.70 24.78 41.8%

cache inv. w/overlap 336.52 25.41 17.9%

1 iter. w/overlap 715.15 24.14 9.4%

10 iter. w/overlap 1143.43 24.78 43.2%

unordered swap using nonblocking sync. send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 328.35 27.39 17.2%

1 iter. 674.66 27.06 17.1%

10 iter. 1148.12 27.33 37.4%

cache inv. w/overlap 322.96 27.76 14.1%

1 iter. w/overlap 724.90 27.98 14.7%

10 iter. w/overlap 1155.89 27.29 37.4%

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.3241e-02 1.2931e-05 19.80 0.61 0.81 1.17

256 6.8935e-03 1.3464e-05 38.03 0.58 0.73 1.12

512 3.8301e-03 1.4961e-05 68.44 0.50 0.69 0.96

1024 2.1362e-03 1.6689e-05 122.72 0.44 0.63 0.87

2048 1.3888e-03 2.1700e-05 188.76 0.35 0.44 0.71

4096 1.2601e-03 3.9378e-05 208.03 0.10 0.09 0.19

8192 9.7621e-04 6.1013e-05 268.53 0.06 0.06 0.13

16384 8.1643e-04 1.0205e-04 321.09 0.08 0.11 0.12

32768 7.6823e-04 1.9206e-04 341.23 0.09 0.11 0.14

65536 7.3343e-04 3.6671e-04 357.42 0.07 0.07 0.11

131072 7.4279e-04 7.4279e-04 352.92 0.06 0.05 0.11

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 6 1 3 4

256 2 6 1 3 4

512 2 6 1 3 4

1024 2 6 1 3 4

2048 6 2 3 1 4

4096 4 6 5 8 2

8192 4 6 8 2 5

16384 4 5 6 2 7

32768 5 4 2 9 1

65536 5 4 2 8 6

131072 8 5 3 6 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 2 4

256 1 1 4

512 1 1 4

1024 2 2 4

2048 1 2 4

4096 1 1 9

8192 1 2 9

16384 1 2 9

32768 1 2 9

65536 1 3 9

131072 2 4 9

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.2747e-02 1.2448e-05 20.56 0.63 0.83 1.21

256 6.4581e-03 1.2613e-05 40.59 0.64 0.90 1.22

512 3.4401e-03 1.3438e-05 76.20 0.57 0.72 1.11

1024 1.8334e-03 1.4324e-05 142.98 0.52 0.70 0.99

2048 1.0445e-03 1.6321e-05 250.97 0.44 0.58 0.88

4096 9.8579e-04 3.0806e-05 265.92 0.09 0.08 0.20

8192 6.3101e-04 3.9438e-05 415.43 0.10 0.10 0.22

16384 4.7839e-04 5.9798e-05 547.98 0.08 0.08 0.14

32768 4.0565e-04 1.0141e-04 646.23 0.08 0.08 0.14

65536 3.5499e-04 1.7749e-04 738.46 0.08 0.10 0.14

131072 3.5869e-04 3.5869e-04 730.83 0.09 0.04 0.37

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 6 1 3 4

256 6 2 1 3 4

512 6 2 1 3 4

1024 2 6 1 3 4

2048 6 2 1 3 4

4096 4 6 8 5 2

8192 4 6 7 2 8

16384 4 5 2 7 9

32768 5 4 7 8 9

65536 4 5 1 6 7

131072 5 9 8 3 7

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 2 4

256 1 2 4

512 2 2 4

1024 1 1 4

2048 1 2 4

4096 1 2 9

8192 1 2 9

16384 1 2 9

32768 2 2 9

65536 2 3 9

131072 1 5 8

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.2792e-02 1.2492e-05 20.49 0.62 0.84 1.20

256 6.5333e-03 1.2760e-05 40.12 0.60 0.82 1.17

512 3.3530e-03 1.3098e-05 78.18 0.58 0.77 1.14

1024 1.7521e-03 1.3689e-05 149.61 0.54 0.73 1.09

2048 8.9051e-04 1.3914e-05 294.38 0.57 0.72 1.09

4096 8.0754e-04 2.5236e-05 324.62 0.14 0.16 0.22

8192 4.4877e-04 2.8048e-05 584.14 0.14 0.15 0.20

16384 2.8324e-04 3.5405e-05 925.51 0.11 0.11 0.20

32768 2.1219e-04 5.3048e-05 1235.41 0.10 0.10 0.16

65536 2.0709e-04 1.0355e-04 1265.83 0.07 0.09 0.12

131072 2.6370e-04 2.6370e-04 994.11 0.03 0.03 0.06

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 2 6 1 3 4

256 6 2 1 3 4

512 6 2 1 3 4

1024 2 6 1 3 4

2048 2 6 1 3 4

4096 4 6 8 2 7

8192 4 6 2 5 8

16384 4 6 5 2 8

32768 4 5 6 7 8

65536 5 4 6 7 3

131072 3 4 2 6 8

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 2 4

256 2 2 4

512 2 2 4

1024 2 3 4

2048 1 2 4

4096 1 1 9

8192 1 1 9

16384 1 1 9

32768 1 2 9

65536 2 2 9

131072 2 8 9

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.3402e-02 1.3088e-05 19.56 0.56 0.66 1.17

256 6.8861e-03 1.3449e-05 38.07 0.55 0.63 1.16

512 3.7324e-03 1.4580e-05 70.24 0.52 0.58 1.09

1024 2.1427e-03 1.6739e-05 122.35 0.47 0.59 1.07

2048 1.3757e-03 2.1495e-05 190.55 0.33 0.33 0.66

4096 1.2441e-03 3.8877e-05 210.72 0.12 0.13 0.20

8192 9.8247e-04 6.1404e-05 266.82 0.09 0.10 0.17

16384 8.5351e-04 1.0669e-04 307.14 0.05 0.05 0.10

32768 7.8321e-04 1.9580e-04 334.70 0.06 0.06 0.14

65536 7.2792e-04 3.6396e-04 360.13 0.09 0.09 0.19

131072 7.6263e-04 7.6263e-04 343.74 0.04 0.05 0.08

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 6 2 1 3 4

256 6 2 1 3 4

512 6 2 1 3 4

1024 2 6 1 3 4

2048 2 6 1 3 4

4096 4 5 8 2 1

8192 4 5 8 2 6

16384 4 5 3 8 2

32768 4 5 7 6 8

65536 5 4 3 8 1

131072 5 4 3 2 7

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 2 4

256 1 2 4

512 1 1 4

1024 1 2 4

2048 1 2 4

4096 1 2 9

8192 1 2 9

16384 1 4 9

32768 1 4 9

65536 1 2 9

131072 1 4 9

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.3348e-02 1.3035e-05 19.64 0.53 0.61 1.15

256 6.4823e-03 1.2661e-05 40.44 0.59 0.69 1.22

512 3.3838e-03 1.3218e-05 77.47 0.55 0.63 1.14

1024 1.8548e-03 1.4491e-05 141.33 0.49 0.61 1.02

2048 1.0525e-03 1.6446e-05 249.06 0.41 0.49 0.86

4096 9.2331e-04 2.8853e-05 283.92 0.14 0.15 0.27

8192 6.1656e-04 3.8535e-05 425.17 0.12 0.12 0.26

16384 4.6597e-04 5.8247e-05 562.57 0.09 0.10 0.20

32768 3.8127e-04 9.5317e-05 687.56 0.12 0.14 0.19

65536 3.5415e-04 1.7707e-04 740.21 0.10 0.10 0.19

131072 3.3364e-04 3.3364e-04 785.71 0.15 0.11 0.56

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 6 2 1 3 4

256 2 6 1 3 4

512 2 6 1 3 4

1024 2 6 1 3 4

2048 2 6 1 3 4

4096 4 5 6 2 8

8192 4 5 6 2 8

16384 4 5 2 6 9

32768 4 5 2 7 6

65536 4 5 8 6 9

131072 5 4 9 6 8

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 2 2 4

256 1 2 4

512 1 2 4

1024 2 2 4

2048 1 2 4

4096 1 1 8

8192 1 2 8

16384 1 1 9

32768 1 2 9

65536 2 2 9

131072 1 2 8

Protocol Sensitivity Summary for Bidirectional Swap of 131072 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

128 1.2943e-02 1.2639e-05 20.25 0.63 0.85 1.17

256 6.5852e-03 1.2862e-05 39.81 0.62 0.83 1.15

512 3.2873e-03 1.2841e-05 79.74 0.64 0.87 1.16

1024 1.7624e-03 1.3769e-05 148.74 0.56 0.78 1.04

2048 9.2116e-04 1.4393e-05 284.58 0.53 0.71 0.99

4096 8.6078e-04 2.6899e-05 304.54 0.08 0.07 0.13

8192 4.5844e-04 2.8653e-05 571.81 0.11 0.13 0.16

16384 3.0014e-04 3.7517e-05 873.41 0.06 0.05 0.12

32768 2.1732e-04 5.4330e-05 1206.26 0.08 0.07 0.13

65536 2.0643e-04 1.0322e-04 1269.86 0.09 0.10 0.12

131072 2.6554e-04 2.6554e-04 987.22 0.03 0.03 0.06

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

128 6 2 1 3 4

256 6 2 1 3 4

512 6 2 1 3 4

1024 2 6 1 3 4

2048 6 2 1 3 4

4096 4 2 5 6 8

8192 4 6 8 2 5

16384 5 6 4 8 2

32768 5 4 8 2 7

65536 5 4 2 6 9

131072 5 4 8 7 9

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

128 1 1 3

256 1 2 3

512 1 1 3

1024 2 2 4

2048 1 2 4

4096 1 4 9

8192 1 1 9

16384 1 4 9

32768 1 2 9

65536 1 2 9

131072 2 5 9

DISCUSSION

The p690 processing node consists of 32 processors. In these tests, we examine MPI communication performance between processors in a node communicating in a ring pattern. The performance is reported for the processor with the maximum execution time, representing the worst case latency and bandwidth.