COMMTEST IBM p690 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM p690 SWAP Performance

(ordered swap of 8KB message using MPI within a node)

(performance measured per processor when all processors in node participating in a ring shift)

Date/Person:	April 4, 2002 / P. Worley
Platform:	IBM p690 at Oak Ridge National Laboratory (cheetah.ccs.ornl.gov):
	p690 32-way Turbo SMP node (1.3 GHz POWER4)
Environment:	AIX 5.1L
Communication Library:	MPI
SWAP size:	1024 REAL*8 floating point values each direction
Message size:	Largest - 1024 REAL*8 floating point values
	Smallest - 1 REAL*8 floating point values
Processors:	receive from i-1, send to i+1, i=0,1,...,31
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

ordered simple swap

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 116.89 11.61 38.1%

1 iter. 171.19 11.79 34.9%

10 iter. 294.74 11.20 51.9%

ordered swap using nonblocking send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 121.32 12.49 31.2%

1 iter. 176.45 11.84 27.1%

10 iter. 283.02 12.53 43.3%

cache inv. w/overlap 111.96 12.38 20.8%

1 iter. w/overlap 184.17 12.78 35.8%

10 iter. w/overlap 291.22 12.42 44.1%

ordered swap using nonblocking receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 109.89 11.25 29.0%

1 iter. 175.37 11.00 32.8%

10 iter. 292.29 11.97 48.6%

cache inv. w/overlap 123.21 11.73 28.4%

1 iter. w/overlap 195.39 11.02 26.3%

10 iter. w/overlap 286.06 11.85 45.4%

ordered swap using nonblocking send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 107.42 13.78 36.3%

1 iter. 173.63 13.58 46.5%

10 iter. 279.80 14.03 47.9%

cache inv. w/overlap 106.05 12.96 29.9%

1 iter. w/overlap 161.15 13.16 42.9%

10 iter. w/overlap 283.80 13.37 46.3%

ordered swap using ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 119.46 15.64 22.8%

1 iter. 174.67 16.00 34.1%

10 iter. 361.09 15.40 67.9%

cache inv. w/overlap 139.99 15.90 27.2%

1 iter. w/overlap 188.38 15.67 36.0%

10 iter. w/overlap 394.42 16.75 80.6%

ordered swap using nonblocking ready send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 118.00 17.51 25.2%

1 iter. 170.19 17.07 35.5%

10 iter. 353.77 16.44 71.0%

cache inv. w/overlap 130.57 17.19 33.2%

1 iter. w/overlap 210.41 17.11 43.9%

10 iter. w/overlap 386.55 18.30 86.4%

synchronous

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 112.76 15.02 20.7%

1 iter. 148.16 13.98 25.3%

10 iter. 257.98 14.42 45.4%

ordered swap using nonblocking sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 116.64 23.15 33.0%

1 iter. 193.45 22.59 53.3%

10 iter. 285.02 22.24 77.4%

cache inv. w/overlap 104.26 22.29 29.9%

1 iter. w/overlap 175.87 21.93 47.1%

10 iter. w/overlap 285.20 22.53 78.4%

ordered swap using nonblocking receive with sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 114.78 22.33 31.3%

1 iter. 162.52 22.26 44.2%

10 iter. 296.48 21.95 79.5%

cache inv. w/overlap 118.98 22.02 32.0%

1 iter. w/overlap 167.71 21.91 44.8%

10 iter. w/overlap 296.11 21.73 78.5%

ordered swap using nonblocking sync. send and receive

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 110.25 23.57 31.7%

1 iter. 176.10 23.12 49.7%

10 iter. 269.46 23.13 76.1%

cache inv. w/overlap 121.99 24.78 36.9%

1 iter. w/overlap 171.21 24.27 50.7%

10 iter. w/overlap 286.24 23.71 82.8%

ordered simple swap using sync. send

Data Statistics
unidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 120.38 21.42 31.5%

1 iter. 184.17 20.92 47.0%

10 iter. 302.95 21.10 78.0%

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 2.3529e-02 2.2978e-05 0.70 0.48 0.37 1.04

16 1.1737e-02 2.2925e-05 1.40 0.48 0.39 1.03

32 5.7578e-03 2.2492e-05 2.85 0.52 0.46 1.08

64 2.9615e-03 2.3136e-05 5.53 0.49 0.35 1.04

128 1.5103e-03 2.3599e-05 10.85 0.48 0.36 1.00

256 7.7335e-04 2.4167e-05 21.19 0.48 0.40 1.02

512 4.4576e-04 2.7860e-05 36.76 0.42 0.35 0.87

1024 2.6641e-04 3.3302e-05 61.50 0.38 0.36 0.80

2048 1.6872e-04 4.2180e-05 97.11 0.36 0.40 0.65

4096 1.6616e-04 8.3080e-05 98.60 0.09 0.10 0.19

8192 1.2623e-04 1.2623e-04 129.80 0.07 0.09 0.13

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 0 1 3 6

16 0 2 1 3 6

32 0 2 1 3 6

64 0 2 1 3 6

128 0 2 1 3 6

256 0 2 1 3 6

512 0 2 1 3 6

1024 2 0 1 3 6

2048 0 1 2 3 6

4096 4 0 2 5 1

8192 0 5 10 4 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 2 4

16 1 2 4

32 1 2 4

64 1 2 5

128 1 2 4

256 1 2 4

512 1 1 5

1024 1 2 4

2048 1 1 4

4096 1 2 11

8192 1 4 11

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 2.2901e-02 2.2364e-05 0.72 0.49 0.39 1.06

16 1.1429e-02 2.2322e-05 1.43 0.48 0.36 1.06

32 5.6560e-03 2.2094e-05 2.90 0.49 0.39 1.09

64 2.7990e-03 2.1867e-05 5.85 0.53 0.50 1.10

128 1.3766e-03 2.1509e-05 11.90 0.55 0.47 1.16

256 6.8984e-04 2.1557e-05 23.75 0.60 0.56 1.19

512 3.7425e-04 2.3391e-05 43.78 0.53 0.48 1.05

1024 2.1760e-04 2.7200e-05 75.29 0.44 0.41 0.87

2048 1.3667e-04 3.4167e-05 119.88 0.34 0.40 0.65

4096 1.2359e-04 6.1793e-05 132.57 0.12 0.12 0.34

8192 8.9253e-05 8.9253e-05 183.57 0.09 0.10 0.16

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 0 1 3 6

16 0 2 1 3 6

32 0 2 1 3 6

64 0 2 1 3 6

128 0 2 1 3 6

256 0 2 1 3 6

512 0 2 1 3 6

1024 0 2 1 3 6

2048 2 3 0 1 6

4096 4 5 1 0 2

8192 9 5 1 4 3

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 2 4

16 1 2 4

32 2 2 4

64 1 2 3

128 1 1 4

256 1 1 3

512 1 1 4

1024 1 2 4

2048 1 3 4

4096 1 2 10

8192 2 3 11

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 2.3180e-02 2.2637e-05 0.71 0.48 0.36 1.04

16 1.1711e-02 2.2873e-05 1.40 0.46 0.35 1.01

32 5.6880e-03 2.2219e-05 2.88 0.50 0.40 1.08

64 2.8923e-03 2.2596e-05 5.66 0.48 0.38 1.03

128 1.4388e-03 2.2481e-05 11.39 0.49 0.37 1.06

256 7.2768e-04 2.2740e-05 22.52 0.48 0.40 1.05

512 3.5001e-04 2.1876e-05 46.81 0.54 0.42 1.15

1024 1.8072e-04 2.2590e-05 90.66 0.54 0.42 1.13

2048 9.5585e-05 2.3896e-05 171.41 0.52 0.41 1.10

4096 7.5689e-05 3.7845e-05 216.46 0.29 0.32 0.52

8192 4.4984e-05 4.4984e-05 364.22 0.27 0.30 0.44

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 2 1 3 6

16 0 2 1 3 6

32 0 2 1 3 6

64 0 2 1 3 6

128 0 2 1 3 6

256 0 2 1 3 6

512 0 2 1 3 6

1024 0 2 1 3 6

2048 0 2 1 3 6

4096 4 5 10 0 8

8192 4 5 0 10 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 1 4

16 1 2 4

32 1 2 4

64 2 2 4

128 1 2 4

256 1 2 4

512 1 1 4

1024 1 1 3

2048 1 1 4

4096 1 2 2

8192 1 2 3

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 2.2960e-02 2.2421e-05 0.71 0.51 0.41 1.20

16 1.1121e-02 2.1721e-05 1.47 0.56 0.44 1.25

32 5.6352e-03 2.2013e-05 2.91 0.54 0.47 1.22

64 2.8496e-03 2.2263e-05 5.75 0.53 0.46 1.20

128 1.4083e-03 2.2004e-05 11.63 0.57 0.52 1.29

256 7.5655e-04 2.3642e-05 21.66 0.52 0.51 1.14

512 4.3685e-04 2.7303e-05 37.50 0.47 0.38 0.98

1024 2.6819e-04 3.3523e-05 61.09 0.35 0.29 0.75

2048 1.7660e-04 4.4150e-05 92.77 0.29 0.18 0.67

4096 1.4587e-04 7.2933e-05 112.32 0.23 0.25 0.36

8192 1.1239e-04 1.1239e-04 145.78 0.20 0.22 0.29

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 2 1 3 6

16 0 2 3 1 6

32 0 2 3 1 6

64 0 2 3 1 6

128 0 2 1 3 6

256 0 2 3 1 6

512 0 1 2 3 6

1024 0 2 1 3 6

2048 2 0 3 1 4

4096 5 4 0 2 1

8192 4 5 10 0 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 2 4

16 1 1 4

32 1 2 4

64 1 1 4

128 1 1 4

256 1 2 4

512 1 1 4

1024 1 2 5

2048 1 1 7

4096 1 2 5

8192 1 1 7

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 2.2712e-02 2.2180e-05 0.72 0.51 0.40 1.18

16 1.1052e-02 2.1586e-05 1.48 0.54 0.42 1.23

32 5.4625e-03 2.1338e-05 3.00 0.56 0.43 1.30

64 2.7629e-03 2.1585e-05 5.93 0.55 0.42 1.24

128 1.3721e-03 2.1439e-05 11.94 0.56 0.45 1.32

256 6.8833e-04 2.1510e-05 23.80 0.61 0.54 1.25

512 3.7821e-04 2.3638e-05 43.32 0.53 0.45 1.15

1024 2.0941e-04 2.6177e-05 78.24 0.48 0.41 0.93

2048 1.3752e-04 3.4380e-05 119.14 0.33 0.28 0.69

4096 1.1483e-04 5.7413e-05 142.68 0.19 0.22 0.38

8192 8.4027e-05 8.4027e-05 194.99 0.13 0.12 0.32

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 2 0 3 1 6

16 0 2 3 1 6

32 0 2 3 1 6

64 0 2 3 1 6

128 0 2 1 3 6

256 0 2 1 3 6

512 0 2 1 3 6

1024 0 2 1 3 6

2048 2 1 0 3 5

4096 4 5 0 2 10

8192 4 5 1 10 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 2 2 4

16 1 2 4

32 1 1 4

64 1 1 4

128 1 1 4

256 1 1 4

512 1 1 4

1024 1 2 4

2048 1 3 4

4096 2 2 9

8192 2 2 10

Protocol Sensitivity Summary for Unidirectional Swap of 8192 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

8 2.3277e-02 2.2732e-05 0.70 0.50 0.47 1.08

16 1.1740e-02 2.2930e-05 1.40 0.49 0.46 1.06

32 5.8156e-03 2.2717e-05 2.82 0.50 0.47 1.08

64 2.8216e-03 2.2044e-05 5.81 0.55 0.52 1.16

128 1.4424e-03 2.2537e-05 11.36 0.52 0.50 1.11

256 6.8754e-04 2.1486e-05 23.83 0.59 0.57 1.20

512 3.5287e-04 2.2054e-05 46.43 0.57 0.54 1.22

1024 1.8565e-04 2.3206e-05 88.25 0.51 0.51 1.07

2048 9.7221e-05 2.4305e-05 168.52 0.51 0.50 1.09

4096 7.4455e-05 3.7227e-05 220.05 0.32 0.35 0.53

8192 4.3505e-05 4.3505e-05 376.60 0.28 0.33 0.47

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

8 0 2 1 3 6

16 0 2 1 3 6

32 0 2 1 3 6

64 0 2 1 3 6

128 0 2 1 3 6

256 0 2 1 3 6

512 0 2 1 3 6

1024 0 2 1 3 6

2048 0 2 1 3 6

4096 4 5 10 0 8

8192 5 4 10 7 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

8 1 2 4

16 1 2 4

32 1 2 4

64 1 1 4

128 1 2 4

256 1 1 4

512 1 1 4

1024 1 1 4

2048 1 1 4

4096 1 1 2

8192 2 2 2

DISCUSSION

The p690 processing node consists of 32 processors. In these tests, we examine MPI communication performance between processors in a node communicating in a ring pattern. The performance is reported for the processor with the maximum execution time, representing the worst case latency and bandwidth.