COMMTEST SP3-200 Point-to-Point Communication Performance

Performance Studies using

COMMTEST

IBM SP3-200 SWAP Performance

(unordered swap of 2MB message using MPI and US between nodes)

(performance measured per processor when all processors in node communicating)

Date/Person:	May 25, 2000 / P. Worley
Platform:	IBM SP3 at National Energy Research Scientific Computing Center (Gseaborg)
	256 2-way Winterhawk I SMP nodes (200 MHz POWER3 with 4MB L2 cache, equivalent to RS/6000 Model 260
Environment:	AIX 4.3.2; POE 2.4.0.12
Communication Library:	MPI over the switch using User Space
SWAP size:	262144 REAL*8 floating point values each direction
Message size:	Largest - 262144 REAL*8 floating point values
	Smallest - 256 REAL*8 floating point values
Processors:	0 and 2
	1 and 3
Latency Definition:	(T₁₀₂₄-T₅₁₂)/512
Model Error Range:	[1,1024]
Results:

unordered simple swap

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 79.15 63.51 85.3%

1 iter. 80.82 61.45 33.7%

10 iter. 85.38 67.45 34.7%

unordered swap using nonblocking send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 93.61 63.15 39.0%

1 iter. 93.87 62.95 40.4%

10 iter. 93.73 66.69 37.7%

cache inv. w/overlap 93.19 59.85 40.4%

1 iter. w/overlap 93.79 65.86 38.9%

10 iter. w/overlap 93.67 72.17 36.2%

unordered swap using nonblocking receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 94.36 66.41 47.8%

1 iter. 94.65 64.73 48.7%

10 iter. 82.66 69.65 40.5%

cache inv. w/overlap 93.12 63.57 47.8%

1 iter. w/overlap 93.57 68.17 46.0%

10 iter. w/overlap 86.10 72.00 41.1%

unordered swap using nonblocking send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 94.75 69.98 47.1%

1 iter. 93.77 66.54 48.1%

10 iter. 81.32 67.33 41.1%

cache inv. w/overlap 93.00 66.77 47.6%

1 iter. w/overlap 92.67 72.63 45.9%

10 iter. w/overlap 79.86 70.87 38.7%

unordered swap using ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 94.20 118.79 11.3%

1 iter. 92.52 117.55 11.1%

10 iter. 93.92 120.14 9.3%

cache inv. w/overlap 93.40 67.64 6.8%

1 iter. w/overlap 95.57 69.03 8.3%

10 iter. w/overlap 94.18 71.68 4.8%

unordered swap using nonblocking ready send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 93.13 125.69 10.2%

1 iter. 92.49 129.76 8.7%

10 iter. 94.04 123.38 9.4%

cache inv. w/overlap 93.96 65.38 10.6%

1 iter. w/overlap 93.78 68.88 8.5%

10 iter. w/overlap 94.15 70.71 5.4%

native sendrecv

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 93.49 63.25 49.4%

1 iter. 92.98 70.22 46.2%

10 iter. 87.69 68.30 43.3%

unordered swap using nonblocking sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 93.55 170.91 12.6%

1 iter. 93.43 171.82 12.5%

10 iter. 93.70 169.08 11.6%

cache inv. w/overlap 93.19 169.01 13.2%

1 iter. w/overlap 93.07 171.08 11.5%

10 iter. w/overlap 92.91 168.20 11.0%

unordered swap using nonblocking receive with sync. send

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 93.25 163.47 27.8%

1 iter. 92.77 165.85 26.5%

10 iter. 82.81 166.05 19.6%

cache inv. w/overlap 93.42 168.09 26.0%

1 iter. w/overlap 93.94 157.55 28.2%

10 iter. w/overlap 82.68 165.96 18.2%

unordered swap using nonblocking sync. send and receive

Data Statistics
bidirectional bandwidth estimated latency model error

(peak MByte/sec) (usec/msg) (max. rel. error)

cache inv. 93.73 166.45 26.4%

1 iter. 91.48 165.24 26.8%

10 iter. 87.37 167.11 21.9%

cache inv. w/overlap 93.45 167.83 29.1%

1 iter. w/overlap 92.77 172.50 27.4%

10 iter. w/overlap 78.10 168.14 17.9%

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (cache inv./no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 1.1567e-01 1.1296e-04 36.26 0.46 0.54 1.19

4096 8.3333e-02 1.6276e-04 50.33 0.37 0.41 1.01

8192 8.4431e-02 3.2981e-04 49.68 0.26 0.38 0.43

16384 6.7375e-02 5.2637e-04 62.25 0.22 0.32 0.40

32768 5.7710e-02 9.0173e-04 72.68 0.17 0.25 0.29

65536 5.1906e-02 1.6220e-03 80.81 0.14 0.16 0.27

131072 4.8263e-02 3.0165e-03 86.90 0.75 0.15 6.64

262144 4.6422e-02 5.8028e-03 90.35 0.07 0.11 0.14

524288 4.5490e-02 1.1372e-02 92.20 0.08 0.11 0.20

1048576 4.4749e-02 2.2375e-02 93.73 0.05 0.01 0.29

2097152 4.4269e-02 4.4269e-02 94.75 0.04 0.02 0.34

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 1 6 0 2 3

4096 1 6 2 0 3

8192 4 5 1 7 0

16384 4 5 7 1 0

32768 4 5 1 7 0

65536 4 5 1 7 0

131072 4 5 7 1 9

262144 4 5 1 7 3

524288 7 4 1 5 8

1048576 9 4 1 3 5

2097152 3 2 4 7 6

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 2 5 5

4096 2 5 5

8192 1 2 5

16384 2 2 5

32768 1 2 5

65536 1 3 9

131072 2 4 9

262144 3 4 10

524288 2 4 10

1048576 4 7 9

2097152 3 9 9

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (1 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 1.1615e-01 1.1343e-04 36.11 0.45 0.53 1.18

4096 8.2027e-02 1.6021e-04 51.13 0.38 0.42 1.06

8192 8.4858e-02 3.3148e-04 49.43 0.26 0.38 0.40

16384 6.6717e-02 5.2122e-04 62.87 0.23 0.34 0.37

32768 5.7511e-02 8.9861e-04 72.93 0.17 0.27 0.30

65536 5.1952e-02 1.6235e-03 80.73 0.13 0.21 0.26

131072 4.7877e-02 2.9923e-03 87.61 0.10 0.13 0.19

262144 4.6291e-02 5.7864e-03 90.61 0.08 0.11 0.14

524288 4.5112e-02 1.1278e-02 92.98 0.05 0.02 0.18

1048576 4.4893e-02 2.2446e-02 93.43 0.05 0.02 0.27

2097152 4.4312e-02 4.4312e-02 94.65 0.05 0.02 0.29

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 1 2 0 6 3

4096 6 2 1 3 0

8192 4 5 7 0 1

16384 4 5 7 1 0

32768 5 4 1 7 0

65536 4 5 1 7 0

131072 4 5 7 1 0

262144 5 1 4 7 6

524288 6 5 4 8 7

1048576 7 1 8 2 5

2097152 2 1 3 7 8

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 3 5 5

4096 1 5 5

8192 1 2 5

16384 1 2 5

32768 2 2 5

65536 2 4 9

131072 1 4 10

262144 3 4 10

524288 2 8 10

1048576 2 7 9

2097152 3 9 9

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (10 iterations/no overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 1.1511e-01 1.1241e-04 36.44 0.45 0.53 1.18

4096 8.0622e-02 1.5746e-04 52.02 0.38 0.42 1.06

8192 8.2942e-02 3.2399e-04 50.57 0.26 0.39 0.41

16384 6.6203e-02 5.1721e-04 63.35 0.21 0.33 0.34

32768 5.6530e-02 8.8329e-04 74.20 0.17 0.27 0.28

65536 5.1467e-02 1.6083e-03 81.50 0.13 0.22 0.24

131072 4.7694e-02 2.9809e-03 87.94 0.11 0.18 0.20

262144 4.5941e-02 5.7427e-03 91.30 0.10 0.16 0.19

524288 4.5168e-02 1.1292e-02 92.86 0.09 0.14 0.18

1048576 4.4709e-02 2.2354e-02 93.81 0.10 0.15 0.22

2097152 4.4601e-02 4.4601e-02 94.04 0.08 0.08 0.24

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 1 6 2 3 0

4096 6 2 1 3 0

8192 4 5 1 7 0

16384 4 5 1 7 0

32768 4 5 7 1 0

65536 4 5 7 1 0

131072 4 5 7 1 0

262144 4 5 7 1 0

524288 4 5 7 1 0

1048576 4 5 1 7 9

2097152 5 4 1 7 6

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 2 5 5

4096 3 5 5

8192 1 2 5

16384 2 2 5

32768 2 2 5

65536 2 4 10

131072 2 4 10

262144 4 4 10

524288 4 4 10

1048576 4 4 10

2097152 4 4 10

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (cache inv. with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 1.1634e-01 1.1361e-04 36.05 0.34 0.04 1.14

4096 8.3791e-02 1.6365e-04 50.06 0.28 0.04 0.94

8192 6.6325e-02 2.5908e-04 63.24 0.55 0.76 0.79

16384 5.7488e-02 4.4913e-04 72.96 0.39 0.50 0.63

32768 5.2788e-02 8.2481e-04 79.46 0.28 0.37 0.49

65536 4.9476e-02 1.5461e-03 84.77 0.19 0.24 0.38

131072 4.6490e-02 2.9056e-03 90.22 0.15 0.18 0.28

262144 4.5787e-02 5.7234e-03 91.61 0.10 0.13 0.20

524288 4.5661e-02 1.1415e-02 91.86 0.06 0.06 0.15

1048576 4.4907e-02 2.2454e-02 93.40 0.04 0.01 0.30

2097152 4.4637e-02 4.4637e-02 93.96 0.04 0.01 0.29

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 2 0 6 1 4

4096 2 6 4 0 1

8192 4 5 1 7 0

16384 4 5 7 1 0

32768 4 5 7 1 0

65536 5 4 1 7 0

131072 5 4 7 1 0

262144 5 4 1 7 6

524288 5 4 1 7 6

1048576 4 5 9 6 7

2097152 5 6 9 8 1

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 3 7 7

4096 3 7 7

8192 1 2 2

16384 1 2 2

32768 1 2 5

65536 2 2 6

131072 2 2 8

262144 2 4 10

524288 2 5 10

1048576 7 8 9

2097152 6 9 9

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (1 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 1.1599e-01 1.1327e-04 36.16 0.35 0.06 1.15

4096 8.3090e-02 1.6228e-04 50.48 0.29 0.03 0.98

8192 6.6792e-02 2.6091e-04 62.80 0.54 0.73 0.78

16384 5.6670e-02 4.4274e-04 74.01 0.41 0.55 0.64

32768 5.2694e-02 8.2335e-04 79.60 0.27 0.39 0.47

65536 4.9196e-02 1.5374e-03 85.26 0.21 0.30 0.40

131072 4.7083e-02 2.9427e-03 89.08 0.12 0.15 0.26

262144 4.6286e-02 5.7857e-03 90.62 0.09 0.12 0.16

524288 4.4898e-02 1.1224e-02 93.42 0.06 0.06 0.16

1048576 4.4643e-02 2.2321e-02 93.95 0.04 0.01 0.30

2097152 4.3889e-02 4.3889e-02 95.57 0.05 0.02 0.32

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 6 2 1 0 4

4096 2 4 1 0 3

8192 4 5 7 1 0

16384 4 5 1 7 0

32768 4 5 7 1 0

65536 5 4 1 7 0

131072 4 5 7 1 0

262144 4 5 1 7 2

524288 4 5 1 7 2

1048576 4 7 2 8 1

2097152 4 6 8 1 5

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 1 5 7

4096 1 6 7

8192 1 2 2

16384 2 2 2

32768 2 2 5

65536 1 2 5

131072 1 3 9

262144 1 4 10

524288 2 4 10

1048576 2 9 9

2097152 1 9 9

Protocol Sensitivity Summary for Bidirectional Swap of 2097152 Bytes (10 iterations with overlap)
Runtime Statistics
Msg Size min Sec min Sec/Msg max MBytes/Sec (mean-min)/min (median-min)/min (max-min)/min

2048 1.1523e-01 1.1253e-04 36.40 0.34 0.04 1.13

4096 8.0776e-02 1.5777e-04 51.93 0.29 0.04 0.98

8192 6.4844e-02 2.5330e-04 64.68 0.55 0.76 0.78

16384 5.5820e-02 4.3609e-04 75.14 0.40 0.56 0.64

32768 5.1592e-02 8.0612e-04 81.30 0.28 0.38 0.50

65536 4.8185e-02 1.5058e-03 87.05 0.21 0.31 0.40

131072 4.6275e-02 2.8922e-03 90.64 0.15 0.20 0.30

262144 4.5264e-02 5.6580e-03 92.66 0.13 0.20 0.27

524288 4.4762e-02 1.1190e-02 93.70 0.12 0.19 0.25

1048576 4.4584e-02 2.2292e-02 94.08 0.10 0.14 0.21

2097152 4.4533e-02 4.4533e-02 94.18 0.10 0.14 0.24

Five Fastest
Protocols

Msg Size 1st 2nd 3rd 4th 5th

2048 6 0 1 2 4

4096 6 1 2 4 0

8192 4 5 7 1 0

16384 4 5 7 1 0

32768 4 5 7 1 0

65536 4 5 7 1 0

131072 4 5 7 1 0

262144 4 5 7 1 0

524288 4 5 1 7 0

1048576 4 5 1 7 2

2097152 4 5 1 7 2

Number of Proctocols With
Runtimes Within X% of Min

Msg Size 1% 5% 25%

2048 2 7 7

4096 4 7 7

8192 1 2 2

16384 2 2 2

32768 2 2 5

65536 2 2 5

131072 2 3 8

262144 2 4 8

524288 2 4 9

1048576 3 4 10

2097152 3 4 10

DISCUSSION

The SP3 Winterhawk I processing node consists of two processors. MPI between processors within a node uses a shared memory implementation for communication, in contrast to communication between nodes which uses a network interface card and an external switch.