Evaluation of Early Systems

Evolution of Performance in the Community Atmospheric Model (CAM)

Introduction

CAM has two primary computational phases, the dynamics and the physics. The physics has the higher serial complexity for the target problem resolutions, but it is also easily parallelized. Parallel algorithms for the dynamics require significantly more interprocessor communication than for the physics. In addition, the parallel implementation of the spectral Eulerian dynamics currently supports only a 1-D decomposition of the computational domain, limiting the number of usable processors to 64 for T42L26. By using hybrid MPI/OpenMP parallelism, more processors can be applied to the physics than to the dynamics, improving scalability.

The physics is "column-based", with computation on different columns (longitude-latitude coordinates) being independent. The work required on a column varies with location and with both diurnal and seasonal cycles, but is relatively insensitive to other factors and static load balancing is quite effective. The primary code modification examined here is a redefinition of the basic data structures used in the physics, allowing the basic unit of work, the number and choice of columns assigned to a "chunk", to be determined at runtime. The maximum number of columns in a given chunk is set at compile time. The choice of the chunk size and composition can affect both serial performance, due to the effect on memory access patterns, and parallel performance, as both MPI and OpenMP parallelism act at the chunk level. For example, smaller chunks imply more chunks and more exploitable parallelism. Optimal load balancing comes at the cost of increased interprocessor communication, remapping data from the domain decomposition used in the dynamics, but approximate load balancing that avoids interprocessor communication is also an option. For more details on chunk definition, see the CAM1 Performance tuning and benchmarking web page.

The experiments look at performance for a number of different algorithmic options:

• Original latitude-slice decomposition using 128 column chunks, MPI-only
• 16 column chunks, approximately load-balanced within the dynamics latitude-slice domain-decomposition, MPI-only
• 16 column chunks, optimal static load balancing of chunks, MPI-only
• 16 column chunks, optimal static load balancing of chunks, hybrid MPI/OpenMP

Two additional experiments were run. In CAM2.0, the CLM2 land model communicates with the atmosphere via a gather/scatter algorithm, gathering all necessary data into a single processor, remapping it to the appropriate data structure, then scattering back to the other processors. This has recently been replaced with an alltoall-based algorithm, eliminating the single processor bottleneck and decreasing the interprocessor communication overhead. This update replaced code in CAM2.0, and is not an option that can be enabled and disabled.

• 16 column chunks, optimal static load balancing of chunks, hybrid MPI/OpenMP, updated CLM2/atmosphere interchange.

• 16 column chunks, optimal static load balancing of chunks, hybrid MPI/OpenMP, updated CLM2/atmosphere interchange, updated dynamics algorithm.

The following graph describes the performance from the above experiments when run on the p690 cluster. The p690 is a 32-way SMP node, with a limitation of 2 network cards per OS image. However, the 32-way system can be partitioned into 4 8-way "logical" SMP nodes (LPARs), each of which can be populated with 2 network cards. This increases the bandwidth for internode communication significantly. In the following graph we used a single 32-way SMP node for up to 32 processors, then ran on the 8-way logical SMP nodes for 64 or more processors.

As can be seen, the performance improvement due to the new data structure first becomes significant when using 32 processors and load balancing. Using MPI/OpenMP then makes it possible to use much larger processor counts. The improved dynamics algorithms and CLM2 interchange also have significant impacts, enabling reasonable scaling out to 256 processors.

Moving from a single 32-way SMP node to the 8-way LPARs, requiring communication over the switch rather than via shared-memory, prevented performance improvement for the MPI-only experiments when going to 64 processors. The hybrid algorithm continues to show performance improvement despite the transition, especially when using the updated dynamics and CLM2 interchange, which have significantly less interprocessor communication.