Many of the CCM2 routines have been drastically altered in the migration to PCCM2. This is particularly true of routines having to do with spectral transforms and the dynamics calculation. This section collects comments on individual routines. It is not an exhaustive list of routines modified but may prove helpful for researchers who wish to explore the code and make modifications to the algorithms.
CCM2: A call to PARLYZ has been added to initialize the parallel versions processor configuration. STEPDRV, called from CCM2, is now the driver routine for STEPON to allow dynamic memory allocation of the history buffer.
INIDAT: The first spectral transform takes place in this routine. The FFT routines are called along with the SPETRU routine which performs the spectral truncation. These sections are completely reworked for parallelism. The accumulation of the global statistics for DRY MASS and MASS OF MOISTURE are computed in INIDAT. This section of code ensures that these global statistics will be exactly the same (bit-for-bit) regardless of the number of processors used in the computation. See the reproducibility discussion above.
SCAN1: The latitude loop in SCAN1 has been split into two parts. The first calculates physics (calls PHYSDRV) for all latitudes on the processor. Then FFT's are performed on all the latitudes simultaneously allowing efficient block and transpose algorithms to be employed. The second latitude loop transforms to spectral space (calls XFORMDRV).
Since the global integrals of the prognostic fields are computed in SCAN1, there is are constructs to ensure reproducibility of the sum on different numbers of processors. This makes the code look somewhat more complicated than the standard version.
SCAN1A: The SLT calls are in SCAN1A. The major change is the addition of an hemisphere index to the extended field arrays. This provides storage of the overlap regions between processors.
BANDIJ: The output of this routine is now a local latitude index in the extended field array. When something is wrong in the physics it often shows up first with an excessive wind which blows departure points out of the range of the overlap region. An error message is printed from BANDIJ when this occurs.
OVRLAP: The message passing for the semi-Lagrangian update of the overlap regions is done in this routine. OVRLAP is called from SLTINI which is executed once per timestep. The overlap region can be increased or decreased by modifying the parameter nxpt in pmgrid.com. The amount of data sent for the overlap is dynamically varied depending on the wind conditions at a given latitude up to the maximum specified by nxpt.
RFTLON: This is the wrapper for all the FFT routines and parallel algorithms. There is no restriction in the FFT's on power of two points.
SCAN2: The global integrals involved in the moisture calculation are performed in SCAN2. Due to reproducibility considerations this routine contains added parallel constructs for the global sums. The spectral synthesis of SCAN2 is performed in SPEGRD.