Note: The CCM2 Users Documentation refers to ``history volumes'' (or sometimes ``history tapes'') and ``history files.'' A CCM ``history volume'' (``history tape'') refers to a file created by the model as it runs; such a file consists of output from one or more history periods (in other words, it is a file in ordinary computing terminology). A CCM ``history file'' is the data in a volume for one period; it is thus a logical subset of the data in a volume. The following discussion uses the CCM2 terminology unless specifically indicated.
PCCM2 history output volumes are identical in structure to those written by the NCAR CCM2. The model history volumes may be written by the parallel model in one of two modes, either volume-complete (a single file containing data from all processors) or volume-decomposed (multiple files containing spatially decomposed partial data that, when combined, contain the complete set of data for the volume).
Volume-complete mode exactly duplicates CCM2 history output format and requires no additional postprocessing to recompose history output. One processor (or a subset of processors) collects data from the other processors and then writes the history output by using the standard Fortran write mechanism for unformatted (binary) data. The parallel I/O implementation on the Intel Paragon uses the parallel file system (PFS) facilities. The write, though actually in parallel from a number of processors, is to a single binary file. The PFS software manages the file pointers in an appropriate fashion. History volumes are named h dddd, in accordance with the CCM2 User's Guide, where dddd is the number of the history volume. In volume-complete mode, the PCCM2 namelist settings to specify history frequency and the number of history writes per volume are the same as CCM2.
Volume-decomposed mode is specified by setting the namelist variable PARHIST to .TRUE.. In this mode, each processor row writes its set of local latitude records in a separate partial history volume. Later, the partial volumes may be combined to generate a complete history volume, or they may be left in partial history volume form (if postprocessing software can handle this). This mode provides a simple method of exploiting parallel I/O on systems where processors can write to different devices. For example, on the IBM SP2, processors may write to their local disks or to separate I/O nodes if available. In volume-decomposed mode, data for each partial history volume is collected and written by the ``western-most'' processor in each row of the mesh. In other words, the model generates PP_NPROC_LT partial volumes, where PP_NPROC_LT is the number of processors decomposing the latitudinal dimension.
The name of a partial history volume is appended with a four-digit processor row-identifier. In volume-decomposed mode, the volume with row identifier ``0000'' contains the CCM history header and records containing data from the latitudes processed on the first row of processors; the other partial volumes contain no header but only latitude records. In volume-decomposed mode, only one history write is made per volume (equivalent to MFILT = 1 in the namelist). This allows recombination of the output by simple concatenation:
cat h0001.* > h0001
When you are using volume-decomposed mode, the namelist variable NREFRQ should be set higher than the default of 1. Otherwise, the model will generate restart data each time it writes a history file to a volume. A suitable value for NREFRQ when PARHIST is set to .TRUE. is 5. That is, the model will generate restart data after every five history writes. By contrast, in volume-complete mode ( PARHIST = .false.), the default number of history writes per file is five (i.e., MFILT = 5), so it is not necessary to modify NREFRQ.