#ifndef MxNCtrlPort_h_seen #define MxNCtrlPort_h_seen // We use these AVIs #include "DistArrayDescriptor.h" #include "MxNDataHandle.h" namespace gov { namespace cca { /* $Id: MxNCtrlPort.h,v 1.5 2002/04/08 01:12:44 kohl Exp $ */ /** * MxN Parallel Data Redistribution Port * * Connection & Transfer Methods * * CCA MxN Working Group */ class MxNCtrlPort : public virtual gov::cca::Port { public: /** * MxN Interface * * The purpose of this interface is to allow CCA components to * identify and exchange data elements among parallel, decomposed * data objects. One parallel component with "M" instances might * want to share or correlate data with another parallel component * with "N" instances. In some cases M == N, and the data mapping * is straightforward, although M and N could have different data * decompositions which would require remapping the data elements. * In other cases, N == 1, and the parallel data is collected into * a central location, such as for most visualization purposes. * When M != N, then a full "redistribution" of the parallel data * objects is required, to map each element of one data distribution * to a corresponding element in another data distribution. * * There is no implicit support for any data "munging" here, * such as spatial or temporal interpolation or units conversions. * These functions are outside the scope of MxN data exchange * and should be implemented in another interface, likely as a * "filter" service, component or port. * * As the CCA definitions for Data Objects becomes more complete, * this MxN interface should be expected to support a wide variety * of data organizations and decompositions, including both * structured and unstructured meshes. But for the purposes * of this initial interface description, the "Data Object" is * left open and unspecified. Ultimately, the MxN interface * will require sufficient information from a given Data Object * interface to be able to: * * 1. Identify the owner/location of any specific data element. * * 2. Extract and Assign the value(s) associated with a specific * data element. * * As long as all necessary information is available to support * these operations, then the MxN interface can function. Such * information includes, but may not be limited to: pointers to * actual data values, the data type(s), the dimensionality and * cardinality of any arrays, the allocated data size, leading * dimensions and boundary/ghost regions of any locally stored * subarrays, the data layout (as in Row Major or Column Major), * the element ids, coordinates, and vertex displacement of values * for an unstructured mesh, the spatial grid type and mapping, * some instantaneous temporal reference value, the units of the * data values, the processor topology, the decomposition types * along each axis for structured meshes, and the details of each * parallel instance's portion of a data distribution (logical * processor address, block or cycle size for rectangular * decompositions, and any explicit decomposition bounds or * patches). * * Note that some of the information above may be used purely for * error checking and validation purposes in the MxN interface * (e.g. spatial grid and units), but this information will be * essential for *other* support interfaces (such as interpolation * and units conversion) that must be present in any practical * implementation. * * Typical use of this interface will proceed as follows (see * interface specification below for method argument descriptions): * * Parallel components A and B "decide" to exchange parallel data. * How this happens is a mystery left up to the applications * programmer, but suffice to say that this could be done via * explicit component connections at composition time, to choose * which components / data objects should be coupled. Or this * could be arranged dynamically at run-time via some programmatic * control, given some method extensions to this MxN interface. * * To allow any sharing and exchange of parallel data, each * parallel data object must be registered with an instance of * the MxN implementation - say, an "MxN-er" (which could be a * port on a component or a system service, etc). Data objects * can be registered by any parallel component that has access * to a given MxN-er (see MxNDataPort.h). Once all data objects * have been registered, the MxN-er can support a variety of * dynamic MxN interconnections among the parallel component * instances. * * Similar to the base "decision" to share parallel data, one * other exchange must occur to actually pass some information * between the parallel components, A and B (or elsewhere to a * third party) about the actual data objects that are to be * selected for a given MxN connection. One of the parallel * components, on either side of the MxN connection, (or else * some third party) must be responsible for specifying the source * and destination data objects for a connection, and when and * how the data is to actually be transferred. This can be * accomplished via explicit method invocations over user-defined * ports that pass data object handles or references, or could * be handled by an additional MxN interface method, such as * "getRegisteredData()", that could return a list of all data * handles that have been registered to the given MxN instance. * * After the proper data handles have been exchanged/delivered, * then a "communication schedule" needs to be generated for * coordinating the communication between the parallel instances * of A and B. The communication schedule is determined based * on the pair of data decompositions for the given data objects. * The schedule can be re-used for any similar data objects or * connections. Only one side of an MxN connection, or some * third party, need set up this communication schedule. (Either * a "push" or "pull" model is possible, as is an externally * controlled coordination). * * To be efficient, and to be amenable to typical SPMD programming * style, the "createCommSched()" call is made collectively by * any or all parallel instances of the coordinating component * cohort. The same opaque communication schedule handle is * returned to all collective callers. This handle references * the MxN mapping information for the given pair of decompositions: * * All A: createCommSched( Bdata.decomp, Adata.decomp, * csHandle ); * * Next, given this handle to a communication schedule and the * data handles for the MxN-registered source and destination * data objects, a specific MxN "connection" can be made. The * "makeConnection()" method is used to construct a parallel * communication channel for moving data values from the source * data object to the destination data object. The creation of * this channel includes performing a synchronization among the * parallel A and B component instances, as well as setting the * "frequency" of MxN transfers (only for persistent MxN channels). * Note that each A and B can have their own transfer frequency * to account for any significant difference in their relative * iteration timings, or for algorithmic reasons (in this example, * A has a frequency of "4" and B has a frequency of "5", so A * will send data every 4 iterations to B's 5). Each side can * also have its own synchronization override settings, to control * the overhead from expensive synchronization operations that may * not be deemed necessary for a given connection (in this example, * A is prevented from synchronizing because B does not require it). * * Like the createCommSched() method, this invocation happens * collectively, on one side of the MxN connection or via some * third party controller. A connection handle is returned, * for use in actually executing transfers, or in later releasing * the connection: * * All A: makeConnection( Bdhandle, Adhandle, csHandle, * NoSynch, Default, 4, 5, cHandle ); * * Subsequent to the makeConnection() call, any synchronization * specified in the invocation will now be active. The parallel * instances on either side of the MxN connection will exchange * a simple flow control protocol (as part of the dataReady() * method execution - see below) to maintain any required * synchronization of their relative "iteration counts" or timing. * * When an actual transfer of data elements is desired from A to B, * the "requestTransfer()" method must be invoked to request the * transfer. This method can be invoked by one of the connected * parallel component cohorts or by the third party component * (i.e. the component that created the communication schedule * and made the connection :-). The requestTransfer() method is * non-blocking and returns immediately, before the transfer is * completed. A "transfer" handle is returned for monitoring the * progress of the transfer. (The waitTransfer() and testTransfer() * methods can be used to wait for or check the completion of the * transfer, respectively. However, these calls are not strictly * necessary for completion of the transfer.) * * All A: requestTransfer( cHandle, tHandle ); * . . . * waitTransfer( tHandle ); * * Once the transfer request has been posted, the data elements * will be communicated from source to destination as each * parallel instance become "ready". The instances of A and B * must each indicate that their portion of the data object is * "ready" for the transfer, i.e. the parallel data object is * in some "consistent" state for reading or writing. The * "dataReady()" method is used to indicate this state (see * MxNDataPort.h). During an invocation of dataReady(), * (or between calls to the non-blocking "dataReadyBegin()" * and companion "dataReadyEnd()" method), the MxN-er will * process all pending transfers that involve the given data * object. * * The transfer of elements need not take place "en masse" as * a fully synchronized operation. As each "pair" of mapped * parallel instances (one from A, one from B - as specified in * the communication schedule) becomes "ready", the data elements * mapped between them can be transferred. * * Note that calls to dataReady() or dataReadyEnd() will *block* * under MxN "flow control" until ALL pending transfers involving * the given data object are completed. For "one-shot" connections * (frequency == 0) a single pending transfer will be posted for * each invocation of requestTransfer(). For "periodic" transfers * (frequency > 0), a series of pending transfers will be generated * from a single invocation of requestTransfer(); as the iteration * count or "time" associated with the given frequency elapses * (as counted by invocations of dataReady() or dataReadyBegin()), * a new transfer request is automatically posted. Calling the * requestTransfer() method again for a periodic connection will * reset the iteration count offset for triggering transfers at * the given frequency. * * More details of the methods in this interface are described * below. */ /** * mxnCommSchedHandle Type * * Someday we may need something more here, but for now it's an int. */ typedef int mxnCommSchedHandle; /** * int createCommSched( DistArrayDescriptor *dst_decomp, * DistArrayDescriptor *src_decomp, * mxnCommSchedHandle &csHandle ) * * Create a Communication Schedule for use in MxN Parallel Data * Transfer. The schedule is generated based on the mapping * between the two given parallel data decompositions, and an * opaque handle is returned which can subsequently be passed * to the makeConnection() method to actually initiate an MxN * communication "channel". The createCommSched() method is * called "collectively" by all parallel instances in a given * cohort. * * @param dst_decomp Input the data decomposition information for * the destination data objects. * @param src_decomp Input the data decomposition information for * the source data objects. * @param csHandle Output a handle to identify the communication * schedule or mapping between the two decompositions. * @return execution status (0 if OK, !=0 if error) */ virtual int createCommSched( DistArrayDescriptor *dst_decomp, DistArrayDescriptor *src_decomp, mxnCommSchedHandle &csHandle ) = 0; /** * int freeCommSched( mxnCommSchedHandle csHandle ) * * Free the given Communication Schedule, as previously created * by a call to createCommSched(). (Also a collective call.) * * @param csHandle Input the handle to the communication schedule * to be freed. * @return execution status (0 if OK, !=0 if error) */ virtual int freeCommSched( mxnCommSchedHandle csHandle ) = 0; /** * int extrapolateCommSched( mxnCommSchedHandle csHandle, * DistArrayDescriptor *new_dst_decomp, * DistArrayDescriptor *new_src_decomp, * mxnCommSchedHandle &new_csHandle ) * * Duplicate and "tweak" a Communication Schedule for use in * MxN Parallel Data Transfer. The new schedule is generated * based on the mapping between the two new given parallel data * decompositions (if present), and may be implemented as a slight * variation on the existing schedule. If either "new_dst_decomp" * or "new_src_decomp" are NULL references, then the existing * source or destination decompositions in "csHandle" are re-used. * A new opaque handle is returned which can subsequently be * passed to makeConnection() to actually initiate an MxN * communication "channel". The extrapolateCommSched() method * is called "collectively" by all parallel instances in a given * cohort. * * @param csHandle Input a handle to an existing communication * schedule or mapping between two decompositions. * @param new_dst_decomp Input the new data decomposition * information for the destination data objects * (or NULL). * @param new_src_decomp Input the new data decomposition * information for the source data objects * (or NULL). * @param new_csHandle Output a new handle to identify the * "extrapolated" communication schedule or mapping * between the two (possibly new) decompositions. * @return execution status (0 if OK, !=0 if error) */ virtual int extrapolateCommSched( mxnCommSchedHandle csHandle, DistArrayDescriptor *new_dst_decomp, DistArrayDescriptor *new_src_decomp, mxnCommSchedHandle &new_csHandle ) = 0; /** * mxnSynch Enumerated Type * * Enumerated type for specifying the synchronization required * when making a MxN channel connection, as specified in the * makeConnection() method. When not "Default", can be * used to force a specific connection synchronization type. * * Can have the values: * Default, Synch and NoSynch */ enum mxnSynch { Default, Synch, NoSynch }; /** * mxnConnHandle Type * * Someday we may need something more here, but for now it's an int. */ typedef int mxnConnHandle; /** * Time Type * * This should really be its own object / AVI. * But int is close enough for SC... :-) */ typedef int Time; /** * int makeConnection( mxnDataHandle dst, mxnDataHandle src, * mxnCommSchedHandle csHandle, * enum mxnSynch dst_synch, enum mxnSynch src_synch, * Time dst_freq, Time src_freq, * mxnConnHandle &cHandle ) * * Create a connection or "communication channel" between two * parallel data objects, as specified using two data handles * returned by regsiterData() and a corresponding "communication * schedule" returned by createCommSched(). This connection * is to be used for an MxN Transfer of data elements from the * "source" data object to the "destination" data object. * This routine does *not* actually move any data, but rather * sets up the desired MxN communication channel (and any * synchronization) for use by the requestTransfer() method. * * @param dst Input handle for the destination data object. * @param src Input handle for the source data object. * @param csHandle Input the handle (as returned by the * createCommSched() method) to the communication * schedule or mapping between the two data * decompositions, to be used to perform this MxN * Transfer. * @param dst_synch Input indicates whether a synchronization * is required among the instances of the parallel * cohort at the destination side of the connection. * The purpose of this synchronization is to make sure * that "consistent" collections of elements are * transferred. If set to "Default", then the * synchronization type specified for the destination * data object (in the call to regsiterData()) will * be used for all synchronization requirements. * If *not* set to "Default", then dst_synch will * *override* the default destination data object * synchronization settings and either force / not force * synchronization at the destination throughout the * lifetime of the connection. A synchronization type * of "Synch" implies that a loose synchronization will * be maintained among the given parallel instances * of the cohort. If "NoSynch" then each of the * parallel instances can execute independently, * transferring their data elements as needed without * any synchronization within the cohort. * (Synchronization here is determined by common * iteration counts on all instances of a given * parallel cohort.) * @param src_synch Input indicates whether a synchronization * is required among the instances of the parallel * cohort at the source side of the connection. * (Same as dst_synch, but at the other side of the * connection. :-) * @param dst_freq Input the periodic "frequency" at which each * specific data transfer is to occur at the destination * data object. If freq is 0, then this is a "one-shot" * transfer and no subsequent transfer requests will be * issued automatically after the initial request is * posted (see requestTransfer()). If freq is > 0, * then this informs the MxN-er to automatically generate * new transfer requests at the given frequency after * the initial transfer is posted. The frequency can * either be a scalar that increments some local * iteration count in each parallel instance, or can * represent some notion of simulated "time" associated * with each instance of the given data object. * @param src_freq Input the periodic "frequency" at which each * specific data transfer is to occur at the source * data object. Typically the same as "dst_freq", * but can be used to prevent a slowdown between two * parallel components with dramatically different * iteration timings. The dst_freq and src_freq * parameters must either *both* be equal to 0, or else * *both* greater than zero - "one-shot" and "periodic" * connections cannot be "mixed". * @param cHandle Output a handle for this MxN connection, as * needed for actual MxN transfer requests over this * communication channel (via requestTransfer()), * or to release the communication channel (via * releaseConn()) when no more transfers are needed. * @return execution status (0 if OK, !=0 if error) */ virtual int makeConnection( mxnDataHandle dst, mxnDataHandle src, mxnCommSchedHandle csHandle, enum mxnSynch dst_synch, enum mxnSynch src_synch, Time dst_freq, Time src_freq, mxnConnHandle &cHandle ) = 0; /** * int releaseConn( mxnConnHandle cHandle ) * * Release the current MxN data transfer connection: cease any * loose synchronization that the MxN-er may be maintaining, * release any flow control between the source and destination * data objects, and clean up any internal state / storage for * this particular parallel data transfer channel. * * @param cHandle Input the data connection handle returned by * makeConnection(). * @return execution status (0 if OK, !=0 if error) */ virtual int releaseConn( mxnConnHandle cHandle ) = 0; /** * int changeFreqs( mxnConnHandle cHandle, * Time new_dst_freq, Time new_src_freq ) * * Modify the frequency of transfer for a connection or * "communication channel" between two parallel data objects, * as created by a call to makeConnection(). This routine * updates the source and destination transfer frequencies * of the given connection without breaking any ongoing * synchronization. * * @param cHandle Input the data connection handle returned by * makeConnection(). * @param new_dst_freq Input the new periodic "frequency" at which * each specific data transfer is to occur at the * destination data object (see makeConnection()). * @param new_src_freq Input the new periodic "frequency" at which * each specific data transfer is to occur at the source * data object (see makeConnection()). * @return execution status (0 if OK, !=0 if error) */ virtual int changeFreqs( mxnConnHandle cHandle, Time new_dst_freq, Time new_src_freq ) = 0; /** * (Potential Method...?) * int modifyConn( mxnConnHandle cHandle, * mxnCommSchedHandle csHandle, * enum mxnSynch dst_synch, enum mxnSynch src_synch, * Time dst_freq, Time src_freq ) * * Modify a connection or "communication channel" between two * parallel data objects, as created by a call to makeConnection(). * This routine updates the properties of the given connection * without breaking any ongoing synchronization. * * @param cHandle Input the data connection handle returned by * makeConnection(). * @param csHandle Input the handle (as returned by the * createCommSched() method) to the new communication * schedule or mapping between the two data * decompositions, to be used to perform subsequent MxN * Transfers. * @param dst_synch Input indicates the new synchronization property * for the destination side of the connection (see * makeConnection()). * @param src_synch Input indicates the new synchronization property * for the source side of the connection (see * makeConnection()). * @param dst_freq Input the new periodic "frequency" at which each * specific data transfer is to occur at the destination * data object (see makeConnection()). * @param src_freq Input the new periodic "frequency" at which each * specific data transfer is to occur at the source * data object (see makeConnection()). * @return execution status (0 if OK, !=0 if error) */ /* * virtual int modifyConn( mxnConnHandle cHandle, * mxnCommSchedHandle csHandle, * enum mxnSynch dst_synch, enum mxnSynch src_synch, * Time dst_freq, Time src_freq ) = 0; */ /** * mxnTransferHandle Type * * Someday we may need something more here, but for now it's an int. */ typedef int mxnTransferHandle; /** * int requestTransfer( mxnConnHandle cHandle, * mxnTransferHandle &tHandle ) * * Request the execution of an actual MxN Transfer of data elements * over the given communication channel, from the "source" data * object to the "destination" data object. Once the request for * the transfer has been posted, the transfer will occur when * each parallel instance, on both sides of the connection, * has called "dataReady()" to indicate the readiness of the data * object to have data elements read or written. * * This operation assumes that there is *no* discrepancy between * the global coordinate system, or the global grid/mesh associated * with "src" and "dst". (This is *not* the place for spatial or * temporal interpolation or units conversions - do that somewhere * else, before or after the MxN parallel data exchange, using a * "filter" component/port. :-) * * Invocation of the requestTransfer() method is a non-blocking * call that returns immediately, before the transfer completes. * The "tHandle" transfer handle output argument can be passed to * waitTransfer() or testTransfer() to wait for or determine if * the transfer has completed, respectively. * * @param cHandle Input the connection handle for the desired * communication channel. * @param tHandle Output a handle for this data transfer, as needed * to check completion of the data transfer via the * waitTransfer() and testTransfer() methods. * @return execution status (0 if OK, !=0 if error) */ virtual int requestTransfer( mxnConnHandle cHandle, mxnTransferHandle &tHandle ) = 0; /** * int waitTransfer( mxnTransferHandle tHandle ) * * Wait until the current MxN data transfer is completed. * This is a blocking method call that doesn't return until * the transfer completes or an error is encountered. * * @param tHandle Input the data transfer handle returned by * requestTransfer(). * @return execution status (0 if OK, !=0 if error) */ virtual int waitTransfer( mxnTransferHandle tHandle ) = 0; /** * int testTransfer( mxnTransferHandle tHandle ) * * Test if the current MxN data transfer is completed. * This is a non-blocking method call that returns immediately * with a status code that indicates whether the transfer has * completed or not, or whether an error has occurred. * * @param tHandle Input the data transfer handle returned by * requestTransfer(). * @return execution status (1 if completed, 0 if not, else error) */ virtual int testTransfer( mxnTransferHandle tHandle ) = 0; /** * int getRegisteredData( char *dataName, * mxnDataHandle &dHandle ) * * Return the MxN data handle for the given data name, if it * exists and has been registered with the given MxN-er. * Essential routine for any practical "discovery" process, * like for the Viz Proxy, to connect up to another data object. * * @param dataName Input the name of the desired data object. * @param dHandle Output the data handle to the MxN-registered * data objectof the given name, as registered with * registerData(). * @return execution status (0 if OK, !=0 if error (like "data * by that name not found")) */ virtual int getRegisteredData( char *dataName, mxnDataHandle &dHandle ) = 0; /** * int getAllRegisteredData( mxnDataHandle * &dHandles, * int &nHandles ) * * Provide a list of all data objects registered with the MxN-er. * Essential routine for any practical "discovery" process, * like for the Viz Proxy, to connect up to another data object. * * @param dHandles Output an array of data handles to MxN-registered * data objects, as registered with registerData(). * @param nHandles Output the number of elements in dHandles. * @return execution status (0 if OK, !=0 if error) */ virtual int getAllRegisteredData( mxnDataHandle * &dHandles, int &nHandles ) = 0; }; // MxNCtrlPort } // namespace cca } // namespace gov #endif // MxNCtrlPort_h_seen