Dedicated Connections to ORNL Cray X1(E): Experimental Results
USN capabilties are utilized in teating dedicated wide-area connections to ORNL Cray X1(E) supercomputer.
Complexity of Data and Execution Paths
The data-paths internal to Cray X1(E) from compute and system nodes are complex and must be interconnected and aligned appropriately with wide-area connections. The data packets from system nodes reach the Cray cross connect via SPC chnnels where they transit to FiberChannel to reach linux front-end server called Cray Network Subsystem (CNS). These data packet are then converted to Ethernet frames and sent via its 1GigE NIC into wide-area network. The execution path is also complex since each call to TCP/IP stack from an application node results in a thread migration to a system node wherein it competes with other system tasks. The impedences along both data and execution paths must be properly matched to ensure high-levels of throughputs over wide-area connections.
Experimental Setup and Results
The default wide-area connection to Cray X1(E) is via 1GigE of CNS which is shared among all its network connections. This sharing is thought to be one of the main bottlenecks to TCP throughputs. We set up a different dedicated data channel by connecting separate FiberChannel connections to a dedicated host called UCNS. This host is then connected to 1Gbps dedicated channel as shown in the following:
Over a dedicate 1Gbps connection, initial experiments revealed very low TCP throughputs (5Mbps), which could be boosted to 30-40Mbps using multiple streams of bbcp. Then we used Hurricane protocol to send the pacekts as as fast a the host node would allow; this rate was observed to be a function of otther jobs on the system. With no other jobs running, Hurricane could achieve 400Mbps but could only achieve 200Mbps with other production jobs running. These experiments helped us to isolate the bottleneck to be within the system. While these experiments are specific to Cray X1(E) configurations, some observations and conclusions are of wider importance:
A detailed description of these experiments were published in our recent paper in Annals of Telecommunications. They we also described in our presentation at 2006 Winter ESCC meeting.