Shift: Self-Healing Independent File Transfer

In high-end computing environments, remote file transfers of very large data sets to and from computational resources are commonplace, as users typically are widely distributed across different organizations and must transfer in data to be processed, and transfer out results for further analysis. Local transfers of this same data across file systems are frequently performed by administrators to optimize resource utilization when new file systems come online or storage becomes imbalanced between existing file systems. In both cases, files must traverse many components on their journey from source to destination, where there are numerous opportunities for performance optimization as well as failure. A number of tools exist for providing reliable and/or high-performance file transfer capabilities, but most do not support local transfers, require specific security models and/or transport applications, are difficult for individual users to deploy, and/or are not fully optimized for highest performance.