Petabytes in Motion: Ultra High Speed Transport of Media Files: A Theoretical Study and its Engineering Practice of Aspera fasp

With the explosive growth of file-based entertainment content, fast movement of massive digital data over global distances is vital to business success. The Transmission Control Protocol (TCP) that provides reliable data movement has inherent bottlenecks in performance, especially for WANs with large bandwidth, high round-trip time and packet loss. Network file systems (NFS, CIFS) and wide area block based storage protocols (iSCSI) that utilize TCP have severe performance bottlenecks over such WANs. Block-based protocols such as fibre channel provide a high-speed data pathway in the LAN but are impractical over the WAN. This paper introduces fasp (fast and secure protocol), which overcomes the network bottleneck, only to expose other bottlenecks in the end systems, such as in the disk I/O and network file systems. This “last foot” of data movement is our focus. We cover multi-Gigabit per second transfer benchmarks with major storage vendors, across SAN, LAN, and WAN, and explain the key engineering practices learned to maximize the transmission speed and efficiency.

Published

2010-10

Content type

Original Research

Keywords

High-speed file transfer, wide area networks, Aspera, fasp, protocol, TCP, 10 Gbps, storage, I/O, NFS, CIFS, iSCSI, FTP, congestion control, rate control

DOI

10.5594/M001352

ISBN

978-1-61482-944-7