Transforming the Cloud for Big Data
Jay Migliaccio, Director of Cloud Solutions, Aspera
"BGI’s researchers can now transfer genomic data at a rate of nearly 10Gbps over a new link connecting US and China research and education networks, greatly reducing the time between sequencing and analysis"
However, cloud adoption by businesses has been limited due to the problem of moving data into and out of the cloud. Often dealing with data sets measuring tens of terabytes, organizations have had to rely on traditional means for moving big data: shipping hard disk drives to a cloud provider and hoping that they don’t get delayed, damaged or lost, or transferring data via the web using TCP-based methods like FTP or HTTP.
To become a practical option for big-data management, processing and distribution, cloud services need a high-speed transport mechanism that addresses two main bottlenecks: the degradation in WAN transfer speeds that occurs over distance using traditional transfer protocols, and the “last foot” bottleneck inside the cloud data center caused by the HTTP interfaces to the underlying object-based cloud storage.
Solving the Big data challenge
The latest advances in high-speed file transfer technologies solve both these problems, delivering the performance needed to transfer large files, or large collections of files, in and out of the cloud. Transfers occur at line speed, securely, to and from any location in the world. Files of any size and any format can be transferred over any distance, any network, and under any condition. Transfer capacity can easily scale out and back, on demand.
Transfers of any size into and out of the cloud achieve perfect efficiency, independent of network delays and robust to extreme packet loss. Users have extraordinary control over individual transfer rates and bandwidth sharing, and full visibility into bandwidth utilization. File transfer times can be guaranteed, regardless of network distance and conditions, including transfers over satellite, wireless, and unreliable long-distance international links. Complete security is built in, including secure endpoint authentication, on-the-fly data encryption, and integrity verification.
A high-speed software bridge transfers data at line speed, from source directly into cloud storage, with no hops or stops in between, enabling direct I/O in and out of cloud storage, ensuring intra-cloud I/O keeps up with the high-speed file transfer over the WAN, and transparently handling cloud-specific I/O requirements such as multi-part uploads.
Using parallel HTTP streams between the transfer server running on a cloud virtual machine and the cloud storage, the intra-cloud data movement no longer constrains the overall transfer rate. Files are written directly to cloud storage, without stop off on the cloud compute server.
These advances are transforming the way all kinds of organizations transfer and manage large amounts of data. Media and entertainment companies, for example, can quickly scale their operations without the constraints of fixed data centers to deliver more content to more subscribers quickly.
Entertaining the cloud
In a historic world, a large-scale system for high-resolution end-to-end live streaming in the cloud was deployed to deliver World a–90 2014 World Cup soccer matches to viewers’ second screens. Live video feeds from up to 24 cameras were transferred in real time using high-performance WAN transport from stadiums in Brazil to a cloud platform in Europe for real-time processing into multiple formats and bit rates for a range of online viewing formats, before being delivered to regional broadcasters and streamed to online viewers across the globe.
High-speed cloud transfer technology delivered consistent transfer of live feeds despite heavy round-trip latency and packet loss, enabling consumers to enjoy live and near-live coverage of the games from any laptop or mobile device with optimized viewing - and from multiple camera angles. During the tournament, over 27 terabytes of video data was moved at high speed in under 14 hours of transfer time.
The world’s leading promoter of mixed martial arts, Ultimate Fighting Championship, broadcasts its programming to half a billion homes in 150 countries. In addition to live match broadcasting on pay-per-view television, it also features a host of content on its website before each match, including training footage, weight cutting, and interviews with fans and fighters.
UFC found traditional FTP technology to be slow, unreliable, and unsuitable for large transfers. By deploying high-speed cloud transfer technologies to move clips from the production teams’ laptops at different venues directly to the cloud has enabled it to take advantage of the cloud without losing time waiting for files to upload. UFC now saves 50–90 percent a month in storage costs, and transfer speeds that are hundreds of times faster.
Powering medical innovation in the cloud
In the field of life sciences, research organizations can now use the cloud to exchange huge amounts of next-generation sequencing data, fostering more creative collaboration with geographically dispersed research scientists, and accelerating medical discovery. BGI, for example, has integrated high-speed transfer technology into its Easy Genomics service for faster bioinformatics in the cloud, allowing users to rapidly upload sequencing data to the cloud for processing and to download completed projects.
BGI’s researchers can now transfer genomic data at a rate of nearly 10Gbps over a new link connecting US and China research and education networks, greatly reducing the time between sequencing and analysis. During a recent live demo, BGI transferred 24GB of genomic data from Beijing to UC Davis in California in under 30 seconds. A file of the same size sent over the public internet took more than 26 hours.
GenoSpace has launched a cloud-based system for connecting individual patients’ genomic analyses with clinical laboratory results and case history to identify patterns and relationships between genetic signatures and therapeutic options. High-speed cloud transfer technology moves genomics data to the GenoSpace system for analysis and reporting, and disseminates information back to researchers, physicians, clinical labs and patients, while maintaining patient privacy and speeding potential access to individualized treatment.
From fueling a new age of data-driven innovation in life sciences to powering radically new consumer viewing experiences, high-speed file transfer technologies are unleashing the full potential of the cloud – not just for media companies and life sciences organizations with global footprints, but by organizations of all types and sizes. As more organizations across industries of all kinds recognize the benefits that high-speed cloud transfer can deliver, cloud adoption is set to soar.
Employing Cloud Technology for Effective Learning
Jeffrey Cepull, CIO and VP for Information Resources, Philadelphia University
Three Companies that had to Prove their Disaster Recovery Plan would Work
Pat O’Day, CTO, Bluelock
A Chief Data Officer's View of Storage Strategies
Jim Tyo, VP & CDO, Nationwide
Moving Towards a More Agile and Outcomes Based Model
Raiford Smith, Vice President, Energy Technology and Analytics, Entergy