Move your big data lightning fast and at a low cost!

facebooktwittergoogle_plusredditlinkedinmail
All applications need at least some data to function. Big data applications like gaming analytics, weather modeling, and video rendering as well as tools such as flume, MapReduce, and database replication are obvious examples of software that process and move large amounts of data. Even a seemingly simple website might have to copy dictionaries, articles, pictures, and all sorts of data across VMs, and that can add up to a lot. Sometimes that data must be accessible through a file system, and traditional tools like secure copy (scp) might not be enough to handle the increasing data sizes.
Big data applications commonly read data from disk, transform it, then use a tool like secure copy (scp) to move it to another VM for further computation. Scp is limited by several factors, from its threading model to encryption hardware in the virtual machines CPU’s, and is eventually limited by the Persistent Disk read and write quota per virtual machine. It can transfer close to 128MBytes/sec (single stream) or 240MBytes/sec (multiple streams).
This is what the current flow looks like:
Screen Shot 2015-04-30 at 7.22.49 AM.png
this post we will describe an innovative new way of transferring large amount of data between VMs. Google Compute Engine Persistent Disks offer a feature called Snapshots,  which are point-in-time copies of all data on the disk. While snapshots are commonly used for backups, they can also rapidly be turned into new disks. These new disks can be attached to a different running virtual machine than where they were created, thereby moving the data from the first virtual machine to the second. The process of transferring data using snapshot involves three simple steps:
  1. Create a snapshot from the source disk.
  2. Download the snapshot to a new destination disk.
  3. Attach and mount the destination disk to a virtual machine.


Using Persistent Disk Snapshot you can move data between your virtual machines at speeds upwards of 1024MBytes/sec (8Gbps). That’s an up to 8x speed increase over scp! Below is a graph that shows a comparison of moving data with secure copy and snapshots.
Diagram: Data Transfer comparisons


The huge advantage of the snapshot-based approach stems from the performance of Google Cloud Platform’s Persistent Disk snapshotting process. The following graph shows the time it takes to snapshot Persistent Disks of increasing size, along with the effective throughput (PD-SSD was used in this experiment). The time it takes to do the snapshot is roughly the same up to 500GB (bars in the graph) and steps up at the 1TB mark. Therefore, the effective throughput (i.e., “speed”) of the snapshot process, which is shown as the line in the graph, increases almost linearly.

enterprise cloud cloud computig enterprise cloud computing

This article has 1 Comment

Leave a Reply

Your email address will not be published. Required fields are marked *