Tools for moving data to your cloud infrastructure.

Luc Van SteenCloud, Cloud ComputingLeave a Comment

Luc Van SteenGetting your servers into the cloud, means you need to move your data into that cloud too.  Not an easy task, as a migration project is always executed in a phased approach.

Main challenges when it comes to data migration are getting a master copy of the data to the cloud, and keeping the data in sync during the migration project.

Here is a list of some tools I use for data migration.

1. Getting a master copy of your data to the cloud.

Some cloud provides offer services to restore your data that is being delivered on tape or disk which is great but do come with some risk.
What happens with the tape(s) or disk(s) that you have shipped after the copy?
Personally, I want to be in control over my data, and that is when following software comes into play.

To move data to the cloud I created a “data mover appliance”. Actually you need (a minimum of) two of these appliances.  One appliance will be installed on-premise (the source), the other one you need to setup in the cloud data center (the target).

datamover_to_citycloud
The picture shows a possible setup to move san storage via the data mover appliance to local disks on the virtual machines.

This data mover appliance is a basic Linux server, with iSCSI and FC support to access the LUNs.
On top of that you need to install a tool that is called “Fast Data Transfer” (FDT).
FDT is an application for efficient data transfers which is capable of reading and writing at disk speed over wide area networks (with standard TCP). It is written in Java, runs an all major platforms and it is easy to use. The software is free and can be downloaded from http://monalisa.cern.ch/FDT/.

tip The most efficient way to transfer data is NOT to transfer it at all.  Make sure you have a clear view on what needs to be in scope of the
data migration.

 

2. Keep the data in sync during the migration project.

Once you have your master copy ready, you still need to keep that data in sync.

Again, I use the same data mover appliance.  Only difference is I don’t use FDT by the well-know rsync which runs as a cronjob for each lun or datastore I want to sync. Be aware that rsync only supports a one-way sync.  For two-way syncronization, I use unison.

warning-2-256Moving databases is always a challenge.  My preferred option here is to setup a temporary replication.
And speaking of databases, City Cloud is now also offering MSSQL 2014 images.

 

There are also commercial services that are worthwhile considering like Aspera, Vision, … These products have a lot more capabilities but they come with a price tag.

One last point, data migrations aren’t easy so needless to say that for complex (data) migrations it is best to consult your preferred IT services company.