What is Data Migration?

Database migration — in the context of enterprise applications — means moving your data from one platform to another.  

It’s the process of transferring data between different types of file formats, databases, or storage systems. It’s often more than just simply moving data from one system or database to another. Data migration can involve more complicated processes like data mapping and re-formatting.

Moving data to a new storage appliance to replace an old system or moving from an on-premises to a cloud storage solution are just two common examples of data migration. You might also be upgrading an application database with new hardware, necessitating data migration to new devices or equipment.

Why Data Migration?

There are many reasons you might want to move to a different platform. For example, a company might decide to save money by moving to a cloud-based database. Or, a company might find that some particular database software has features that are critical for their business needs. Or, the legacy systems are simply outdated. The process of database migration can involve multiple phases and iterations — including assessing the current databases and future needs of the company, migrating the schema, and normalizing and moving the data. Plus, testing, testing, and more testing.

Types of Data Migration

Data migrations can take a variety of different forms. Below are some of the most common types of data migration and what each one entails.

• Storage Migration. These types of migrations focus on transferring data from one storage device to a new or different one. This can take place either on-premises or in the cloud. Although storage migrations are the most straightforward type of migration on the surface, it doesn’t mean you can simply take a copy/paste approach with something like 5TB of information. You’ll need a solid plan coupled with execution, along with understanding where data is going and who should have access to it.
• Database Migration. With database migrations, you’re typically upgrading a database engine and moving the entire installed database of files to a new device. Database migrations are typically more involved than storage migrations, largely because you’re dealing with higher volumes of data that might be formatted differently. You’ll need to back up the databases, detach them from the engine, and migrate the files to a new engine. The files can then be restored to the new database and location.
• Application Migration. This type of migration can be a combination of both database and storage migrations. When you’re moving an entire software application from one place to another, they typically have databases, folders, and installation files that all need to be relocated to a different server, for instance. You may also need to work with the application vendor to take any additional steps that are specific to that software to ensure things go properly and the application functions properly post-migration.
• Wholesale vs. Incremental. While the above types of data migrations refer to the more technical aspects of migration, there are different strategic approaches to migration. You may choose a wholesale approach, where the entire migration is done at once and completed within a specified time. You can also migrate incrementally, and transfer data in phases. A wholesale approach is good because it’s simply “one and done,” while incremental migrations allow you to keep systems online during the process. You’ll need to assess your business needs and technology strategy to decide which approach is best for your organization.

Challenges

Database migration can be very complex, but with proper planning, these common challenges can be mitigated:

Challenge #1: Finding your siloed databases

You likely have many disparate databases that exist within various parts of your company. They may be in different departments and different geographies. They may have been brought in through acquisitions. Part of your task in migrating databases is to locate the disparate databases and plan how you will normalize data and convert schemas.

Challenge #2: Data loss or corruption

When migrating databases, it’s critical to ensure your data is safely moved without loss or corruption. You’ll need to plan how to test for data loss or corruption that can happen when you move data from one system to another.

Challenge # 3: Security

When you move data from one platform to another, it’s critical that the data is secure. Unfortunately, there are many nefarious actors who would love to get their hands on the personal data you have stored away. You might choose to encrypt the data or remove personally identifiable information (PII) as a part of the migration process.

Data Migration Strategy Components

Relocating sensitive or important data isn’t an affair to be taken lightly. For that reason, it’s extremely unwise to begin a data migration without a concrete plan and strategy in place. Thankfully, there are a set of tried and true principles that will serve you well when formulating your data migration strategy.

Migration Planning

Decide what data will need to be moved and the systems that will be affected. Migration planning means evaluating and anticipating the effect the project will have on business stability.

Project Initiation

Every data migration plan needs a set date and action steps for initiation. Identify key stakeholders and brief them on what role everyone will play.

Analyze Landscape 

Get a feel for how the migration fits within your larger IT ecosystem. Establish processes like how you’ll decommission legacy systems.

Design Solutions

Map out from a technical standpoint how you’re going to move which data and to where. Put processes in place to ensure data quality.

Building & Testing

This is where you’ll code the software logic that will help automate the migration. Make sure to test the code in a mirror sandbox environment.

Execute & Validate

You’ll need to demonstrate that the migration software and processes will be effective in moving data safely and is viable for business use.

Decommission & Monitoring 

Shutting down and disposing of old legacy systems is the final key component to your migration strategy. Have a plan to do so securely.

By integrating these key elements into your data migration strategy, you’ll ensure that information reaches its proper destination securely and with minimal disruption to your business and IT operations.

Planning Data Migration

Commonly, an ETL (Extract, Transform, and Load) tool is used to move the data; however, finding a good ETL tool is only part of the picture. The migration plan will determine the ultimate success of your project.

Step 1: Understand your goals

While understanding your goals may seem obvious, it’s easy to miss the nuances of the requirements for the project if you don’t document and review them with critical stakeholders. Miss this crucial step, and you may end up solving the wrong problem. Gather the business requirements, write them down, and review them with stakeholders. Once you have a clear understanding of the business requirements, you can start to consider the solutions that make sense. As a part of this process, you should define what success looks like. What will be your measures of success? Do your systems need to remain up during the migration? What percentage of your data should be migrated successfully?

Step 2: Know what you have — and what you don’t

After you have a clear sense of your goals, it’s time to assess what you have and what it will take to meet the project requirements. You need a full inventory of all your data assets, and the associated applications. This includes an understanding of upstream and downstream applications that may be affected by your proposed change. This analysis is sometimes called a gap analysis — measuring the gap between what you have and what you need.

Step 3: Data profiling — know your data

Before you migrate your data, it’s essential that you understand the current state of your data. If your existing data is in bad shape, moving it to a new system isn’t going to improve it. Profiling your data will help you to understand if there are blank or null values, if the data is unique or duplicated, or if the data patterns and values fall into a range you expect.

Understanding these aspects of your data will help to determine what sort of cleansing you’ll need to do as a part of the migration process.

Step 4: Detailed planning

Once you understand your goals, and you’ve analyzed the current state of your assets and your data, you need to get planning. For each phase of the project, you need to strategize:

• Schema matching or mapping. The high-level organization of the data in a data store is called a schema. When you move from one system to another, you need to map the schema of the source data to a schema in the target system.
• Data mapping. Think of the schema mapping, but a little more granular. When you map data, you’ll need to consider the source field’s data types. You will likely need to transform data types and cleanse the data as a part of this process.
• Recovery plan. Each stage of the data migration should include recovery options. Plan how you can roll back changes during extraction, transformation, and loading. You may choose to use a staging area or move data in batches to ensure that you can safely recover your data.
• Test planning. As a part of your project planning, you should have a test plan in place. Consider what steps will need to be tested before you can safely move data. Will you use a staging area or a test environment before you move data? How will you verify that the data quality is acceptable? What amount of error is acceptable? Consider resiliency — how much downtime or latency is acceptable?
• Go live plan. This a high-level plan that encompasses the steps to go live — ensuring that the systems are prepared, the personnel is available and trained to perform the migration, affected stakeholders have been notified, and a time window for the move has been scheduled.
• Security planning. You need to plan how you will ensure the security of your data while it is being migrated. Additionally, just as with traditional on-prem solutions, considerations should be made regarding how applications secure objects and data. For instance, how can you ensure data can be secured at the appropriate granularity (row or column level)? Do you need to mask data or ensure that PII (Personally Identifiable Information) is removed? Consider the end-to-end handling of the data from moving it (whether you need encryption in motion and at rest), to how long it should be securely retained, and what systems can access it during transit.

Selecting the Right Team of Experts

Along with an ideal data migration solution, skilled experts are also required to support the data migration process, which is unique to the organizational requirements.

For more information on the ideal data migration solution for your organization, contact us