Migration Guide:

Migrating from Teradata to Yellowbrick, the data warehouse for distributed clouds

Developed by Next Pathway and Yellowbrick

Introduction

Yellowbrick Data and Next Pathway, an industry leader in cloud migration, have partnered on an end-to-end solution that streamlines your move from your legacy Teradata enterprise data warehouse platform to Yellowbrick Data Warehouse — alleviating the need for extensive manual development and migration efforts. With Teradata outsourcing support to IBM--which, ironically, has endlessly confused its own customers about the Netezza roadmap--there's never been a better time to activate an "Off Teradata" initiative

This document is intended for customers seeking a migration path for key workloads from Teradata to Yellowbrick Data Warehouse. This guide will provide customers with best practices and key considerations when planning and executing their migration.

Modernizing your data warehouse with Yellowbrick

Yellowbrick Data Warehouse is an advanced, massively parallel (MPP), SQL database designed for the most demanding batch, real-time, ad hoc, and mixed workloads. It can run complex queries at up to petabyte scale across numerous nodes, with guaranteed sub-second response times. Yellowbrick was conceived with the goal of optimizing price/performance. It's not uncommon for customers to see their workloads run tens or hundreds of times faster at a fraction of the cost compared to cloud-only or legacy data warehouses.

modern data warehouse

End to end migration approach, powered by Next Pathway

Once pioneers in the Relational Database Management System (RDBMS) space, Teradata is now deemed as legacy. Organizations with ambitions of moving to cloud-native environments for data warehouse and advanced analytic capabilities are left behind with the legacy Teradata world, including high licensing costs, inability to perform advanced analytics, and their complex library of functions and commands. These include BTEQ, FastLoad, MultiLoad, TPT, and various others. Simply put, migration from Teradata is not a trivial task for customers today.

To help customers modernize their data warehouse quickly and easily, Next Pathway designed the SHIFT™ Migration Suite with Teradata specifically in mind and have solved for the end-to-end challenges customers experience when moving from Teradata to Yellowbrick

Next Pathway’s migration methodology focuses on the three core steps of migration:

  • a. Plan Effectively
  • b. Automate Code Conversion and Data Migration
  • c. Prioritize Validation and Testing
modern data warehouse

Key features of Next Pathway’s SHIFT™ Migration Suite

Crawler Image

SHIFT > scans and catalogs legacy data sources, including ETL pipelines, scheduler jobs, and downstream consuming applications, to uncover actionable insights to plan your migration efficiently.

Shift Analyzer Image

SHIFT ANALYZER assesses various legacy application code types to create inventories of all objects, define complexity, and provide automation rates in order to right-size your migration.

Shift Translator Image

SHIFT TRANSLATOR automates the translation of complex workloads, including SQL, Stores Procedures, ETL, and various other code types for various source and target platforms.

Shift Jet Interpreter Image

SHIFT JET INTERPRETER serves as a migration accelerator to get customers off of Teradata by eliminating the need to re-write Teradata utilities like BTEQ and FLOAD, and thus, move these utilities off of the migration critical path.

Shift Tester Image

SHIFT TESTER automates key tasks in the testing life-cycle when executing and optimizing workloads within the cloud.

Migration planning

Migration planning is incredibly important to justify the migration and define the most efficient migration approach. Without defining the right migration strategy and plan upfront, customers often hit major migration bottlenecks during later phases of the migration project, which either stall or derail the migration project entirely

An effective migration strategy in plan helps answer questions including:

  • 1. How long will the migration project take, and how much will it cost?
  • 2. What type of migration strategy should I employ?
  • 3. What are the ‘x-factors’ in Teradata that I need to plan for?
  • 4. What workloads should be prioritized as migration candidates?

To answer these, and many other, questions that often come up during early parts of migration planning, we encourage customers to employ a data-driven and automated planning process.

Next Pathway’s SHIFT™ > and SHIFT™ ANALYZER are automated tools to accelerate this process and help quickly define answers to these, and many other, questions to define the right migration plan.

SHIFT™ > is used to identify the workloads within the Teradata environment that are migration candidates, in order to define the appropriate migration approach. Further, SHIFT™ > will provide a lineage view for the data pipelines feeding Teradata, to develop the migration plan for repointing those pipelines to Yellowbrick.

SHIFT™ ANALYZER is then used to scan the Teradata applications to identify the complete code inventory, complexity of objects (including DDL, DML, Stored Procedures, Functions, among others), and “x-factors” that will require custom solutions when moving to Yellowbrick.

Both SHIFT™ > and SHIFT™ ANALYZER are important utilities to help define the appropriate migration strategy, project timeline, as well as help to define the right size of Yellowbrick environment required to replace Teradata.

modern data warehouse

Automate code translation and data migration

Code translation and data migration are two steps that often go hand in hand in this phase of the Teradata migration project.

There are two core data migration steps: historical data migration, and on-going data ingestion. Historical data migration refers to the one-time load of historical data from the legacy Teradata environment, at a point in time, while on-going data ingestion refers to the delta, or incremental, data loads, once all of the source system data feeds are repointed to the target platform.

Key considerations for data migration that will vary from project to project may include:

  • Determining how much historical data is required to migrate, which will be dictated by migration approach defined during planning
  • Employing the appropriate historical migration tool and method, that gets data migrated as quickly and efficiently as possible
  • Determining if a new on-going data ingestion tool is required, depending on whether the legacy Teradata environment is reliant on Teradata utilities like FLOAD and MLOAD for data import, or a third-party ETL tool like Informatica that are still within the migration strategy

Code translation involves automating the process to alleviate manual development efforts to write table, queries, and other transformations happening within the Teradata environment.

This step involves employing TRANSLATOR, which will automatically translate all Teradata SQL and Stored Procedure objects to Yellowbrick syntax, with accuracy of at least 95% out of the box. Further, our Professional Services team leverages a vast knowledge base of solutions to help address common Teradata implementation scenarios.

For Teradata specifically, an approach must also be taken for Teradata’s vast library of proprietary utilities that handle data movement within the EDW. For these utilities, the JET INTERPRETER will handle the interpretation and execution of these Teradata utilities like BTEQ and TPT with precision and speed, matching or exceeding existing business SLAs, in order to remove these discrepancies from the migration critical path.

Lastly, for Teradata environments dependent on third-party ETL tools like Informatica or DataStage, TRANSLATOR can be employed to automatically translate and repoint the legacy data pipelines (that are moving data from source systems into Teradata) to now point against the Yellowbrick environment. This task cannot be underestimated. Traditionally ETL repointing involves complicated manual refactoring efforts to unpack each pipeline, convert legacy data transformation logic to Yellowbrick syntax, as well as update source and target connectors. TRANSLATOR handles all aspects of this conversion process automatically, thus alleviating this typical migration bottleneck from the migration timeline.

modern data warehouse

Validation, testing, and cutover

Once data and code has been migrated, organizations must prioritize testing and validation, as well as cutover tasks to ensure the legacy Teradata environment can be decommissioned.

Testing focuses primarily on evaluating or assessing the quality and completeness of the various data and code elements migrated in the earlier phases. This should be an iterative process aligned with the prior phases to deliver a complete and validated migration solution according to a defined migration schedule outlined in a Migration Plan for the engagement

The key objectives of the Testing phase should include:

  • To find and document defects and problems in the migrated application
  • Validate that requirements are implemented properly and the solution works as intended
  • Ensure existing business SLAs are either met or exceeded between Teradata and Yellowbrick

The helps accelerate this phase by automating data validation and hash-level attribute comparison between the Teradata and Yellowbrick environments. By automating this process, this helps organizations get to later phases of testing faster, including systems integration and business acceptance testing.

The key consideration to remember at this phase is to budget as much time as possible for testing and validation to ensure your business partners are happy.

Naturally, as testing is accelerated and performance/validation are accounted for, organizations can then decide on their ‘parallel run’ strategy to determine the length of time required to keep an active-active set-up between Teradata and Yellowbrick, before fully decommissioning the Teradata environment.

Summary

Together, Next Pathway and Yellowbrick are providing Teradata customers with the industry’s fastest and more effective migration path off of their legacy Teradata EDW to Yellowbrick’s data warehouse for distributed clouds. By leveraging the Migration Suite, customers can benefit from automation at every phase of their migration, including planning, code translation and testing/validation to accelerate the time to cut-over.

Benefits of Migration Suite offers over manual migration

  • Planning: Whether you’re translating complex workloads, or testing, ’s automation parallelizes development efforts to deliver faster, more efficiently and in a truly agile model
  • Testing: By applying automation to the migration life-cycle and removing manual efforts, human error is eliminated, and standardization is enforced to respect and enforce various governance and security controls for your organization
  • Translation: ’s automation brings HUGE benefits over manual efforts, in order for your business to focus their time on downstream tasks that matter most
  • End-to-end approach: automates key parts of the end-to-end migration life cycle to get you to cut-over faster, including ETL and BI conversion, in addition to code translation and testing
  • Scalability: ’s unique architecture enables it to support any custom requirement or edge-cases with the simple addition of new rules, and thus avoid typical project bottlenecks