Developed by Next Pathway and Yellowbrick
Yellowbrick Data and Next Pathway, an industry leader in cloud migration, have partnered on an end-to-end solution that streamlines your move from your legacy Teradata enterprise data warehouse platform to Yellowbrick Data Warehouse — alleviating the need for extensive manual development and migration efforts. With Teradata outsourcing support to IBM--which, ironically, has endlessly confused its own customers about the Netezza roadmap--there's never been a better time to activate an "Off Teradata" initiative
This document is intended for customers seeking a migration path for key workloads from Teradata to Yellowbrick Data Warehouse. This guide will provide customers with best practices and key considerations when planning and executing their migration.
Yellowbrick Data Warehouse is an advanced, massively parallel (MPP), SQL database designed for the most demanding batch, real-time, ad hoc, and mixed workloads. It can run complex queries at up to petabyte scale across numerous nodes, with guaranteed sub-second response times. Yellowbrick was conceived with the goal of optimizing price/performance. It's not uncommon for customers to see their workloads run tens or hundreds of times faster at a fraction of the cost compared to cloud-only or legacy data warehouses.
Once pioneers in the Relational Database Management System (RDBMS) space, Teradata is now deemed as legacy. Organizations with ambitions of moving to cloud-native environments for data warehouse and advanced analytic capabilities are left behind with the legacy Teradata world, including high licensing costs, inability to perform advanced analytics, and their complex library of functions and commands. These include BTEQ, FastLoad, MultiLoad, TPT, and various others. Simply put, migration from Teradata is not a trivial task for customers today.
To help customers modernize their data warehouse quickly and easily, Next Pathway designed the SHIFT™ Migration Suite with Teradata specifically in mind and have solved for the end-to-end challenges customers experience when moving from Teradata to Yellowbrick
Next Pathway’s migration methodology focuses on the three core steps of migration:
SHIFT > scans and catalogs legacy data sources, including ETL pipelines, scheduler jobs, and downstream consuming applications, to uncover actionable insights to plan your migration efficiently.
SHIFT ANALYZER assesses various legacy application code types to create inventories of all objects, define complexity, and provide automation rates in order to right-size your migration.
SHIFT TRANSLATOR automates the translation of complex workloads, including SQL, Stores Procedures, ETL, and various other code types for various source and target platforms.
SHIFT JET INTERPRETER serves as a migration accelerator to get customers off of Teradata by eliminating the need to re-write Teradata utilities like BTEQ and FLOAD, and thus, move these utilities off of the migration critical path.
Migration planning is incredibly important to justify the migration and define the most efficient migration approach. Without defining the right migration strategy and plan upfront, customers often hit major migration bottlenecks during later phases of the migration project, which either stall or derail the migration project entirely
An effective migration strategy in plan helps answer questions including:
To answer these, and many other, questions that often come up during early parts of migration planning, we encourage customers to employ a data-driven and automated planning process.
Next Pathway’s SHIFT™ > and SHIFT™ ANALYZER are automated tools to accelerate this process and help quickly define answers to these, and many other, questions to define the right migration plan.
SHIFT™ > is used to identify the workloads within the Teradata environment that are migration candidates, in order to define the appropriate migration approach. Further, SHIFT™ > will provide a lineage view for the data pipelines feeding Teradata, to develop the migration plan for repointing those pipelines to Yellowbrick.
SHIFT™ ANALYZER is then used to scan the Teradata applications to identify the complete code inventory, complexity of objects (including DDL, DML, Stored Procedures, Functions, among others), and “x-factors” that will require custom solutions when moving to Yellowbrick.
Both SHIFT™ > and SHIFT™ ANALYZER are important utilities to help define the appropriate migration strategy, project timeline, as well as help to define the right size of Yellowbrick environment required to replace Teradata.
Code translation and data migration are two steps that often go hand in hand in this phase of the Teradata migration project.
There are two core data migration steps: historical data migration, and on-going data ingestion. Historical data migration refers to the one-time load of historical data from the legacy Teradata environment, at a point in time, while on-going data ingestion refers to the delta, or incremental, data loads, once all of the source system data feeds are repointed to the target platform.
Key considerations for data migration that will vary from project to project may include:
Code translation involves automating the process to alleviate manual development efforts to write table, queries, and other transformations happening within the Teradata environment.
This step involves employing TRANSLATOR, which will automatically translate all Teradata SQL and Stored Procedure objects to Yellowbrick syntax, with accuracy of at least 95% out of the box. Further, our Professional Services team leverages a vast knowledge base of solutions to help address common Teradata implementation scenarios.
For Teradata specifically, an approach must also be taken for Teradata’s vast library of proprietary utilities that handle data movement within the EDW. For these utilities, the JET INTERPRETER will handle the interpretation and execution of these Teradata utilities like BTEQ and TPT with precision and speed, matching or exceeding existing business SLAs, in order to remove these discrepancies from the migration critical path.
Lastly, for Teradata environments dependent on third-party ETL tools like Informatica or DataStage, TRANSLATOR can be employed to automatically translate and repoint the legacy data pipelines (that are moving data from source systems into Teradata) to now point against the Yellowbrick environment. This task cannot be underestimated. Traditionally ETL repointing involves complicated manual refactoring efforts to unpack each pipeline, convert legacy data transformation logic to Yellowbrick syntax, as well as update source and target connectors. TRANSLATOR handles all aspects of this conversion process automatically, thus alleviating this typical migration bottleneck from the migration timeline.
Once data and code has been migrated, organizations must prioritize testing and validation, as well as cutover tasks to ensure the legacy Teradata environment can be decommissioned.
Testing focuses primarily on evaluating or assessing the quality and completeness of the various data and code elements migrated in the earlier phases. This should be an iterative process aligned with the prior phases to deliver a complete and validated migration solution according to a defined migration schedule outlined in a Migration Plan for the engagement
The key objectives of the Testing phase should include:
The helps accelerate this phase by automating data validation and hash-level attribute comparison between the Teradata and Yellowbrick environments. By automating this process, this helps organizations get to later phases of testing faster, including systems integration and business acceptance testing.
The key consideration to remember at this phase is to budget as much time as possible for testing and validation to ensure your business partners are happy.
Naturally, as testing is accelerated and performance/validation are accounted for, organizations can then decide on their ‘parallel run’ strategy to determine the length of time required to keep an active-active set-up between Teradata and Yellowbrick, before fully decommissioning the Teradata environment.
Together, Next Pathway and Yellowbrick are providing Teradata customers with the industry’s fastest and more effective migration path off of their legacy Teradata EDW to Yellowbrick’s data warehouse for distributed clouds. By leveraging the Migration Suite, customers can benefit from automation at every phase of their migration, including planning, code translation and testing/validation to accelerate the time to cut-over.
Benefits of Migration Suite offers over manual migration