One of the key business applications of ThreatMetrix (a LexisNexis Risk Solution), a global digital fraud and identity-authentication service, is an online portal accessed by thousands of users around the world. The portal serves over 5,000 global brands, helping them verify more than 20 billion financial transactions each year.
Via the portal, ThreatMetrix customers query a 300TB multi-tenant database over 25,000 times per day, with up to 1TB of new data ingested daily in real-time. Often, several hundred users might be simultaneously writing queries. And many queries are also complex, accessing over six months of stored data spread across millions of records in a multi-billion-row table with over 1,000 columns. To handle this heavy mixed workload, legacy Apache Impala and Greenplum systems required a complex table-partitioning method to manage contention between the complex queries and continual data inserts. Admins juggled over 1,000 partitions each day, trying to keep the database’s performance at an acceptable level as required by SLAs. Even with this intricate partition management, the query engine couldn’t keep up, and writes would queue up.
Performance: After replacing Impala and Greenplum with Yellowbrick, portal users noticed performance improvements immediately: Complex queries that used to take 30 seconds now execute in under 2 seconds, and short queries that executed in several seconds are now measured in milliseconds—all while Yellowbrick constantly ingests terabytes of new data.
Management: Yellowbrick also dramatically simplified data management. ThreatMetrix uses Yellowbrick pool management to easily reallocate resources and respond to spikes or unusual usage patterns, allowing it to optimally allocate resources to queries and database operations. Because Yellowbrick also does away with the need for complex partition management, a significant amount of admin time is saved.
High Availability and Resiliency: The previous ThreatMetrix portal experienced significant outages, often forcing the company to restart the database and taking its customers offline. ThreatMetrix deployed Yellowbrick in each of its global regions, allowing it to shift workloads seamlessly between clusters when needed.
Footprint: In addition to increased performance, simplified management, and better stability, the Yellowbrick footprint is significantly smaller: A single 15-node, 6u Yellowbrick on-premises instance replaced 31 40-core machines, reducing the amount of needed hardware by a third, requiring 20x less memory and a quarter of the compute core.
For ThreatMetrix customers, speed and stability improved dramatically, allowing them to run their businesses faster and more effectively. Best of all, Yellowbrick helped sharply reduce ThreatMetrix support costs and delivered its customers significant capital and operational savings.
- Matthias Baumhof,
Replacement of legacy analytic database with Yellowbrick on-premises data warehouse