Some people are suggesting that with the explosion of “Big Data,” data warehouses are becoming a thing of the past, that they are the “new mainframes.” The perception is understandable, as big data is certainly sucking the oxygen out of the server room. But for anyone taking data seriously, the old maxim “perception is reality” is not acceptable, especially if you are working in the financial industry.
So, is there any fact-based validity to the idea that data warehouses are becoming the new legacy system, holding financial services companies back from new insights and greater efficiencies? I argue the notion is absurd, like saying there should be no rail transportation because someone invented the automobile. A closer investigation shows that data warehouses, overshadowed by excitement about the new possibilities of tapping into new kinds of data, have actually lost none of their appeal or real-world power, particularly as compliance, risk management, regulatory reporting and operational efficiency remain hot topics.
To compare data warehouses and big data, it is useful to first establish definitions for each: Big data refers to managing vast quantities – 100 terabytes or more – of unstructured data, generated from machines, social media and the like; data warehouses are designed to handle structured data, pulled from trading, customer management, portfolio accounting and other operational systems. Further, the vast majority of financial firms are not dealing with anywhere close to 100 terabytes of actual data. In fact, few organizations approach the outer limits of data warehouse capabilities – most involve no more than 15 terabytes, and those at that level could likely greatly reduce it with an improved data model.
Beyond size, big data falls short for financial firms because two of the primary drivers for technology investment are compliance and risk management, both of which require limited volumes of structured data from operational systems, and therefore are best served by data warehouses. In addition, because all data in a data warehouse is traceable, compliance departments at any moment can quickly and easily determine where and when information was derived, for an accurate audit trail, something big data is not designed to handle.
[For more on It's More About "Wide Data" Than Big Data]
When one considers these differences in evaluating the unique capabilities of each data management approach, it becomes clear that news of data warehouses’ demise is premature. In fact, there is reason to believe the future for data warehouse is brighter than ever.
We speak with many CIOs and Lead Architects and some are looking into big data technology because they feel their data warehouses are becoming too big. On closer inspection, we find that bloated data warehouses can be easily fixed by applying financial domain-specific knowledge to how the data is organized.
Even without applying domain-specific knowledge, it is my firm belief that that the real data problem for the financial industry is not vast volumes of data; it is the complexity of the data. Sure, there are times when financial firms need to store huge volumes of data – historical tick data for example. But for the vast majority of financial organizations, the real trick is to figure out how to get multiple operational systems to work with each other without losing or duplicating data. Introducing the hot new technology may be appealing, but solving the complexity puzzle effectively is better served by the data warehouse.
The story of big data is still largely one of potential (though incredibly exciting potential, most would agree) and no bank should see it as a replacement for current data warehouse initiatives that have proven their value time and time again.
Instead, financial technology professionals should continually seek out any and all solutions that could make their firms more profitable, productive, secure and compliant. This means not saying “this or that,” but instead “this and that.”
Jonas Olsson is the founder and CEO of Graz, a data warehouse provider for the financial industry.