Data Management

11:04 AM
Jonas Olsson, Graz
Jonas Olsson, Graz

Big Data: No Replacement for the Financial Data Warehouse

News of data warehouses’ demise is premature, argues Jonas Olsson. In fact, there is reason to believe the future for data warehouse is brighter than ever.

Some people are suggesting that with the explosion of “Big Data,” data warehouses are becoming a thing of the past, that they are the “new mainframes.” The perception is understandable, as big data is certainly sucking the oxygen out of the server room. But for anyone taking data seriously, the old maxim “perception is reality” is not acceptable, especially if you are working in the financial industry.

Jonas Olsson, Graz
Jonas Olsson, Graz

So, is there any fact-based validity to the idea that data warehouses are becoming the new legacy system, holding financial services companies back from new insights and greater efficiencies? I argue the notion is absurd, like saying there should be no rail transportation because someone invented the automobile. A closer investigation shows that data warehouses, overshadowed by excitement about the new possibilities of tapping into new kinds of data, have actually lost none of their appeal or real-world power, particularly as compliance, risk management, regulatory reporting and operational efficiency remain hot topics.

To compare data warehouses and big data, it is useful to first establish definitions for each: Big data refers to managing vast quantities – 100 terabytes or more – of unstructured data, generated from machines, social media and the like; data warehouses are designed to handle structured data, pulled from trading, customer management, portfolio accounting and other operational systems. Further, the vast majority of financial firms are not dealing with anywhere close to 100 terabytes of actual data. In fact, few organizations approach the outer limits of data warehouse capabilities – most involve no more than 15 terabytes, and those at that level could likely greatly reduce it with an improved data model.

Beyond size, big data falls short for financial firms because two of the primary drivers for technology investment are compliance and risk management, both of which require limited volumes of structured data from operational systems, and therefore are best served by data warehouses. In addition, because all data in a data warehouse is traceable, compliance departments at any moment can quickly and easily determine where and when information was derived, for an accurate audit trail, something big data is not designed to handle.

[For more on It's More About "Wide Data" Than Big Data]

When one considers these differences in evaluating the unique capabilities of each data management approach, it becomes clear that news of data warehouses’ demise is premature. In fact, there is reason to believe the future for data warehouse is brighter than ever.

We speak with many CIOs and Lead Architects and some are looking into big data technology because they feel their data warehouses are becoming too big. On closer inspection, we find that bloated data warehouses can be easily fixed by applying financial domain-specific knowledge to how the data is organized.

Even without applying domain-specific knowledge, it is my firm belief that that the real data problem for the financial industry is not vast volumes of data; it is the complexity of the data. Sure, there are times when financial firms need to store huge volumes of data – historical tick data for example. But for the vast majority of financial organizations, the real trick is to figure out how to get multiple operational systems to work with each other without losing or duplicating data. Introducing the hot new technology may be appealing, but solving the complexity puzzle effectively is better served by the data warehouse.

The story of big data is still largely one of potential (though incredibly exciting potential, most would agree) and no bank should see it as a replacement for current data warehouse initiatives that have proven their value time and time again.

Instead, financial technology professionals should continually seek out any and all solutions that could make their firms more profitable, productive, secure and compliant. This means not saying “this or that,” but instead “this and that.”

Jonas Olsson is the founder and CEO of Graz, a data warehouse provider for the financial industry.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
9/13/2013 | 12:23:00 PM
re: Big Data: No Replacement for the Financial Data Warehouse
It depends what they are doing with the data and what kind of data whether this makes. Certainly, you can run a DW in the cloud and gain exceptional savings, too. Think Redshift. The notion of a DW isn't obsolete unless you want to abandon the notion of consistent, aligned data, arranged dimensionally, so it's easy to access and query for both standard dashboards and ad hoc queries.
Greg MacSweeney
Greg MacSweeney,
User Rank: Apprentice
8/7/2013 | 4:08:15 PM
re: Big Data: No Replacement for the Financial Data Warehouse
It's nice to see a provider being so open and transparent with pricing, which is increasingly rare, nowadays.
User Rank: Apprentice
8/7/2013 | 1:45:50 PM
re: Big Data: No Replacement for the Financial Data Warehouse
There isn't a single piece of supporting evidence in this defense. FinQloud maintains our R3 offering on an EMR cluster (AWS version of big data) and here's our evidence that it works just fine: Signed three of top ten banks in first six months. Have not quoted a single firm where it wasn't at least half the cost of their internal solution. Here's the pricing sheet if you want to do your own math.
More Commentary
5 Tips On How To Prepare For A Data Breach
If you are a financial institution your cyber security defenses will be breached -- again and again. Here are five tips to respond quickly and minimize damage.
Wall Street CIOs Have a Vendor Management Problem
If Wall Street CIOs want to stay ahead of competition and ensure high-speed trading software doesn't start the next flash crash, they need better insight into vendor delivered software.
Technology Innovation Returns to Financial Services
Capital Markets Outlook 2015: Following a few years dominated by regulatory compliance and cost saving technology initiatives, financial organizations are finally investing in innovative technology and tools.
Voice Biometrics Improve Transaction Monitoring Fraud Detection
Why voice biometrics should be a part of your fraud prevention strategy in the call center.
Fintech Fast Forward 2015
What will shape the future of Fintech in 2015 and beyond?
Register for Wall Street & Technology Newsletters
White Papers
Current Issue
Wall Street & Technology - Elite 8, October 2014
The in-depth profiles of this year's Elite 8 honorees focus on leadership, talent recruitment, big data, analytics, mobile, and more.
5 Things to Look For Before Accepting Terms & Conditions
5 Things to Look For Before Accepting Terms & Conditions
Is your corporate data at risk? Before uploading sensitive information to cloud services be sure to review these terms.