Data Management

11:04 AM
Jonas Olsson, Graz
Jonas Olsson, Graz
Commentary
50%
50%

Big Data: No Replacement for the Financial Data Warehouse

News of data warehouses’ demise is premature, argues Jonas Olsson. In fact, there is reason to believe the future for data warehouse is brighter than ever.

Some people are suggesting that with the explosion of “Big Data,” data warehouses are becoming a thing of the past, that they are the “new mainframes.” The perception is understandable, as big data is certainly sucking the oxygen out of the server room. But for anyone taking data seriously, the old maxim “perception is reality” is not acceptable, especially if you are working in the financial industry.

Jonas Olsson, Graz
Jonas Olsson, Graz

So, is there any fact-based validity to the idea that data warehouses are becoming the new legacy system, holding financial services companies back from new insights and greater efficiencies? I argue the notion is absurd, like saying there should be no rail transportation because someone invented the automobile. A closer investigation shows that data warehouses, overshadowed by excitement about the new possibilities of tapping into new kinds of data, have actually lost none of their appeal or real-world power, particularly as compliance, risk management, regulatory reporting and operational efficiency remain hot topics.

To compare data warehouses and big data, it is useful to first establish definitions for each: Big data refers to managing vast quantities – 100 terabytes or more – of unstructured data, generated from machines, social media and the like; data warehouses are designed to handle structured data, pulled from trading, customer management, portfolio accounting and other operational systems. Further, the vast majority of financial firms are not dealing with anywhere close to 100 terabytes of actual data. In fact, few organizations approach the outer limits of data warehouse capabilities – most involve no more than 15 terabytes, and those at that level could likely greatly reduce it with an improved data model.

Beyond size, big data falls short for financial firms because two of the primary drivers for technology investment are compliance and risk management, both of which require limited volumes of structured data from operational systems, and therefore are best served by data warehouses. In addition, because all data in a data warehouse is traceable, compliance departments at any moment can quickly and easily determine where and when information was derived, for an accurate audit trail, something big data is not designed to handle.

[For more on It's More About "Wide Data" Than Big Data]

When one considers these differences in evaluating the unique capabilities of each data management approach, it becomes clear that news of data warehouses’ demise is premature. In fact, there is reason to believe the future for data warehouse is brighter than ever.

We speak with many CIOs and Lead Architects and some are looking into big data technology because they feel their data warehouses are becoming too big. On closer inspection, we find that bloated data warehouses can be easily fixed by applying financial domain-specific knowledge to how the data is organized.

Even without applying domain-specific knowledge, it is my firm belief that that the real data problem for the financial industry is not vast volumes of data; it is the complexity of the data. Sure, there are times when financial firms need to store huge volumes of data – historical tick data for example. But for the vast majority of financial organizations, the real trick is to figure out how to get multiple operational systems to work with each other without losing or duplicating data. Introducing the hot new technology may be appealing, but solving the complexity puzzle effectively is better served by the data warehouse.

The story of big data is still largely one of potential (though incredibly exciting potential, most would agree) and no bank should see it as a replacement for current data warehouse initiatives that have proven their value time and time again.

Instead, financial technology professionals should continually seek out any and all solutions that could make their firms more profitable, productive, secure and compliant. This means not saying “this or that,” but instead “this and that.”

Jonas Olsson is the founder and CEO of Graz, a data warehouse provider for the financial industry.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
wecker
50%
50%
wecker,
User Rank: Apprentice
9/13/2013 | 12:23:00 PM
re: Big Data: No Replacement for the Financial Data Warehouse
It depends what they are doing with the data and what kind of data whether this makes. Certainly, you can run a DW in the cloud and gain exceptional savings, too. Think Redshift. The notion of a DW isn't obsolete unless you want to abandon the notion of consistent, aligned data, arranged dimensionally, so it's easy to access and query for both standard dashboards and ad hoc queries.
Greg MacSweeney
50%
50%
Greg MacSweeney,
User Rank: Apprentice
8/7/2013 | 4:08:15 PM
re: Big Data: No Replacement for the Financial Data Warehouse
It's nice to see a provider being so open and transparent with pricing, which is increasingly rare, nowadays.
ahonore
50%
50%
ahonore,
User Rank: Apprentice
8/7/2013 | 1:45:50 PM
re: Big Data: No Replacement for the Financial Data Warehouse
There isn't a single piece of supporting evidence in this defense. FinQloud maintains our R3 offering on an EMR cluster (AWS version of big data) and here's our evidence that it works just fine: Signed three of top ten banks in first six months. Have not quoted a single firm where it wasn't at least half the cost of their internal solution. Here's the pricing sheet if you want to do your own math. http://www.nasdaqomx.com/digit...
More Commentary
Data Integrity: A Necessity, Not an Option
Financial institutions that have taken on the data integrity task in the past now have to spend more money on hardware, software, and people just to keep up with the demand.
What Colombia’s New IT Campaign Means for Latin American Tech Investment
Colombia’s campaign is the latest example of how Latin America is trying to edge into the global technology space.
Initial Margin: When Does More Turn Out to Be Less?
Changing margin regulations are set to affect the OTC derivative market, including initial margin risk models for non-cleared OTCs.
The Mainframe Innovation Drag
It may be time for a consortium of firms motivated around the objective of eliminating the mainframe. What if every self-clearing firm decided to participate in building a modern, back-office system as an open-source, cloud-based project?
Big Data DIY
Now that we have passed the initial hype phase of big data, companies are searching for real business value from their investments. Consultants can play a part, but only if financial firms insist on a new partnership model.
Register for Wall Street & Technology Newsletters
White Papers
Current Issue
Wall Street & Technology - July 2014
In addition to regular audits, the SEC will start to scrutinize the cyber-security preparedness of market participants.
Video
5 Things to Look For Before Accepting Terms & Conditions
5 Things to Look For Before Accepting Terms & Conditions
Is your corporate data at risk? Before uploading sensitive information to cloud services be sure to review these terms.