Data Management

11:04 AM
Jonas Olsson, Graz
Jonas Olsson, Graz
Commentary
50%
50%

Big Data: No Replacement for the Financial Data Warehouse

News of data warehouses’ demise is premature, argues Jonas Olsson. In fact, there is reason to believe the future for data warehouse is brighter than ever.

Some people are suggesting that with the explosion of “Big Data,” data warehouses are becoming a thing of the past, that they are the “new mainframes.” The perception is understandable, as big data is certainly sucking the oxygen out of the server room. But for anyone taking data seriously, the old maxim “perception is reality” is not acceptable, especially if you are working in the financial industry.

Jonas Olsson, Graz
Jonas Olsson, Graz

So, is there any fact-based validity to the idea that data warehouses are becoming the new legacy system, holding financial services companies back from new insights and greater efficiencies? I argue the notion is absurd, like saying there should be no rail transportation because someone invented the automobile. A closer investigation shows that data warehouses, overshadowed by excitement about the new possibilities of tapping into new kinds of data, have actually lost none of their appeal or real-world power, particularly as compliance, risk management, regulatory reporting and operational efficiency remain hot topics.

To compare data warehouses and big data, it is useful to first establish definitions for each: Big data refers to managing vast quantities – 100 terabytes or more – of unstructured data, generated from machines, social media and the like; data warehouses are designed to handle structured data, pulled from trading, customer management, portfolio accounting and other operational systems. Further, the vast majority of financial firms are not dealing with anywhere close to 100 terabytes of actual data. In fact, few organizations approach the outer limits of data warehouse capabilities – most involve no more than 15 terabytes, and those at that level could likely greatly reduce it with an improved data model.

Beyond size, big data falls short for financial firms because two of the primary drivers for technology investment are compliance and risk management, both of which require limited volumes of structured data from operational systems, and therefore are best served by data warehouses. In addition, because all data in a data warehouse is traceable, compliance departments at any moment can quickly and easily determine where and when information was derived, for an accurate audit trail, something big data is not designed to handle.

[For more on It's More About "Wide Data" Than Big Data]

When one considers these differences in evaluating the unique capabilities of each data management approach, it becomes clear that news of data warehouses’ demise is premature. In fact, there is reason to believe the future for data warehouse is brighter than ever.

We speak with many CIOs and Lead Architects and some are looking into big data technology because they feel their data warehouses are becoming too big. On closer inspection, we find that bloated data warehouses can be easily fixed by applying financial domain-specific knowledge to how the data is organized.

Even without applying domain-specific knowledge, it is my firm belief that that the real data problem for the financial industry is not vast volumes of data; it is the complexity of the data. Sure, there are times when financial firms need to store huge volumes of data – historical tick data for example. But for the vast majority of financial organizations, the real trick is to figure out how to get multiple operational systems to work with each other without losing or duplicating data. Introducing the hot new technology may be appealing, but solving the complexity puzzle effectively is better served by the data warehouse.

The story of big data is still largely one of potential (though incredibly exciting potential, most would agree) and no bank should see it as a replacement for current data warehouse initiatives that have proven their value time and time again.

Instead, financial technology professionals should continually seek out any and all solutions that could make their firms more profitable, productive, secure and compliant. This means not saying “this or that,” but instead “this and that.”

Jonas Olsson is the founder and CEO of Graz, a data warehouse provider for the financial industry.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
wecker
50%
50%
wecker,
User Rank: Apprentice
9/13/2013 | 12:23:00 PM
re: Big Data: No Replacement for the Financial Data Warehouse
It depends what they are doing with the data and what kind of data whether this makes. Certainly, you can run a DW in the cloud and gain exceptional savings, too. Think Redshift. The notion of a DW isn't obsolete unless you want to abandon the notion of consistent, aligned data, arranged dimensionally, so it's easy to access and query for both standard dashboards and ad hoc queries.
Greg MacSweeney
50%
50%
Greg MacSweeney,
User Rank: Apprentice
8/7/2013 | 4:08:15 PM
re: Big Data: No Replacement for the Financial Data Warehouse
It's nice to see a provider being so open and transparent with pricing, which is increasingly rare, nowadays.
ahonore
50%
50%
ahonore,
User Rank: Apprentice
8/7/2013 | 1:45:50 PM
re: Big Data: No Replacement for the Financial Data Warehouse
There isn't a single piece of supporting evidence in this defense. FinQloud maintains our R3 offering on an EMR cluster (AWS version of big data) and here's our evidence that it works just fine: Signed three of top ten banks in first six months. Have not quoted a single firm where it wasn't at least half the cost of their internal solution. Here's the pricing sheet if you want to do your own math. http://www.nasdaqomx.com/digit...
More Commentary
Is Your Corporate Data Being Auctioned on eBay?
Researchers purchased 20 used Android phones to see what data they could retrieve using off-the-shelf recovery software. The results were astonishing.
Getting Social: Top Tips for Establishing a Social Media Plan
As the influence of social media channels continues to grow, organizations must have a handle on the regulations and the risks social can introduce to a firm.
Why Settle for Less in the Front Office?
Recent research shows that sell-side firms are less than satisfied with their order management system (OMS) technology. Many front offices, however, continue to make do with their current solutions. Are they selling themselves short?
BYOD Policy: Don't Reinvent the Wheel
Financial firms still feel overwhelmed by BYOD risks and challenges. But these can be addressed by a good policy, and the guidelines are already out there.
The BYOD Challenge
Having a policy in place to manage mobile devices used by employees for work purposes is necessary in this current day.
Register for Wall Street & Technology Newsletters
White Papers
Current Issue
Wall Street & Technology - July 2014
In addition to regular audits, the SEC will start to scrutinize the cyber-security preparedness of market participants.
Video
5 Things to Look For Before Accepting Terms & Conditions
5 Things to Look For Before Accepting Terms & Conditions
Is your corporate data at risk? Before uploading sensitive information to cloud services be sure to review these terms.