Data Management

11:10 AM
Ivy Schmerken
Ivy Schmerken
Connect Directly

Big Data: The Next New Thing?

Venture capitalists are investing in "Big Data" companies with analytic platforms, but hasn't Wall Street been doing this for 20 years?

Big data is a hot area in venture capital circles and on the conference circuit. With data spewing out of social networking sites like Twitter and Facebook and corporate web sites VCs have been investing in analytic platforms that can analyze customer behavior, optimize digital marketing and help companies with fraud detection.

I attended a lunch panel last week as part of the O’Reilly Media Strata Summit in New York about where Wall Street and technology meet. Most of the conference was about how do you apply technology to filter and curate massive amounts of data. Isn’t this what Wall Street has been doing forever? Why is it new?

Though Wall Street firms have been focusing on data management for about 20 plus years – scrubbing data, normalizing data and setting up data warehouses to create golden copies to feed risk management systems and back-office settlement systems— this is old stuff. So what is big data?

I asked a few people attending the lunch this question and was told, “Yes, Wall Street has been doing this but now the rest of the world is catching up.”

According to Terradata, which acquired Aster Data Systems, one of the data analytics companies sponsoring the event: “Big data is a term commonly applied to large data sets where volume, variety, velocity, or complexity are beyond the ability of commonly used software tools to efficiently capture, manage, and process. Examples include Web logs, sensor networks, social media, medical records and more.”

In high frequency trading, firms can look for unusual patterns in complex data sets across asset classes and geographies. Now that the cost of servers with multi-core processors has fallen, many businesses can afford to crunch large data stores using analytics. But has the Internet democratized access to information and leveled the playing field for investors against the most savvy hedge funds and prop trading firms?

The answer is Yes, if you can afford the tools and infrastructure required to process what’s known as Big Data. One panelist was skeptical of this outcome.

Roger Ehrenberg, a managing partner and founder of IAVentures, who spoke on the panel, suggested the Internet has actually raised the barriers to entry. In fact, the cost of managing the gross amounts of data goes up, asserts Ehrenberg, who points out that it’s the synthesis of data using the tools that has become difficult. According to Ehrenberg, quant hedge funds like Renaissance Technologies, Bridgewater, 2 Sigma and AQR that have unlimited budgets for data technology and dedicated teams to clean, index, store and create the data stores, he said. “As democratization of information has exploded, the cost of extracting an information signal from that mass of data has exploded, (as well),” said Ehrenberg. It’s the synthesis of the data that has gotten so difficult,” said Ehrenberg.

That is why the topic of Big Data is on the radar screen. Also, the costs of commodity servers have come down and speed has gone up. Whether the science behind big data is new or not, the technologies of data mining, business intelligence, standard query language (SQL), and data storage, are being packaged up for consumption by all businesses and industries. And that is democratization. With new laws like Dodd-Frank Wall Street Reform and Consumer Protection Act requiring regulators to conduct surveillance on derivatives transactions, the tools of big data are going to be in demand.

Ivy is Editor-at-Large for Advanced Trading and Wall Street & Technology. Ivy is responsible for writing in-depth feature articles, daily blogs and news articles with a focus on automated trading in the capital markets. As an industry expert, Ivy has reported on a myriad ... View Full Bio
More Commentary
A Wild Ride Comes to an End
Covering the financial services technology space for the past 15 years has been a thrilling ride with many ups as downs.
The End of an Era: Farewell to an Icon
After more than two decades of writing for Wall Street & Technology, I am leaving the media brand. It's time to reflect on our mutual history and the road ahead.
Beyond Bitcoin: Why Counterparty Has Won Support From Overstock's Chairman
The combined excitement over the currency and the Blockchain has kept the market capitalization above $4 billion for more than a year. This has attracted both imitators and innovators.
Asset Managers Set Sights on Defragmenting Back-Office Data
Defragmenting back-office data and technology will be a top focus for asset managers in 2015.
4 Mobile Security Predictions for 2015
As we look ahead, mobility is the perfect breeding ground for attacks in 2015.
Register for Wall Street & Technology Newsletters
5 Things to Look For Before Accepting Terms & Conditions
5 Things to Look For Before Accepting Terms & Conditions
Is your corporate data at risk? Before uploading sensitive information to cloud services be sure to review these terms.