Trading Technology

10:32 PM

HFT and Big Data: Not a Technology Barrier, but a Cultural One

Here are three fundamental cultural differences that are going to be harder to resolve than integration of technologies as part of a successful integration of “big data” into HFT.

There is a growing consensus that the limits of value generation from the low-latency segment of high-frequency trading (HFT) are being reached and that the next wave of algorithmic trading will be based on the combination of real-time processing with unstructured “big data” sources.

While the “race to zero” in the low-latency segment of HFT is not entirely over – witness the recent investments in low-latency microwave networks by McKay Brothers and others – profitability has diminished and the market is consolidating for both HFT firms and their vendors.

Right now, everyone is struggling to position for the next wave by examining the technical architecture differences between the solutions for “big data” (e.g. Hadoop) and those for HFT (e.g. in-memory processing).

[Hadoop 2.0: The Capital Markets Dragon Slayer? ]

In my view, integration of technologies is unlikely to be the greatest obstacle to the successful integration of “big data” into HFT. There are three fundamental cultural differences that are going to be harder to resolve:

1. Data quality. Unstructured data is messy, inconsistent and incomplete. It is best used for situations in which the consequences of making mistakes are limited and there is ample opportunity to integrate the learnings from those mistakes back into the algorithms. I recently attended a presentation by a data scientist from Spotify, the music delivery application. Their recommendation algorithm is kept as simple as possible, given the massive size of the data set, but they are constantly adjusting it are and are more concerned with getting it better than getting it right.

2. Data timeliness. The big data ecosystem revolves around Hadoop, which is inherently a batch processing construct. While this is changing in Hadoop 2.0, the reality is still far from an online, real-time view of the world. (Spotify updates their music recommendations daily based on overnight batch processing. They plan to improve this soon – but to hours or minutes, not microseconds.)

3. Data architecture. Although it is a difficult challenge which is often honored in the breech, capital markets players understand and respect the value of data architecture. The concepts of master data management, data integrity, and golden sources are well-established. The absence of pre-defined schema in non-relational databases often associated with “big data” is just one indication that the “big data” world is not cultural ready for the merger.

Not listed above are the more generalized fears and uncertainties that accompany any innovation wave, especially one that threatens to challenge the status quo. Examples of these are also evident in recent postings in this publication and elsewhere. My advice is to face the cultural challenges head-on by recognizing them and especially by understanding how they play out in your own firm.

[Check out ways companies are Navigating the Big Spectrum of Big Data’s Solutions at Interop, which runs from September 30 through October 4 in NYC.]

Jennifer L. Costley, Ph.D. is a scientifically-trained technologist with broad multidisciplinary experience in enterprise architecture, software development, line management and infrastructure operations, primarily (although not exclusively) in capital markets. She is also a ... View Full Bio
More Commentary
A Wild Ride Comes to an End
Covering the financial services technology space for the past 15 years has been a thrilling ride with many ups as downs.
The End of an Era: Farewell to an Icon
After more than two decades of writing for Wall Street & Technology, I am leaving the media brand. It's time to reflect on our mutual history and the road ahead.
Beyond Bitcoin: Why Counterparty Has Won Support From Overstock's Chairman
The combined excitement over the currency and the Blockchain has kept the market capitalization above $4 billion for more than a year. This has attracted both imitators and innovators.
Asset Managers Set Sights on Defragmenting Back-Office Data
Defragmenting back-office data and technology will be a top focus for asset managers in 2015.
4 Mobile Security Predictions for 2015
As we look ahead, mobility is the perfect breeding ground for attacks in 2015.
Register for Wall Street & Technology Newsletters
Exclusive: Inside the GETCO Execution Services Trading Floor
Exclusive: Inside the GETCO Execution Services Trading Floor
Advanced Trading takes you on an exclusive tour of the New York trading floor of GETCO Execution Services, the solutions arm of GETCO.