Column-Based Databases Making A Wall Street Comeback

The last time I met with Sybase, it was back when the big three database vendors were Informix, Oracle and Sybase - remember the 90s? This week, Gavin Quinn, FSI business development manager at Sybase, says the company is seeing a resurgence of interest in its high-speed, column-based databases among hedge funds and financial institutions.The reason for this, Quinn believes, is a combination of the exponential growth rate of market data and the increased storage demands of Mifid. "If you're not storing your data in an environment that's friendly to market data and time series data, it can be brutally expensive," he says. And traditional methods of archiving older data, such as putting it on tape, are much too slow to meet regulators' requests. Sybase's large, column-based databases are designed to keep massive amounts of data (think petabytes) online indefinitely because they make efficient use of storage space. What's a column-based database? It's a database management system that stores content by column rather than by row. Each quote or order, for instance, gets parsed out into its separate pieces (hour, minute, second, day, month, year, price, CUSIP number, etc.) and stored in columns. As each new price update comes in, a traditional row-based database would repeat all the data elements every time. A column-based database only stores the new portions of the information (such as price, minute, second). Query speed is also enhanced in the column-based database because there's less data for the query engine to have to sift through.

If a set of market data typically requires 100 gigabytes of storage, Sybase's databases will store it in 20, Quinn says, whereas in a row-based environment it might consume 400-1,200 gigabytes.

Quinn says Sybase's fastest column-based database, Sybase IQ, offers subsecond data storage and retrieval rates (due to an in-memory database) and therefore can accommodate real-time analytics. It uses standard SQL language. BNP Paribas uses Sybase IQ to turn aggregated customer and product data into portfolio performance reports. The system is delivering approximately 200,000 reports a month to 10,000 external customers and internal employees. BNP Paribas has reported that because Sybase IQ compresses data by 50 percent, the firm will now be able to keep up to 10 years of data online for trend reporting and advanced analytics.