Exchanges Hardest Hit by Trade Data Avalanche

Although the buy side and sell side have to store trade data, too, exchanges such as ISE are hardest hit by Reg NMS/MiFID trade data volumes.

The SEC's Regulation NMS and Europe's MiFID, which require firms to retain records to prove best execution, unleashed what many have called a data storage and retrieval nightmare. Archived records need to include not only the details of a trade, but other quotes available at the time as well. And this information must be provided within a day or two of an auditor's request or a customer's complaint. As a result, exchanges, brokerages and buy-side firms are making sure they have the infrastructures not just to store millions of pieces of trade data every day, but also to retrieve specific data points easily and quickly.

The exchanges are hardest hit by these demands. Tom Price, senior analyst at TowerGroup, points out that if a customer complains or if something goes wrong with a trade, a passing of the buck can occur. "The buy side will just point to the broker-dealer and say, 'What did you do?' The onus of proof is on them," he says. "The brokers will point to the exchanges and say, 'What did you do? You were supposed to route that and get me best execution.'"

Case in point is the New York-based International Securities Exchange, which trades options and stock. Driven in part by industry changes such as Reg NMS and penny quoting, the ISE stores 800 gigabytes of trade information a day, according to Rob Cornish, the exchange's technology infrastructure and development officer.

Years ago, ISE's IT management realized it would be impossible to store all this data in a regular relational database, so the exchange created a proprietary flat-file data scheme, indexed simply by symbol, date and time, Cornish relates. Trades are stored in a high-performance storage system from EMC for three months, then moved to a second-tier RAID system from HP for six months, then migrated to tape libraries (also from HP); duplicates are sent off-site for long-term storage, he explains.

To handle regulatory issues and customer questions, ISE leverages its data scheme to instantly provide operations and regulatory staff with a snapshot of the enitire market and recreate a full view of what was happening as a particular trade occurred. To lessen the storage burden, the exchange uses the FIX FAST compression algorithm. "Normally compression makes the data unusable; you can't search it until it's decompressed, which takes too long," says Greg Maynard, ISE's system and product strategy officer. "But FAST lets you decompress instantaneously, which allows it to be searched."