The Securities and Exchange Commission is now collecting 400 gigabytes of market data daily, including information on every trade across numerous exchanges, as part of what SEC chairman Elisse Walter on Tuesday called an "unprecedented" push to use data to better understand how the market works.

The big data effort, which centers around a system known as the Market Information Data Analytics System (Midas), is being used not only to understand market trends and rapidly emerging modes of trading such as high-frequency trading, but also, according to the SEC, to inform future policy making.

Midas, which is costing the SEC $2.5 million a year, captures data such as time, price, trade type and order number on every order posted on national stock exchanges, every cancellation and modification, and every trade execution, including some off-exchange trades. Combined it adds up to billions of daily records.

Although the system has been live for only a few months, the data in some cases goes back as far as four or five years. The uncompressed archive of Midas data would amount to about 1 petabyte, although the archive has been compressed to just more than 100 terabytes.

[For some reaction to the SEC's plans for Midas, read: Some Questions for the SEC on the New MIDAS Surveillance Plans.]

According to Walter and the top official overseeing Midas, Gregg Berman, who was recently named associate director of the Office of Analytics and Research in the SEC's Division of Trading and Markets, Midas has potentially wide application for the SEC.

"The downpour of data generated by the markets every hour will lead to better regulation and better investor protection," Walter said in a speech Tuesday, adding that Midas will "dramatically improve our understanding of the way today's markets function."

... Read full story on InformationWeek


Post a comment to the original version of this story on InformationWeek