October 09, 2012

Market participants are rallying to come to grips with three technology glitches that nearly destabilized the markets. I'm talking about a software error that forced BATS to pull its initial public offering in March, technical problems at Nasdaq that messed up Facebook's IPO in May, and Knight Trading's $440 million loss tied to errant software code in August. Altogether, these incidents have raised doubts about how market participants design, test and implement new automated trading systems before they go into production.

At a roundtable put on by the Securities and Ex­change Commission, market players, academics and technologists discussed ways to prevent these sorts of electronic trading errors from ricocheting into the broader market. They also talked about best practices for developing and testing large scale deploy­ments. Chris Isaacson, BATS chief operating officer, called for "better policies and procedures to ensure that adequate time is devoted to testing by the market participants in different environments."

But given the competitive rush to bring out new functionality, firms do take short cuts in releasing new capabilities. Lou Pastina, executive VP of the New York Stock Exchange, said that many of the entrepreneurial designs for systems begin as a sketch on the back of an envelope, but how that idea is filtered, eventually refined into a design, tested and deployed into production varies among firms. Somewhat alarming was the revelation that when the NYSE does industry testing for new systems, the same firms always show up, implying that other firms don't bother to come.

While testing was emphasized, experts at the meeting cautioned that it isn't a panacea. "All software contains errors," said Nancy Leve­son, a professor of aeronautics, astronautics and en­gin­eering systems at MIT. NASA spent $100 million per year to maintain safety-related software and still had errors in the Space Shuttle Program, Leveson said.

Testing isn't fault proof because it's impossible to test for all the variables in a complex trading system. According to Chad Cook, CTO at Lime Brokerage LLC, in an interview, a trading system has many pieces of technology "under the hood," including the Windows operating system, Intel processor, code libraries and connections to networks using someone else's switches and routers.