Nasdaq OMX Cites Flood of Messages in SIP Failure

In a summary of findings, Nasdaq OMX said it received more than 20 connect and disconnect sequences from NYSE Arca, which consumed capacity on its SIP that triggered more problems.

Nasdaq OMX released a summary today that provides details about the failure of its securities information processor (SIP) and decision to halt trading last week. In a summary of preliminary findings based on an internal review, the US stock market operator cited a deluge of messages it received from rival electronic exchange NYSE Arca, which eroded the capacity of the SIP and led to other technical issues.

“The catalyst for the SIP failure was a confluence of unprecedented events that that overwhelmed the processing capacity of the SIP,” stated Nasdaq OMX in a summary of its preliminary findings from an internal review.

However, Nasdaq also said that high frequency trading played no role in the technology events of Aug. 22.

On Aug. 22, Nasdaq’s SIP, a system that consolidates quotes in Nasdaq’s stocks from all the exchanges, “received more than 20 connect and disconnect sequences from NYSE Arca, each of which consumed significant resources,” stated Nasdaq OMX’s summary released earlier today. In addition, the SIP received a stream of quotes for inaccurate symbols from NYSE Arca, and generated quote rejects, the summary stated. Nasdaq maintains that both of these events combined to “degrade the system below the tested capacity of 10,000 messages per per-port, per second. According to Nasdaq OMX, during this period, NYSE Arca sent multiple bursts (of messages) with each connect and discount, exceeding more than 26 000 quote updates per port, per second.

Normally, messages for NYSE Arca on a typical day in August would peak at 1,000 messages per-port, per second, according to Nasdaq OMX’s report. On Aug. 22, the message volume was 26 times greater than the average per-port per second activity, noted the report.

According to Nasdaq’s report, the confluence of these events exceeded the SIP’s planned capacity, which caused its failure and then revealed a latent flaw in the SIP’s software code. Market participants have questioned why Nasdaq’s SIP could not be rolled over to a back-up system.

The latent software flaw prevented the SIP’s built-in redundancy capability from failing over cleanly, according to Nasdaq OMX in its report. All of these events impaired the SIP’s ability to process quotes to an extent that Nasdaq OMX felt it was necessary to shutdown the system to ensure fair and orderly markets. Last Friday, Nasdaq OMX’s CEO Robert Greifeld said on interviews with CNBC and Bloomberg that without the halt, there would have been” an information asymmetry” between the SIP and direct feeds consumed by high speed trading firms.

The three-hour shutdown starting after 12:00 pm frustrated investors and worried traders, who have experienced an increase in technology disruptions from electronic stock exchanges. Although Nasdaq said it had fixed the problem within 30 minutes, the market did not re-open until two and one half hours later at nearly 3:30 pm.

While the problem was "quickly identified and data feeds were operational within 30 minutes of the halt," Nasdaq said extra time was needed to consult with UTP (Unlisted Trading Privileges) SIP committee members, and market participants and to test and evaluate scenarios to re-open the market in a fair and orderly manner. Nasdaq OMX points out that the market was reopened without incident.

Ultimately, Nasdaq OMX took responsibility for the failure of its SIP, calling it “unacceptable to our members, issuers and the investing public.” It’s currently identifying potential design changes to fortify the SIP’s resiliency, including architectural improvements, information security, disaster recovery plans and capacity parameters. It plans to present initial recommendations to the SIP governing committee within 30 days.

Faulted by many market participants for its lack of communication during the halt, Nasdaq OMX said it’s undertaking a comprehensive review of the policies and procedures for communicating with customers, market participants and the broader public during market-wide events. Meanwhile, a conference call is scheduled for next week with exchanges and brokers that are on SIP committee.

Ivy is Editor-at-Large for Advanced Trading and Wall Street & Technology. Ivy is responsible for writing in-depth feature articles, daily blogs and news articles with a focus on automated trading in the capital markets. As an industry expert, Ivy has reported on a myriad ... View Full Bio