November 15, 2012

The financial market is a fickle beast. A bit of good news can send it on a tear, with the bulls running wild. And sometimes a seemingly innocuous news report can spook the market, resulting in a sudden swoon.

Underlying the markets, however, is a technological architecture that's getting more fragile by the day. Now that the equity markets are almost completely electronic, hiccups, snags or technological glitches can result in major losses (or gains) for traders. Knight Capital's trading glitch on Aug. 1, which almost collapsed the firm, was caused by a software error that sent ripples through the marketplace.

Smaller tremors are felt every day in the market because of electronic trading errors, system failures, human errors, rogue algorithms and predatory algos. There are plenty of theories as to why these errors are becoming more frequent, but no single culprit has been found. But this is really a larger and more complex topic for another day.

Most recently, Hurricane Sandy reminded us of how important technology is to the modern market, and how difficult it is to prepare for the unexpected. Despite disaster recovery and business continuity plans that were implemented after Sept. 11 and then enhanced following the Northeast blackout in 2003, Sandy has upped the DR/BC bar once again.

Following 9/11, financial firms moved their backup facilities to New Jersey and other states. When the 2003 blackout struck, many firms made sure their backup locations were on different electrical grids than their primary facilities.

[Exclusive Photos of Knight Capital's Trading Floor]

The aftermath of Sandy, however, is altogether different. With lower Manhattan in the dark for most of the week following the storm (and parts still underwater), firms had to shift to backup facilities, many of which are located in northern and central New Jersey. Unfortunately, much of New Jersey was still without power the week after the storm, forcing backup facilities to run on generators.

Although large data center generators are designed to run for long periods of time, sometimes they fail. Knight Capital told its customers to send their orders to other brokers two days after the storm because its generators failed. The following day, Knight said its systems were back online and running normally. Meanwhile, Nasdaq OMX had its primary data center in Carteret, N.J., running on backup generators. Nasdaq has a backup facility in Virginia that can run operations in case its generators fail or run out of fuel.

Luckily, the financial markets were mostly closed during the first two days of the storm, which gave firms time to work through technological and communication glitches that resulted from the historic storm. All in all, the markets made it through Sandy without many outages. If history is any guide, executives are already enhancing BC/DR plans based on the lessons learned from this disruptive event.

Greg MacSweeney is editorial director of InformationWeek Financial Services, whose brands include Wall Street & Technology, Bank Systems & Technology, Advanced Trading, and Insurance & Technology.