John Wellington, director of wealth-management technology and advisory services at Advest, Inc., a regional financial-services firm, was working at his desk in Hartford, Conn. when the outage struck Thursday afternoon.
A message flashed across his computer screen, notifying him that there would be a slight interruption in service while the retail financial-services firm switched to backup power. Other than that, Wellington was able to continue working. "This building was unaffected, from my perspective," says Wellington.
But that was not the case when it came to the firm's trading operations in New York.
On Friday, it had some trading desks that weren't operating at full capacity, and "a couple of branches were knocked out in Long Island," says Wellington. "The customers couldn't transact with their brokers on Long Island. They couldn't reach them. Nothing was working."
Advest's New York City-based Lebenthal division, the municipal-bond broker acquired in 2001, "was out of it for the day," says Wellington.
Even though Friday was a light trading day, Advest wanted to make contact with its retail investors, who equate wealth management with personal service. The firm took the 800 phone number of the affected branch and rerouted the number to sister branches so that customers could call in. "You call the 800 number and you end up in a Connecticut branch," he says.
While the calls were being rerouted to other branches, brokers who could not report to work in those branches had to get access to their clients' portfolios.
A number of brokers had remote access, says Wellington. Those employees had a laptop and could dial into the corporate IT center, where they hooked straight into the data center.
But not all of the brokers use laptops. Some use desktops, says Wellington. "Those guys were somewhat out of luck," he says.
Outage Knocks Out Small BrokerBy Ivy Schmerken
Though large brokers often have the money to spend on backup facilities, small firms can't always afford the investment in disaster-recovery sites.
"Smaller firms (that) haven't invested the big bucks in disaster recovery sites were in some ways out of the market on Friday," says the CEO of a direct-access broker based in lower Manhattan, who requested anonymity.
Commenting on the impact of the blackout, the CEO says the firm was out of the market on Friday because it didn't have sufficient backup trade-processing systems and generators.
Though the firm did have a generator, "It wasn't robust," he says. If the power failure had lasted one, two or three hours, the generator would have lasted, but "This was so substantial, this knocked out everything," he says.
The firms' customers -- professional trading firms -- that rely on it to provide smart order routing to electronic-communications networks -- could not trade either. "Our customers were out because we were out," says the CEO. Customers also experienced connectivity problems in the form of communication lines and cell phones that didn't work.
Fortunately, the Northeast blackout occurred after 4:00 p.m., so customers that had open positions were able to unwind their risk.
"Most people had already flattened out," he says, referring to customers that had open positions. "You have people who take intra-day positions and have to lessen their overnight risk. If (the outage) had occurred earlier, it would have been a lot more difficult."
But, because of the power glitch, end-of-day processing and the batch overnight settlement cycle had to wait until Monday morning.
Since the equity markets still have a three-day settlement, "You still have time to clean up all the trades and not end up with any settlement problems," he says.
Going forward, can a small brokerage firm afford to make the investment in backup systems? Yes, says the CEO. "It becomes a tolerable but significant cost to being independent, whereas really big firms have those economies of scale."
Ivy is Editor-at-Large for Advanced Trading and Wall Street & Technology. Ivy is responsible for writing in-depth feature articles, daily blogs and news articles with a focus on automated trading in the capital markets. As an industry expert, Ivy has reported on a myriad ... View Full Bio