December 12, 2012

An important part of any business continuity and disaster recovery plan is communication, according to the executives interviewed for this article. Immediately following the storm, employees needed to know if the offices were open and if they should report to work.

Ambac has an 800 number on the back of every employee's ID card that serves an information line. Employees can call it and listen to a message about the status of operations and office facilities.

"Communication with senior business leaders is important," Cotter says. "We were making very quick decisions with management" immediately following Sandy. "It is very important to keep communications lines open."

Likewise, business managers need to communicate with their direct reports as much as possible. "We encouraged the managers to reach out to all of the employees who were not actually up in Kingston," Cotter adds. "That kind of communication tree kept people informed and was very successful."

Also, some employees were facing their own challenges, with a loss of power or storm damage to their homes. In those cases, getting to an office might not be their primary consideration. Companies need to account for this in their planning. "When you have a hole in your roof over your kid's bedroom, the last thing you are thinking about is getting to the office," says a data center vice president from a large broker dealer who asked not to be named, since he was not authorized to speak for this article. "I think the entire industry was surprised at the widespread disruption to personnel because of Sandy."

The lack of available workers in the days immediately following Sandy is the main reason why the stock exchanges shut down, after initially indicating that they would rely only on electronic trading. "Originally, the exchanges were talking about staying open immediately after Sandy," Lime's Cook says. "The technology was fine, but we didn't know how many people could actually make it to their office to do any trading. Overall, though, the technology infrastructure held up pretty well."

Lessons Learned
Following the implementations of their firm's respective emergency plans, executives interviewed for this article say they will review and tweak the plans for future events.

Ambac, for instance, will continue to evaluate which applications are mission critical and which are nice to have. Executives will also evaluate the size of the back-up facility, according to Billy Ng, chief information security officer at Ambac. "One lesson I gathered, is the amount of systems we have running may not always align to the business during a disaster," Ng says, noting that the firm's Kingston facility is a smaller version of its regular production data center in New York. "From a processing standpoint, the [back up facility] has to run at a production level. Specking it out appropriately is an evaluation we will continue to look at."

Lime Brokerage's business continuity plan worked extremely well, Cook says. "When you are building a plan, people have a lot of questions about how much effort you really need to put into the business continuity plan," he says. "But, it paid off. We have spent a lot of time on the technology infrastructure. The success of the plan reinforces the time and decisions we put into it."

Greg MacSweeney is editorial director of InformationWeek Financial Services, whose brands include Wall Street & Technology, Bank Systems & Technology, Advanced Trading, and Insurance & Technology.