December 12, 2012

Business Continuity Planning Pays Off
Meanwhile, a few other firms faired better than companies that have had longer-term problems with returning to their buildings. Lime Brokerage, with its headquarters at 625 Broadway and its data center in Jersey City, NJ, was able to remain in operation with few hiccups, according to Chad Cook, the firm's chief technology officer. "We faired pretty well, but there were logistical challenges," he says. Lime's main office in New York was without power for a week.

Lime moved some people to its Jersey City data center in advance of the storm, so they could work from there and also make sure the data center was up and running. "The employees were in hotels near the data center," Cook says. "But because of damage and flooding, if you weren't near the data center, you couldn't get in or out of Jersey City for a few days" following Sandy. "We were fortunate for the data center didn't lose power," nor was it flooded, Cook adds, noting that there were other areas in and around Jersey City where there was flood damage. Lime's data center is equipped with back-up generators, he says.

Cook and most of his technology team are located in Lime's technology development center in Waltham, Mass., where they coordinated operations. For instance, everyone who was available to work needed to have a phone number assigned to them. "All of the phone system has redundancy through the PBXs in Waltham," Cook says. "We had to reassign all of the phone numbers to the people in Jersey City and to the people who worked remotely."

[For more on how banks reacted to hurricane Sandy, read Cash Still Rules: 2 Sandy Tales of Customer Service.]

Luckily for Cook, because of Lime's data center location across the Hudson river, the phone system was Lime's toughest business continuity challenge during Sandy, which is a blessing compared to what other companies had to face, he notes. "The phone system was the most complex. The complexity of a phone system is something we are always looking to improve. Overall, we did pretty well."

Just two miles to the south near the southern tip of Manhattan, municipal bond insurer Ambac Assurance Corp.'s building had five feet of water in its lobby, with a completely flooded basement. Although Ambac's offices are on a higher floor of One State Street Plaza (Map), the building was unusable for weeks, as major repairs needed to be made, due to the record storm surge.

According to John Cotter, chief technology officer for Ambac, the company enacted its business continuity plan in Kingston, NY (100 miles from Manhattan) before Sandy struck on October 29. "When we heard the storm was coming in and when we heard about the size of the storm surge, we activated the plan," Cotter says. A team was sent up to Kingston to "spin up" the facility and to prepare it in case of a disruption in New York City. "Once the storm surge hit, it took out our New York facility immediately."

Ambac's Kingston back-up data center is actually housed in a former IBM disaster recovery center, according to Cotter. The company bought the building eight years ago after 9/11 and built it to house approximately 90 people. Cotter described it as a mini version of Ambac's main data center that can run the organizations critical systems.

For Ambac, like many companies located in storm-surge prone areas of lower Manhattan, this was the second time in two years that a weather-related event caused the business continuity plan to be implemented. In October 2011, hurricane Irene had Ambac send a team to its back-up facility too. "Last year, we weren't sure with Irene, but we did enact it on a smaller scale," Cotter says. "We spun up the facility and sent a team up to Kingston. We were ready, but Irene didn't impact us."

This year, however, Sandy closed Ambac's main office in New York for more than three weeks, with power finally being restored to the building the week before Thanksgiving. However, Ambac didn't start moving people back to the building until after the holiday weekend. "Once the power was restored, there were air quality issues and we wanted to return in an organized fashion," Cotter says. "We took several days over the Thanksgiving weekend to reverse replicate all of the data to the State Street servers," Cotter says.

NEXT: Communication and Lessons Learned