A Decade After 9/11, Business Continuity Is Still A Work-In-Progress On Wall Street

In the immediate aftermath of a disaster or pandemic, people usually take stock of how unprepared they were for a crisis and are full of good intentions that next time disaster strikes, they will never again be caught unaware. But just a few months after any major crisis, a reality check usually shows that good intentions have given way to what now seems to be more pressing budgetary priorities, and planning momentum for a disaster has subsided.

Still, in spite of these natural ebbs and flows, 9/11 marked a turning point for business continuity. Although it looks like the real game changer in the disaster recovery arena occurred a lot more recently.

Ten years ago, Wall Street firms' back up data centers were often located just a few, unsafe, blocks away. Traders and other workers did not tend to work remotely. Few people had smart phones. (Research In Motion had just developed the BlackBerry in 1999). Disaster recovery planning cost millions of dollars, and few firms were willing to spend the money on an area that didn't have an immediate business value.

Fast forward to 2011: working remotely has become pervasive on Wall Street and elsewhere and no one can imagine life without their BlackBerry or iPhone; Wall Street firms generally have back up data centers tucked safely away in New Jersey or Pennsylvania; and strict regulations on disaster recovery, whereby firms must have contingencies for providing customers with access to their funds and securities during a disaster, as well as data backup and recovery, an alternate physical location for employees, and communication among the firm and its employees, have forced firms to rethink their business continuity plans.

"Everything has changed, and it's changed in so many different ways," says Alex Tabb, partner at TABB Group. "Before the tragic events of 9/11, business continuity existed but it was a quiet little hideaway place to warehouse those who weren't moving forward in the organization. It was a backwater of contingency planning. It was a completely different game. If you look now at prime brokerages, and tier one, tier two and tier three organizations, and even mid-level firms, business continuity capabilities are light years ahead."

The real game-changer

Despite the new focus on business continuity after 9/11, Tabb suggests that it was really the financial crisis of 2008 that forced Wall Street firms to re-think their disaster recovery approach.

"It forced people to say, 'OK, I may have to worry about another terrorist attack, but what I really have to worry about is my business going away. You could no longer say, 'We're doing business continuity and have to spend X amount of dollars.' Now, it's about adding value to the business," he explains.

Today, firms have robust communications in place that are accessible anywhere - not just for business continuity but for broader business purposes. Firms have realized that they can harness virtual technology to increase their employees' efficiency -- not just during a disaster, but on a regular weekday too. "It means you can work from home at 11pm on a Thursday, for example," Tabb points out.

Traders can trade from home or any other location where they have a secure and fast internet connection, thanks to advances in areas such as IP technology and Session Initiation protocol (SIP), the standard language that communications providers now use to send and receive real time data across an IP network, and which has been widely used for about 5 years.

"Whether you're doing a voice call or a live meeting call, it's a default standard that allows two different endpoints to negotiate a connection," explains Jonathan Morton, VP of product marketing at IPC, a network and trading communications provider which makes the private communications lines traders use available at disaster recovery locations as well as their regular work-place."By using SIP protocol and IP technology, they can have calls ring at more than one location at a time," he adds.

Despite the advances in technology and new regulations that have come about in the last decade, experts argue that it is the Great Recession rather than 9/11 which has really transformed the way firms are looking at business continuity.

BC budgets take a hit

Like other areas of IT, firms have been forced to take a good hard look at their business continuity spend. "Today, you cannot get things funded for business continuity reasons. No one will pay for a hot site in upstate New York that's costing $150,000 a month and just sits there. So all the firms are looking at ways to enhance both continuity and business," Tabb asserts.

As the virtualization of trading platforms and other services and cloud computing become ubiquitous on Wall Street, firms are implementing these solutions to more cost-effectively manage their business on a daily basis. Today, disaster recovery is just one other area where they have found they can leverage these technologies.

The use of the cloud for communications, storage and computing mean you can work from anywhere, for whatever reason. "You could be on vacation in Martinique and still have access [to your VPN]," Tabb notes.

Work in Progress

Yet business continuity presents additional challenges that Wall Street firms are far from having resolved.

Cost-cutting and lay offs across the board raise the question as to whether firms even have the personnel to regularly train and test business continuity systems.

But the biggest challenge of all is data, experts agree. Keeping backups of several terabytes of data for a year's storage of data is expensive. And smaller firms may not be able to access their data as quickly as they should. "A lot of people are in a primary data center and pay a lot for low speed connectivity," Tabb points out. "But they don't want to spend millions of dollars of equipment to have the same low speed direct access in a backup data center. If I have a primary site and 50 servers connecting to an exchange and I need a backup site with all the same servers, it gets very expensive. If there's one thing that has yet to be fully solved, it's that question. How do you get robust continuity without spending a fortune?"

Worryingly, most firms are still not confident that their data will be restored completely within hours of a disaster, according to a survey carried out earlier this year by BUMI, the New York-based company formerly known as Back Up My Info!, which provides managed online backup and recovery solutions.

Currently, online data back up is the most popular way of backing up information for 65% of firms, followed by internal appliances (34%) and tape backups (29%), according to the BUMI survey which was completed by 117 business owners, CEOs and IT professionals at small to medium-sized businesses.

Given that the amount of data residing on servers, desktops and laptops is continuing to grow exponentially, businesses must think about recovery first and then develop a strategy to ensure that their most critical data is in the best position to be restored, says Jennifer Walzer, CEO of BUMI. "IT managers don't mind buying more servers," she adds. But in general, they don't think, "How can I restore or recover my data?"

It's not enough to have a business continuity plan in place, she asserts. Firms need to regularly test these plans. But many do not.

Sang Lee, founder and CEO of the advisory firm, Aite Group, agrees that Wall Street has been struggling to handle spiraling amounts of data. Most have not yet settled on an efficient way of dealing with it, he says. "Firms have focused on being able to access data from different locations, but there's a risk involved in that as well, a risk that you could lose your data altogether," he asserts. Firms need multiple instances of backup. "It's not a cheap proposition. The larger firms can deal with it by throwing resources at it. But smaller firms have a lot more challenges."

Cloud computing is a more cost effective way of managing data, Lee adds. "And it will play a role in the future for small to mid-size firms."

Tabb agrees that private clouds, including new offerings such as NYSE's cloud or Thomson Reuters' Elektron could help firms achieve robust continuity and direct access at all times. "The cloud could be a very powerful tool to help with that in the future."

Still, most Wall Street firms continue to debate the pros and cons of cloud computing, even within the context of disaster recovery. "The major concern of the cloud, real or perceived, is security," Lee says. But, he ponders, if disaster strikes again and you risk not having access to your data,"Which is worse?"

Melanie Rodier has worked as a print and broadcast journalist for over 10 years, covering business and finance, general news, and film trade news. Prior to joining Wall Street & Technology in April 2007, Melanie lived in Paris, where she worked for the International Herald ... View Full Bio