Compliance

10:33 AM
Ram Nagappan, Pershing
Ram Nagappan, Pershing
Commentary
50%
50%

Shelter from the Storm: Business Continuity & Rethinking Disaster

The one-year anniversary of Superstorm Sandy has financial services leaders revisiting and rethinking their business continuity plans.

Just one year ago, Superstorm Sandy devastated many parts of the US, proving more than a match for businesses and regional and national government agencies. Some 24 states were affected, including the entire eastern seaboard from Florida to Maine and as far west as Michigan and Wisconsin. Damage in the US has been conservatively estimated at approximately $50 billion, with some 159 people killed.

Ram Nagappan, Pershing
Ram Nagappan, Pershing
The effects on the US Northeast, the center of financial services and international commerce, were shattering. For organizations with any kind of exposure to Sandy, this calamity raised fundamental questions about business continuity (BC) and disaster recovery standards.

Understandably, government agencies have focused on this area. The SEC, CFTC and FINRA collectively issued a staff advisory (PDF download) this summer, in which they stated that an examination of firms with a significant market presence showed the need to review BC plans in order to improve response levels and reduce recovery time.

Best-Laid Plans

In the days leading up to Sandy's arrival, financial firms quickly scrambled to brace for the potential impact of the storm. U.S. trading was deliberately halted on Monday morning, October 29. This marked the first time in more than a century that weather halted activity on Wall Street for more than 24 hours. Most firms had an organized business continuity plan of some kind in place. Regardless, when the Sandy-triggered storm surge hit New York City later that day, flooding streets, tunnels and subway lines and cutting power in and around the city, financial services organizations of all sizes were impacted. For example, one midtown investment firm managed to maintain power during Sandy, but lost Internet access. That problem was compounded by standstill traffic preventing employees from getting to and from the office.

[For a look back at the immediate impact of Superstorm Sandy, read: 7 Wall Street Institutions Slammed By Hurricane Sandy.]

The lessons learned were brutal and instantaneous, in part because the scope and scale of the storm's impact went far beyond the assumptions upon which many business continuity plans were based. The sheer scale of the impact of the storm called attention to the need for a significantly revised and expanded approach to BC in financial services.

A Triple Threat

Sandy exposed a number of critical areas where complacency had crept into the thinking behind BC estimations. The storm reminded us that redundancy and resiliency are at the heart of an effective BC plan. Many of the problems uncovered by Sandy can be classified as falling into a critical trifecta of people, communications and fuel.

People: Before Sandy, many businesses assumed that employees could work at home if office access was compromised. But they overlooked the implications of a potential region-wide power loss. In theory, employees with an internet connection at home could easily log in to their desktops to conduct business on a virtual basis, certainly long enough to bridge any temporary loss of power at headquarters. In reality, employees may well find themselves stranded in dark homes without power or cell phone coverage (owing to power outages or cell tower damage) let alone network connections.

[For more tips that all business continuity plans should have, read: 10 Post-Sandy Business Continuity Must Dos.]

This inability of remote users to access core systems underscores the importance of personnel capacity planning as part of the fundamental rethinking of BC planning. Prior to Sandy, most firms restricted remote access to their critical applications to only a fraction of users under disaster recovery scenarios. But the extraordinarily wide regional impact of Sandy pushed the remote access capabilities of many service providers and their clients beyond capacity.

Communications: The central role of communications -- in terms of personnel and equipment -- cannot be overstated. Sandy brought this to light when many firms were unable to instantly reach certain employees, investors and others. A best-practices based BC plan includes contingency-based communication trees that address not only staff but all key stakeholders. For example, firms should consider contracting with multiple communications carriers to provide a failover to a different carrier to maintain fax, voicemail, landline and VoIP services.

Similar contingencies should be enacted to ensure redundancy and security readiness of data centers, since the ability to switch to back-up sites is crucial. One important consideration is to employ an "active-active" configuration for data center. Under this configuration, discrete applications-based routines are running simultaneously, as a back-up precaution, across two or more data centers. This approach is preferable to an "active-passive" configuration, in which backup machines operating in standby mode are activated only after the primary computers fail. Additionally, vulnerabilities caused by outdated security patches and anti-virus software must be eliminated.

[Following Superstorm Sandy, mobile banking wasn't the best way to do business. One reporter found that cash was still still king. Read: Cash Still Rules: 2 Sandy Tales of Customer Service.]

The emergence of SMS (texting) in recent years provides a valuable communications alternative, although compliance with archiving requirements needs to be considered during the crisis. Time-sensitive regulatory requirements must be considered as well. When Sandy hit in late October, for example, firms that had assigned a low priority to month-end reporting in their BC planning faced material delays in reporting timely and accurate information.

Fuel: During Sandy, fuel and power supplies, commonplace components of our day-to-day work, became an Achilles heel for many companies that neglected to prepare for impassable roads, preventing delivery trucks from replenishing gas stations. Not only did this exacerbate the problems of getting personnel to the office, it meant some workers were either unable or reluctant to burn fuel in their cars in order to charge their smartphones and other devices. Firms should consider providing key employees with "air cards" and uninterruptible power supply (UPS) units to facilitate extended re-charging of laptops and cell phones.

Real Tests vs. Good Tests

Closing the gaps in threat scenarios that were exposed by Sandy is paramount. The idea that an entire region could be crippled must be incorporated into all scenarios. Effective plans cannot be counted on fully in a live situation without regular, rigorous testing. In theory, planning and testing must become part of your core competencies. In practice, the sequence of planning and testing must be considered never-ending.

The disaster recovery and "table-top" exercises we run at Pershing, for example, sometimes in collaboration with clients, are designed with the goal of keeping processes and procedures as close to what is normally done as possible. This is why testing must be as real-world as possible. A real test trumps a good test, but it is essential to be realistic from a technology point of view, thereby challenging people to meet a crisis before a real one hits.

Testing location resiliency is another critical ingredient. We continue to be surprised by the number of large financial services firms that lack geographical diversity when it comes to locating their data centers, for example. Duration is an equally important component. Firms must plan for how a long-term disruption of normal business operations -- potentially weeks, as we saw with Sandy -- affects BC efforts.

As a result of Sandy, BC planning can no longer be seen as a set-it-and-forget-it concept. It has become a normal part of doing business. Contingency programs should be viewed as dynamic, living processes that not only encapsulate outside-the-box thinking, but also are continually refreshed and updated. It is essential to understand and test how you will do anything and everything, recognizing the unique demands of compliance, risk management, back-office operations and regulatory reporting. In sum, the concept of risk management must be re-imagined in the aftermath of Sandy, and there is no better time than on the anniversary of those cold, quiet, candlelit nights. In the course of doing so, firms must address a significant redefinition of enterprise risk.

[For more tips that all business continuity plans should have, read: 10 Post-Sandy Business Continuity Must Dos.]

About The Author: Ramaswamy (Ram) Nagappan is the Chief Information Officer and a Managing Director for Pershing LLC, a BNY Mellon company. Mr. Nagappan is also a member of Pershing’s Executive Committee and BNY Mellon’s Operating Committee.

Mr. Nagappan is responsible for the firm's architecture, technology development, infrastructure management and IT operations for Pershing’s NetExchange suite of solutions and Albridge solutions. This includes Pershing's Internet-based account management, clearance, settlement and trading solutions; NetX360 and NetExchange Client Albridge enterprise data management and wealth reporting; computer telephony and mobile technology.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
KBurger
50%
50%
KBurger,
User Rank: Author
10/30/2013 | 1:22:06 AM
re: Shelter from the Storm: Business Continuity & Rethinking Disaster
Ironically these very tools that are supposed to provide more mobility, flexibility, access, etc. are quite vulnerable when there is some kind of disaster. While Ram rightly notes that businesses must anticipate these likely failures & plan for alternative communications, I'm not clear what the alternatives are. It seems like the financial services industry needs to work with municipalities and telecom companies to develop action plans for making this aspect of the infrastructure more reliable.
Byurcan
50%
50%
Byurcan,
User Rank: Author
10/29/2013 | 12:59:39 PM
re: Shelter from the Storm: Business Continuity & Rethinking Disaster
The effect on people is often overlooked when discussing disaster recovery plans. As Nagappan notes, the region-wide power outage cut off many workers completely from their jobs. In my town in Bergen County, we are all huddled at the town hall using a makeshift charging station just to have cell service, and we were without internet for over a week.
More Commentary
Leaving Out the Welcome Mat for Financial Services Hackers
Everyone knows the financial services industry is a prime target for hackers. Despite the dangers, many applications have software vulnerabilities that expose real risks.
4 Surprising Ways Firms Think About Data Security Costs
Almost 28% of firms are willing to bear the cost of some financial losses due to cybercrime, because it's less than the cost of upgrading IT systems.
CIO + CFO Doesn’t Equal Mars Vs. Venus
From my decades of experience, CIOs and CFOs have more in common than you may think.
Will Apple Legitimize Mobile Payments?
The company announced its new mobile payments system, Apple Pay, during a news media event today.
The Art and Science of Leveraging Cloud Infrastructure
Now that cloud providers have addressed many of the more practical concerns of their users, data segregation has become the major challenge in cloud deployments.
Register for Wall Street & Technology Newsletters
White Papers
Current Issue
Wall Street & Technology - July 2014
In addition to regular audits, the SEC will start to scrutinize the cyber-security preparedness of market participants.
Video
Stressed Out by Compliance, Reputational Damage & Fines?
Stressed Out by Compliance, Reputational Damage & Fines?
Financial services executives are living in a "regulatory pressure cooker." Here's how executives are preparing for the new compliance requirements.