Printer Friendly
Lehman Brothers housed its technology development group on three floors of 1 World Trade Center--38, 39 and 40---comprised of about 750 people. Also at this location was a small data center. On 30 floors of the neighboring ThreeWorld Financial Center, the full service investment bank had approximately 5,000 traders, sales and investment bankers in addition to the firm's primary data center on the 25th floor. Lehman's back-up data facility is located at the Hudson Street location in New Jersey.
In the moments that followed the day's tragic events, Beyman immediately tried to contact his colleagues at the trade center. "I couldn't get anybody on their cells nor their office phones. So I called over to 101 Hudson."
"I got in touch with a colleague on the 33rd floor (who happened to have had a view across the river to the WTC) and he starts screaming that another plane is about to hit. I'm in London watching this on CNN while he's watching it in real time," Beyman says softly.
Of course, Beyman's first concern on that day was for the safety of his 750 colleagues on those three floors of the WTC. In addition, he knew that both the WTC and the WFC had been evacuated and was just hoping that his colleagues fled to safety rather than be caught in the vicinity when the towers eventually fell. "It was a tough day," Beyman said several times throughout the interview. Lehman suffered one casualty.
STEPPING INTO ACTION
Beyman's next concern and primary role was to step into action and ensure that the firm's systems were up and running and being backed up properly. First he had to get in touch with a key figure----CTO Bob Schwartz--also located in the WTC offices, who Beyman notes was one of "the most instrumental in the recovery."
The biggest challenge was external communications. Despite his countless attempts to contact Bob and others in New York, there was no getting through via phone. However, through the use of their Blackberry pagers, Schwartz and Beyman devised a plan, which essentially got the firm in place to trade fixed income as early as Thursday, Sept. 13 and equities once the New York Stock Exchange opened on Monday, Sept. 17.
Beyman also had several conversations with colleagues in Jersey City, who he could contact via phone, to discuss what needed to be done that day. To start, the Hudson Street data center was able to run Tuesday night's batch processes. Then they began working on recovery the next day.
Prior to opening up shop, the duo ran into some immediate issues. "When the World Trade Center collapsed, the World Financial Center was affected. Windows were blown out and there was a lot of dirt that got in. They also lost power to the building. The generators kicked in but there was so much stuff in the air that the generators started to clog," Beyman said.
"... The air conditioning vents started to clog and the AC started to fail and the rooms started to get very hot. So what we began to see was that even though we had generators going, we had to take the data center off line at the WFC because we were afraid everything would overheat and start frying machines," he added. At one point the financial center had to be taken off the global wide area network because it was starting to throw off noise and errors--because of the hot data center. He is not sure of the level of damage that the generators or the firm's technology in the WFC suffered.
Jersey City Data Center to the Rescue
Good thing Lehman planned properly for a disaster. According to Beyman, the reason Lehman was able to come online as quickly as it did was because it had redundant sites. The Hudson Street, Jersey City data center and the WFC's data center mirrored each other.
"They were connected by dark fiber, offering very fast connectivity, and the data storage was mirrored on either side of the river using EMC's storage area network. We had cabled this facility with a gigabyte backbone a long time ago," Beyman said.
Despite many issues throughout the next few days, the firm managed to create a trading facility in its former data center at 101 Hudson. "We effectively turned three floors into trading floors," he noted. One and one half were dedicated to fixed-income trading and the other to equities. The data center housed approximately 1,200 people but fortunately had the capacity for 1,800. Now it houses 2,500 Lehman employees.
Lehman immediately knocked on the doors of its workstation vendors Compaq and IBM and its server vendor Sun Microsystems who "without even needing a purchase order got us the equipment we needed." All sorts of vendors helped out by offering office space including ADP, Oracle, SAP, WorldCom, AT&T, Viant, Random Walk Computing, Ernst & Young, among others.
Despite the generous offers, some of which Lehman accepted, the firm still needed a place where it could house its investment banking operation. This group needed to be in one place rather than scattered around New Jersey and lower Manhattan at various vendor locations.
The solution came to them via one of their clients--Starwood Capital. Starwood, which owns the Sheraton Manhattan, allowed Lehman to take over the building. It quickly moved out beds and moved in desks and laptops. The ballroom was transformed into an IT hub where a virtual private network with connectivity back to 101 Hudson St. was set up.
All told, Lehman purchased approximately 4,000 workstations and laptops from Compaq and IBM. And traders have equal or in some cases better technology than they had at the WFC. Reuters market-data feeds are being routed to the new trading facility in New Jersey from London. Bloomberg feeds were already coming into the Jersey City building prior to the Sept. 11 incident and were operational.
The cost of the relocation and new technology? "Lots and lots," Beyman says, joking that he has worked too hard this week to get in trouble over giving out costs that he is not supposed to. He did say that it was not "too onerous." "It's all covered by insurance and we think we will get it all back." It is unclear if anything in the WFC data center will be reusable and even if it is it will have to be recertified," Beyman notes. From a business continuity perspective, we had to get back to business."
In Beyman's 16 years at Lehman--which included an earthquake in San Francisco and the 1993 World Trade Center bombing, he says he has never experienced anything like this regarding disaster recovery.
"Did things go as seamlessly as they could have? Again, I am smarter now and there are some things I might have done differently in the sense of how phone lines are routed. But in large measure I think we were as prepared as you could be .... I am so proud of the technology organization that I am part of and its spirit. What these guys pulled off two days after what they all went through is just outstanding," Beyman said.
"We spent a lot of time thinking about disaster recovery and had drills around Y2K. In these things what you suffer from most is lack of imagination. In many events the disaster that you plan for is often not the disaster that you have to deal with. If I would have had every kind of contingency plan, I would have never have thought I'd have been looking at the disappearance of the World Trade Center."
