Wall Street Firms Turn to Best Practices Models to Speed Up Software Delivery
Wall Street firms have never been hungrier for new software programs in areas such as trading, foreign currency and asset management. Yet the business side has lost its patience with IT, as well as its tolerance for software bugs.
"People are getting very frustrated with the traditional ways of building software on Wall Street because a lot of projects are not predictable, they come in late, they cost more than anticipated and the users are unhappy with the software that gets delivered," says David Chapman, who at the time of the interview was CTO of financial services IT consultancy Finetix, which was subsequently acquired by Sungard. "There's a lot of IT dollars getting spent, yet there's a horrible track record of delivery and a lot of unhappy users out there. People are starting to realize there's got to be a better way of building software."
Technology is such an embedded part of business strategy that firms' expectations and needs continue to escalate, adds Alex Cullen, research director, IT leadership, at Forrester Research. "Wall Street's No. 1 requirement is predictability," he says. "The IT organization that runs in an undisciplined manner fails the business too often."
Over the past few years, however, investment banks have been more focused on outsourcing and cost reduction in IT than efficient software development. "The banks have been good at reducing cost and moving IT off shore," Sungard's Chapman notes. "But over the last two years, as the markets have picked up, a lot of banks have realized there is a gap now in their time-to-market delivery capability."
How to mend that gap is open to debate. Some, including Merrill Lynch and JPMorgan Chase, are investing resources in the Capability Maturity Model (CMM, or the newer CMMI version — the "I" stands for "integration") that enforces discipline, processes and measurement around software development. Some favor ITIL (Information Technology Infrastructure Library), a set of best practices recommendations for IT service delivery. Others insist that agile software development practices are the best route to speed and quality of software development. And still others, including HSBC and Mellon Financial Corp., use a combination of approaches.
All these methods can be lumped under an umbrella term — IT maturity. A recent study conducted by analyst firm Ovum asked 300 North American IT decision makers about their engagements with IT service management initiatives such as ITIL. Just over half (51 percent) said they have implemented ITIL in at least one or two areas. More than a third (34 percent) said they were evaluating ITIL and/or IT service management best practices, and 15 percent said they were pursuing some kind of IT service management initiative, but not ITIL specifically. Why so much interest? "IT project management is an area many companies are weak in," says Forrester's Cullen. "It's the No. 1 gap for all firms."
Wall Street has special incentives to get its IT house in order. One is the business side's demand for new applications delivered quickly. A second is the need to attract highly skilled developers, especially in lower-cost places such as India and China where the competition — outsourcing firms — is often at CMMI Level 5. (India-based outsourcing firms in particular long ago realized that the way to overcome resistance is to be hyper-credentialed.) A third is the desire to make lower-cost, outsourced development more palatable to the business users; through established best practices, users come to know what to expect. Fourth, CFOs and CEOs are demanding zero- or low-defect applications. And fifth, having common practices and standards among IT and development groups can help ease the integration aspects of mergers and acquisitions.
Here's one more: survival. "IT groups in companies like Goldman Sachs, Merrill Lynch and JPMorgan are having to become more competitive in order to survive; they're under threat" from outsourcers, says Herb Krasner, professor at the University of Texas and president of software excellence consultancy Krasner Consulting. "If they can't do the job and do it well, they're going to be outsourced."
Merrill Improves Quality Via CMM
Merrill Lynch was looking for ways to improve software development and project management within its software technology development organization when Ira Lehrman, CTO, global investment banking and investor client technology, attended a CAI (Computer Aid Inc.) software best practices conference and met Krasner. "The conference inspired me to find out what level my organization at Merrill Lynch was at and [at what level] we should be," Lehrman says. "I asked Dr. Krasner to assess my organization, and in February of this year, he determined that we are at Level 2. We aspire to be a 3 within 18 months."
Lehrman expects to reap good project management, more highly skilled staff and better metrics from the exercise. He also expects challenges when it comes to institutionalizing CMM across the board, but he adds that he and his 350-person team are committed to making it happen. "After we went through the initial assessment, the team was fully engaged and fully supportive," Lehrman relates, adding that his goals include increasing client satisfaction and producing better software in more predictable time frames. "At the end of the day, I want to deliver quality software," he says.
Among organizations that go through such a CMMI transformation, Krasner contends, the overall return on investment can be as high as 15-to-1, largely because employees are more productive. "That's measurable in things like reduction in the number of hours they work per week fighting fires and tackling crises," he explains. "The program removes all the chaotic behavior that can cause software developers to spend 80 hours a week on the job when they should only be spending 50."
Mellon Aiming to Combine CMM With Agile
At Mellon Financial, the Capability Maturity Model has been ingrained in IT operations since the firm started a CMM effort six years ago. CIO Kevin Shearan brought CMM principles with him from Citigroup. The first IT group reached Level 3 in December 2001, and many groups have been recertified at that level three times, according to Mike Keslar, Mellon's director of technology delivery. Now the firm is blending its processes with newer agile-development methods to make ongoing software engineering and project management more efficient.
In the beginning, Mellon's CMM initiative got some pushback from the business side. "The old model was, developers would go in and talk to the business people about the software they wanted built, then the developers would go off and start the code," recalls Keslar. "For the business users, it was easy to easy to sit in a room for an hour and explain what they wanted and have someone start to build it. But in the long run that's very inefficient and ineffective."
CMM requires the business side to carefully document its requirements, which was hard for some, Keslar notes. "But once they've seen the full cycle multiple times, the businesses insist on leveraging the process even if we don't ask them," he says. "They've seen the value in the quality, the time to market and the predictability."
For instance, in the foreign exchange area, traders and salespeople don't typically want to spend time sitting around drafting application requirements. Yet during a recent meeting, according to Keslar, the director of foreign exchange spoke highly of the new model, commenting that now he believes the technology team's estimates and that they've accomplished large conversion and development projects that would have been impossible without the CMM processes. Keslar notes that application upgrades are now performed regularly and consistently, with good communication and no surprises, and software defects are fewer.
CMM also has helped Mellon work with outsourcers. "When we decided to outsource some of our work to two offshore vendors, there were startup and communication challenges; it's a new mode of operating," Keslar says. But the offshore vendors were both certified at CMMI Level 5, and having a common language was a huge advantage, he adds.
Similarly, the CMM efforts should help smooth the upcoming merger between Mellon and Bank of New York, which also has CMM certifications ranging from Level 2 to Level 5. Both Kevin Shearan and BONY CIO Kurt Wetzel have said they'll continue to embrace CMM processes post-merger.
Mellon is currently working toward CMMi certification, CMMi representing a change in appraisal methodology that requires additional evidence to demonstrate maturity compliance. Convinced that consistent utilization of CMM processes has lead to improved quality and predictablity, Keslar says that the way forward now is to streamline the process, removing as much overhead as possible, and starting to use agile development tools. The work in agile development will be particularly helpful to Mellon's asset servicing business, where being quick to market with new technologies provides a significant competitive advantage.
Waterfall vs. Agile
Mellon's shift to agile software development is part of an emerging trend. "Every investment bank and hedge fund I've spoken to is looking at agile," says Sungard's Chapman. A relatively new term, agile development is based on iterative development — developing software in small, manageable chunks that can be modified as requirements change, yet using a disciplined software delivery mechanism.
Historically, the software development approach used throughout Wall Street has been the "waterfall" method, which calls for strict, lengthy analysis and documentation of requirements. For a one-year project, for example, three to six months might be spent on needs analysis. "The business people are expected to define 100 percent of their requirements up front before the project even starts," Chapman says. "People get stuck in this analysis paralysis — they spend months and months trying to define what they want."
Another three to six months can be devoted to software design, then the actual program finally is written. "Inevitably what happens is requirements change, integration becomes very difficult and all the risky software development happens at the end of the development effort," Chapman explains. "The waterfall approach has a horrible track record of delivery."
Agile software development is designed to deliver software more quickly yet maintain high quality. In agile methods, every two or four weeks, businesspeople get a small amount of code to review and the opportunity to change the requirements. "Imagine a hedge fund where traditionally a new credit derivatives trading system would take a year to build using the waterfall approach, with businesspeople writing six months' worth of documentation versus using an agile approach, where some of the system is delivered in two weeks, and it's OK if you change your mind," Chapman says. "For the hedge funds particularly, agile is an extraordinarily good fit because the portfolio managers want to get things done quickly."
But not every project lends itself to short iterations, Chapman concedes. "On Wall Street it's not so easy because there are a lot of other systems you need to integrate with," he says. "But I think there are parts of agile you can use on every project to improve it."
Agile development has three levels: developer, project and enterprise. "Nobody on Wall Street is using agile at the enterprise level," Chapman says. "A lot of education needs to take place within the banks — it's going to take some time. But I think every project could gain some benefit from trying to break down the project into more-manageable chunks that can be delivered in a more iterative and agile way."
Agile methods even improve software quality, Chapman contends, because they emphasize testing. Agile methods encourage developers to do their own testing, often requiring them to write the tests before they write any code and to develop automated testing routines for the programs they deliver.
"Agile development approaches and CMMI are compliant with each other — you can use CMM and CMMI to make agile software development better," Chapman adds. On the other hand, he asserts, trying to use CMM and CMMI on top of waterfall development approaches will just weigh projects down with bureaucracy and paperwork.
HSBC Attracts Talent Through CMM
HSBC has more than 25,000 IT staffers and develops applications at 16 locations around the world. What used to be done locally now has to be done globally, and IT teams need to work together all over the world to build global systems. Its global technology group — 6,000 people in India, China and Brazil — is in the process of being certified at CMMI Level 5. The major GLT centre in India is already certified to CMMI level 5, China is at level 3, and the China and Brazil groups will progress to level 5 certification.
"When you create one center to do IT development in support of teams that are all over the world, you want to have consistency in process," notes John P. Carr, COO of IT at HSBC Global Group. "We wanted to bring in some level of continuity in the way we do systems development.
In addition to boosting the confidence of the internal IT customers in the offshore groups, Carr also had recruitment goals. "Professional certification adds to your attractiveness to bringing in quality staff," he explains.
So far, the CMMI project has cost $700,000, according to Ignacio Vera, global head of global technology, HSBC Software Development. "We trained about 3,000 staff because it's so important that everybody follow the same methodology in all their processes," Vera says. "You need to show consistency in process — that means that you guarantee almost every time you'll deliver the same quality of software."
The most important benefit has been the calming effect that Level 5 certification has on the stateside IT and business managers, according to Carr. "[CMMI certification] helps settle down the initial nervousness of local IT managers, who are the ones held accountable for final delivery, that they're engaging with an [offshore] IT organization that's more structured and certified than they are," he says.
Further, in the past local IT managers sometimes had to turn requested projects down because they couldn't make a business case for them, Carr continues. Today they can turn to the global tech centers to do that work more affordably. "Being able to say 'yes' to the business, being able to add an IT element to a business case justification that lets us do bigger things for a company that has an insatiable appetite for IT is really what this whole thing is about," Carr says.
The Dark Side of CMM
The Capability Maturity Model has its naysayers. Some feel it introduces a lot of time-consuming paperwork and can get in the way of fast development. "The idea was that CMM and CMMI process improvement would help people develop software better, in a more repeatable fashion. But that hasn't really happened," Sungard's Chapman charges. "What happens is people get bogged down in paperwork, artifacts and documentation, and the net result is the amount of software delivered doesn't increase; neither does the quality."
"That's absolutely what happens up front," admits Mellon's Keslar. "But once people are using these processes in the right way, I would argue with the idea that they're a hindrance or too much bureaucracy." He points out that firms need to be flexible in the way they use CMM; it's not always necessary to follow every process to the letter and fill out every piece of documentation. For example, when doing a recent research analysis project for Mellon's foreign exchange department, using internal data to perform predictive analysis for foreign exchange currency, stock and bond markets, the development group followed the CMM processes very lightly. Keslar asked for and received a waiver for certain CMM deliverables because it was only research.
HSBC's Carr concurs. "At the end of the day, IT teams get measured on one thing: Have they consistently provided a quality deliverable — do we deliver what we said we were going to do for the price and the level of quality that business has come to expect," he says. "While some may look at this as bureaucratic, it's worked for us. ... Business satisfaction has continued to be higher and higher as we mature in this process."
Sidebar:
For a quick primer on some common development methods and practices, read:
Tweet This