Risk Management

05:20 PM
Larry Tabb
Larry Tabb
Connect Directly

Flash Crashes, Kill Switches, and 10 Million Easy Buttons

If firms don't start to do a better job of testing software and systems, regulators may do that job for us. And then things will get really complicated.

What's wrong with our markets? Rogue algorithms force exchanges to break trades. Old code causes the near collapse of a global broker. Glitches scuttle one IPO and create millions of dollars in losses for another. This isn't supposed to happen! What is going on?

Have we forgotten how to write code, or have the markets become too complex for our testing scripts? Perhaps the rush to turn over code is so intense that we're bypassing best practice for software development procedures. Have we outsourced one too many technologists? Or were we just incredibly lucky up to this point that we only recently started having these problems?

In fact, we haven't seen these types of software problems since the mid-1980s, when three or four multibillion-dollar technology fiascos temporarily shuttered several businesses and cast egg on several CIOs' faces.

But what should we learn from these events? Instead of adding more regulation, or the SEC hiring super techs to audit every line of code, maybe we should just go to Staples and buy 10 million Easy Buttons reconfigured to stop the trading process at the mere hint of a problem.

OK, perhaps Easy Buttons aren't the answer. But I do think the answer lies in fixing our flawed software development, quality assurance and turnover process. We can speak of Six Sigma, process improvement, total quality control or whatever buzzword we want to use, but ultimately, it seems like quality in the software development process is getting lip service at many an IT organization.

TABB Group estimates IT spending for the U.S. securities industry has declined to $21.4 billion this year and is on its way to $19.8 billion in 2013. This is down almost a third from the peak spend of almost $29 billion in 2007. Now this savings doesn't come from just cutting people. It comes from the increasing cost-and-power dynamic of new technology, the shift away from proprietary platforms, the migration of support services to lower-wage economies (both domestic and overseas) and the consolidation of some major industry players during the credit crisis.

The problem, it seems, at least from the outside, is that our demand for computational and processing services has outstripped our ability to develop software at a service level we've come to expect and demand. Besides getting our business analysis, software development, testing, QA and turnover processes right, we need to think about how we manage technology overall. And we have to consider the possibility that we can't do it all. We may have to rethink how applications fit together, the level of complexity built into our software, how we maintain our systems, and then how we jam new functionality into an increasingly old, kludgey patchwork of legacy technologies.

[High-Speed Trading Slows Down]

This problem won't get better soon, nor will it fix itself. The markets and trading technology will only get more complicated. Not only are financial products getting more complex, but the financial world continues to expand across products, asset classes and geographies. This is occurring at a time when the industry's appetite for IT spending continues to be constrained. Many firms are rationalizing their heterogeneous software platforms into global platforms that can be used across many products and trading strategies. However, this just makes technology platforms more complex and fragile because the systems need to accommodate a wider array of challenges. Consolidating multiple instruments, calculations and regulations into a single platform is much easier said than done.

We need to be careful as we move into 2013. Technology budgets will most likely be down again, and the pressure to do more with less will be greater than ever. The need to manage an increasingly complex array of tasks with an even smaller budget will force firms to rethink their entire processing strategies.

We need to remember this: However strong the pressure to trim budgets, rationalize infrastructure and streamline processes, the industry still needs to get it right. Firms need to have the processes in place to ensure that software is bug-free, tested and deployed appropriately. Otherwise, regulators and legislators will step in and do our jobs for us. While that may sound like an easy way to get extra budget, it will certainly come with a boatload of challenges that will not only make the business more difficult, but it will also make our lives as technologists much more complicated before they get better.

Larry Tabb is the founder and CEO of TABB Group, the financial markets' research and strategic advisory firm focused exclusively on capital markets. Founded in 2003 and based on the interview-based research methodology of "first-person knowledge" he developed, TABB Group ... View Full Bio
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Ninja
10/26/2012 | 7:16:05 PM
re: Flash Crashes, Kill Switches, and 10 Million Easy Buttons
First of all one statement that says it all on code:- "The short order code kitchen burned down a few years ago"...this means now it is complex with aggregation and one set of data reaching out to multiple destinations.- Hey I get questions from analysts about code so this is not just a problem of the IT department.- I would say we have about 12 years in trying to get the IT infrastructure of Social Security out of cobol, does that give you some time elements to think about?- When there's talk about rebuilding Medicare from bottom up and creating voucher systems, would that happen?- No, people don't use some common IT sense and neither do politicians.- It's makes great media chatter and gets everybody's bowels in an uproar and makes some folks not look very smart:)-

Yes for such a project being so massive with huge IT infrastructure needs, shoot we would have to probably outsource to both India and China to get it done and would be looking at 7 to 8 years....total repeal of Obamacare..the talk is a joke that goes nowhere.- We can however make incremental changes as economic conditions change and that we should do

Do yourself a favor and go talk to some technologists to get some back up on what I just said and they will tell you if you ask, nicely:)- Also they have to be in a position of not fearing losing their job if they speak up.-

I know it's frustrating trying to find solutions and all we get in the media are the 50 shades of rape from digital illiterate lawmakers, they don't know what else to talk about sadly as they are the non participants in a great deal of technology and yet they have the responsibility to make laws...they are killing us.- We have some executives too on the other side at the SEC and HHS that need to move on too, same problem and nothing personal intended.-

Yes back in the early days of technology I could go to my code kitchen and whip something up as its easy when you don't have anything and you build everything uniquely from scratch but in today's world it is not like that any longer so nobody has forgotten how to write code for goodness sakes, its more complex.- Again people who write code get this and the new programmers don't have some grass root experience from the "early days" either and that makes a difference.-

Then we have the flawed data and Data Duping that is occurring out there and the Quant business models that nobody understands, how many times did we see Jamie Dimon say "I don't know" when asked about his model.- Ok IT systems are built around these models, getting the picture yet?- So when you have models containing formulas with a bunch of fiction and flawed data, where does it get you when formulas were written for "desired' results with little care for "accuracy" when the formulas can make billions in profit.-

3 videos at this post will explain it better than me, Quant talking, professor explaining bogus formulas and last but not least the impact of algorithms in our lives, how they give and take away.- No offense meant at all but this is the same message I put out to everyone and it's picking up steam as it's logic and it's the truth and even the NISS wrote and told me to keep making noise as the Killer Algorithm series is all about the unfair use of math and formulas running on servers 24/7 that you can't see, talk to or have lunch with:)

More Commentary
A Wild Ride Comes to an End
Covering the financial services technology space for the past 15 years has been a thrilling ride with many ups as downs.
The End of an Era: Farewell to an Icon
After more than two decades of writing for Wall Street & Technology, I am leaving the media brand. It's time to reflect on our mutual history and the road ahead.
Beyond Bitcoin: Why Counterparty Has Won Support From Overstock's Chairman
The combined excitement over the currency and the Blockchain has kept the market capitalization above $4 billion for more than a year. This has attracted both imitators and innovators.
Asset Managers Set Sights on Defragmenting Back-Office Data
Defragmenting back-office data and technology will be a top focus for asset managers in 2015.
4 Mobile Security Predictions for 2015
As we look ahead, mobility is the perfect breeding ground for attacks in 2015.
Register for Wall Street & Technology Newsletters
White Papers
Current Issue
Wall Street & Technology - Elite 8, October 2014
The in-depth profiles of this year's Elite 8 honorees focus on leadership, talent recruitment, big data, analytics, mobile, and more.
Inside Abel Noser's Trading Floor
Inside Abel Noser's Trading Floor
Advanced Trading takes you on an exclusive tour of Abel Noser's New York trading floor, where the agency broker known for transaction cost analysis, is customizing algorithms for the buy side, while growing its fixed income trading and transitions business.