Sometimes the next big thing isn't new at all. That's precisely how Saro Jahani, the new CIO of the Direct Edge stock exchange, views stability and latency: The two fundamental pillars of IT performance are the coming wave on Wall Street, he indicates.
Indeed, prior to joining Jersey City, N.J.-based Direct Edge, the Swedish-born Jahani, who is of Armenian descent, proclaimed during a Wall Street & Technology conference in 2009 that he wouldn't be surprised "if the newest title on the Street was chief latency officer." Having reflected on his comments for a few years, however, Jahani now suggests that "chief stability and latency officer" comes closer to the mark.
"Stability and latency go hand in hand," explains the veteran IT leader, who grew up in Sweden and, after a long career in that country, made his way to the United States in 2000. "I place a higher value on stability, because it's critical to performance management," Jahani adds.
"In our industry, there is a lot of work to be done" to address stability and latency, he continues. "If an exchange, or an exchange operator, doesn't have an architect dedicated to stability, latency and performance management, I don't believe that exchange has much chance of surviving."
Putting Beliefs Into Action
Not surprisingly, Jahani practices what he preaches. Upon joining Direct Edge in April 2011 to manage an IT staff of about 70 with a $35 million annual budget, Jahani charged his chief architect to serve a dual role as chief stability and latency officer. In practice, Jahani says, "50 percent of his time is focused on stability and latency."
Early on at Direct Edge, Jahani established the so-called Extended Architecture Committee (EAC), which brings together Direct Edge's technology partners -- ranging from major industry players such as Cisco, Hewlett Packard, Microsoft, Informatica/29 West and Intel, to internal verticals responsible for everything from automating the exchange's regression environment to managing its power and cooling.
"In our industry, there are no trade-offs. We must work with all of the components of information technology to make our business successful," asserts Jahani "By bringing in our vendors quarterly for the EAC, we're engaging the industry in our game. We're challenging them to live up to each other's expectations and ensuring they stay focused on one common denominator: the Direct Edge platform."
Additionally, two of Jahani's top IT initiatives for this year emphasize stability and latency, with a third that's closely related. "As an exchange, we have to demonstrate we have a technologically savvy, reliable, consistent and sustainable environment," comments Jahani. "Our three programs of work for 2012 address those needs." They include a stability and latency road map, a "sandbox" production simulator, and lights-off systems monitoring.
The goal for the road map is ensuring that the critical elements of the exchange's platforms operate in real time, or as close as possible, according to Jahani. To get the job done, his team started with a "single point of failure" analysis, which served as the basis for determining what IT needs to accomplish. Then the team divided projects within the road map into three distinct categories: order acknowledgement, market data systems and quotes. For each of these, every hardware and software system is fine-tuned regularly to improve stability and reduce latency.
According to Jahani, the order acknowledgement category addresses Direct Edge's transactional systems. Market data focuses on the subscription services the exchange offers its clients and, for quotes, the aim is to minimize the data life cycle "to ensure the most up-to-date quotes are available to our matching engines," he says.
The sandbox is a mini-exchange where every element on the road map runs as a proof of concept before going into production. "Early in my career I worked at a nuclear simulator," recounts Jahani. "There I learned that, no matter how smart you are, everyone makes mistakes."And though Jahani spoke with WS&T before Knight Capital's software error cost the market maker more than $400 million, he stressed the importance of testing.
"At Direct Edge, we've created the sandbox as a simulator," he continues. "Everything that we innovate -- whether a new application or hardware configuration -- is sandboxed. This includes ensuring our innovations are backward-compatible, because that's where most new systems fail."
The third initiative, lights-off monitoring, serves as an early warning system to avert production mishaps. "It starts with agents running on every server, network node, storage devices, applications and so on," relates Jahani. "These probes constantly stream messages to monitoring systems, where we've defined key risk indicators and set thresholds. This, in turn, gives our monitoring systems the ability to proactively sense there's an issue and deliver the right alerts to the right people."
He adds, "For example, if our normal latency is 200 microseconds and it jumps to 250, we want all hands on deck to prevent a problem from occurring."
A fourth area of emphasis, adopting ITIL (Information Technology Infrastructure Library) best practices, actually began in late 2011 and will continue indefinitely, Jahani reports. "Naturally, following ITIL best practices is a lifecycle effort that never goes away," he says.