Next-Generation Data Centers: Wall Street Keeps Data Centers Lean, Mean and Green

Wall Street firms have to make smart technology and design choices to optimize data center performance and minimize latency while remaining lean and green. Colocation, remote data center management, ultra-virtualization and new cooling technology are among the latest techniques for achieving this balance.

Technology executives charged with running Wall Street's data centers have a difficult balance to strike: On one hand, demand for high performance and low latency to support algorithmic trading has never been greater; on the other hand, subprime write-downs and the troubled economy portend IT budget cuts while skyrocketing energy prices necessitate efficient use of energy.

Nasdaq, Lime Brokerage, Capis and the designers of the first Leadership in Energy and Environmental Design (LEED)-certified data center are achieving this balance with the use of state-of-the-art technology and design techniques, including high-speed networks, latency monitoring tools, organized virtualization, improved cooling technology, colocation and remote data center management. (For more on the Capis and LEED-certified data centers, see related articles, pages 33 and 34.)

Driving Low Latency

The top priority for any Wall Street data center is to ensure low data latency so that as market and order data travels to and from exchanges, algorithmic trading programs and order management systems, the data center helps speed it on its way rather than slow it down. Network topology, server speed and physical distance are among the many factors that can reduce or add to data latency in this environment.

To keep up with the pace of change, Lime Brokerage is in the process of rolling out its third new trading platform of the past six years. "The first wave was a millisecond war," says Tony Amicangioli, chief technology officer for the New York-based brokerage. "It was considered amazing if you could turn an order through a machine in one millisecond. Now, with our latest rollout, we're down into the microseconds for performance under loads of tens of thousands of orders per second per box. It's an ongoing thing -- you have to keep up."

Although much of Lime's low-latency technology is proprietary (according to Amicangioli, the firm currently is seeking patents for some of its solutions), Amicangioli says that some of his group's work is directed at "taming the Linux beast." "Linux is a popular operating system that was designed for a different audience," he explains. "Its average user is someone who's less concerned with latency and more concerned with throughput — with how much raw data they can deliver. We're taking a throughput design and finding a way to make it very fast from a latency standpoint."

According to Amicangioli, Lime is constantly working with telecommunications vendors to get as close as possible to real-time, speed-of-light connectivity between its Jersey City, N.J., data center and the trading venues. Most of the networking is gigabit Ethernet, but "wherever the I/O [input/output] rates of a box are an issue of focus or concern, we have 10-gigabit solutions," he says. "We haven't had a need to get to a full 10gigE data center yet, but who knows what next year or the year after holds?"

Lime uses a home-brewed latency monitoring system. "We decided long ago we wanted to build a high-performance box that would measure the latency of data coming in and out of our boxes from clients and then from our box onto various trading venues, such as Inet, Arca and BATS," Amicangioli relates. "It's been a very useful tool; it not only lets us measure latency to the microsecond, but it's also acted as a scope into the venues to see how well they're doing."

Lime also has begun colocating some servers with Savvis, a St. Louis, Mo.-based provider of hosted data center facilities, at the behest of customers that would like their applications to be closer to the exchanges, including Arca and BATS. Amicangioli says, however, that due to the latency improvements the brokerage already has made, colocation is almost a moot point, as it only takes five microseconds (i.e., five millionths of a second) for data to move from Lime's primary data center to Arca and BATS anyway. "But some traders are saying they want this, so in the spirit of listening to the customer, we're proceeding with this colocation initiative." Currently, 200 clients colocate some trading applications with Lime at the Savvis data center.