Archipelago Adopts Streaming Data Platform

Streambase Systems, Inc., of Lexington, Mass., announced that Archipelago was a customer on Tuesday in conjunction with the product rollout at the Elite DEMO@15 Conference, a launch pad for new technologies.

The all-electronic U.S. stock exchange expects to implement the new StreamBase Processing Engine later this year to process and analyze inbound customer messages, according to Steve Rubinow, chief technology officer at Archipelago.

"It's a scalable solution for trying to keep up with large streams of data. You can process the data, analyze it, do calculations on it and report on it, and then store it in a database," says Rubinow.

"Whether it be quotes, data or messages from our customers or other market data feeds, the volumes of this data are growing at a rapid clip and the demands for seeing the data are near to real time," ArcaEx's CTO explains.

One of the primary reasons for Archipelago adopting the new real-time information-processing architecture is the growth of algorithmic trading where mathematical models, instead of humans, generate buy and sell signals as well as cancel and replace orders within milliseconds.

"In the last few months we've seen a very big increase in the growth of just message traffic. The reason is machines are doing most of the sending," says the CTO.

"The algorithms are plugged in and the proprietary traders and everybody in these specialty shops are sending trading and evaluating models as fast as the exchanges can receive it," continues Rubinow.

ArcaEx has been processing 20,000 messages per second in equities for the last few months, which is more than a double-digit increase.

"That doesn't mean that algorithmic trading is the bad guy," says Rubinow. "It just means that they're taking advantage of the capabilities of the newest systems and they take us to a whole new level of play."

With Archipelago looking to enter the options business through its pending acquisition of the Pacific Exchange, the technology could be applied to options data as well, says Rubinow, noting that options data rates are 45,000 messages per second.

ArcaEx will use Streambase in two ways: to process inbound messages from customers as well as to aggregate and load transaction data into ArcaVision, its data warehouse.

In the first application, the data can be bids and offers, orders and cancel, and replace orders and modified orders coming in at the rate of thousands of messages per second. ArcaEx wants to "understand what's coming at us, look at different ratios and, for our own operational purposes, notify customers that have crossed the threshold of certain trading activity," says Rubinow. For example, if a customer is sending 5,000 orders a second, ArcaEx wants to know if there's pattern to the orders, why the customer is sending that many orders and whether 5,000 orders per second is typical behavior for that customer.

In the second application, ArcaEx plans to use Streambase to load ArcaVision, its Oracle-based data warehouse, with transaction data in real time as fast as possible. ArcaVision contains every transaction that comes through ArcaEx, ranging from basic information that the public can access - such as how many shares traded today - to more specific information that requires a password and ID to access, relative to a customer or listing company. ArcaEx loads information today with a lot of slower software and uses hardware to keep it functioning. "Streambase will help us pre-aggregate the data and then we can load it into the data warehouse and we can do it with less powerful hardware," says Rubinow.

In the future, Arca will replace internal software that it's written. "We won't run those programs. We'll run Streambase to crunch the data," he explains.

However, Archipelago has not implemented Streambase yet. Currently it's running the product in a test model to monitor inbound messages from customers. Rubinow says "that was proving the theory with Streambase," and its planning to make that application production ready.

Archipelago first had conversations with the company in the late summer/early fall time frame, Rubinow recalls. "The amount of commercial software that is available to handle the volume we're handling today is a very small subset," Rubinow says, adding that Michael Stonebraker, the company's founder, "is a very credible person in the industry. We tested it and it worked as we expected it to," says the CTO.

Streambase was founded in 2003 (as Grassy Brook) by database guru Michael Stonebraker and a team of 30 professors and students from M.I.T., Brown University and Brandeis University to address the processing demands of high-performance real-time streaming applications.