Data Management

02:11 PM
Connect Directly
Facebook
Google+
Twitter
RSS
E-Mail
50%
50%

Check the Time: Majority of Firms have Time Synchronization Errors

Financial firms wrongly believe the timing on trade and reference data. Increased awareness has spurred demand for accurate solutions.

Automated trading and historical analysis depends on knowing the time of events and the order in which they occur. With inaccurate time stamp many enterprises are at risk of unsuccessful trading strategies, even falling out of compliance. Unfortunately, the problem is all too common, and many firms would be surprised at the extent of the damage.

Victor Yodaiken, is CEO of FSMLabs, an operational software firm that offers the TimeKeeper, a hardware or software tool to track multiple time sources, detect and attack bad data sources, and generate automated failover. He says with the standard tools out there are generally a couple milliseconds off. In financial services firms are on a 10gig or 40gig network, but most devices still being sold are 100 megabits per second, causing data to slow, and results in a lot of errors, he explains.

"Traditional providers have not kept up with the technology, link management, or support for the network speed," he said in an interview. "Even people who are not on the cutting edge want to trade at a higher rate, and when you try trading at a higher range using machines all over the world, time becomes critical."

Essentially, when computers ask the operating systems for the time it needs to be close to synchronized with the government run official time which comes from a relatively weak, albeit precise, GPS satellite signal that isn't always accessible. The next challenge is getting that signal to the application code across hundreds and thousands of computers around the world.

"It's a classic computer science problem, it seems trivial, but getting it to work is really difficult," says Yodaiken, "Currently, the state of the art out there is very limited. People have inherited timing infrastructure that was built up without much planning or thought, and the large majority have inadequate infrastructure. The situation is usually terrible and much worse than they understand it to be."

Even with that connection accomplished, there are common scenarios that impact the accuracy and distribution of time. Deliberate sabotage (spoofing), an unplugged cable, or even overcast weather that disrupts the satellite's low power signal. Equipment can break, routers can be disconfigured, unnoticed for extended periods. Imagine a network device, somewhere in basement, with hundreds of feet of cable to an antenna on the roof. A lot of physical things can go wrong.

It's difficult to detect these errors. According to Yodaiken people set systems up and believe whatever it tells them. "They say to us things like, 'it says it's within 12 microseconds,' but it's hard to test that and know what it really means." In one case, after setting up their TimeKeeper application, a firm came back and said 'your software doesn't work' because it seems to be going haywire, "But we looked and saw the time feed they relied on was wrong. Every once in a while it would jump forward a couple hours [HOURS!]. Nobody had noticed."

In another example, "We were in a big company and they were complaining about time problems. It turns out IT was telling them they had a high quality time source, but our software showed an antenna wasn't working, also that it was sending time from NY to Holland every now and then… A lot of things can go wrong."

Another common problem, Yodaiken says, is that people lock down networks and say we only take this kind of information packets for security reasons, and they don't know about time particle packets. When they try to get feeds from data centers, they'll find routers have been misconfigured to not carry protocol properly.

False Correlations

Algorithms are largely built based on trends and correlations of observed events. Developers are looking for relationships that suggest A leads to B leads to C. But if the software is misaligned, and time the things are happening is not right, you'll miss real correlations and get fake ones, throwing off the entire algorithm.

Having accurate time stamps is also handy with regulators. "Forensic records can show customers and regulators that they did the right thing. If timing is off, it may look ilk you're front running but you're not," explains Yodaiken. "People on the up and up are very concerned with being able to preserve a high quality audit trail of time. We can give them multiple time sources, which nobody else can do, and we can show GPS time that agreed with this, and we show we did due diligence and kept the time as accurately as possible. Whereas, with other technology you don't really know. 'My software did this, but where did the time come from? Was it a hack? I don't know. Did anything go wrong? I don't know?'"

In Search of Solutions

For an industry that lives and breathes the mantra 'time is money,' the failover is increasingly weighing on the industry.

"They know there's an issue and want to understand it." They see their systems are inadequate, extremely delicate, with no records, and no trace of what's going on. "With all this technology, you can get silent failures very easily - we've had customers discover using this technology that they had a place offline and nobody had noticed. Systems were just drifting. We see people who think they are getting high precision when they're not. We see some patchy distribution times, - not distributing to everyone who needs it. I will say that there are a lot of companies who don't know how badly their data is."

Yodaiken reports a large increase in inquiries over the last two years, and perhaps a 50% increase in sales the last year for the time stamping technology. "There's been a phenomenal pickup, and it's much wider. We used to only get high frequency traders and other technologically advanced people - and now we're getting inquiries on the business level… Due to regulation people are much more worried," he adds. "They want to make sure components are solid." Becca Lipman is Senior Editor for Wall Street & Technology. She writes in-depth news articles with a focus on big data and compliance in the capital markets. She regularly meets with information technology leaders and innovators and writes about cloud computing, datacenters, ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Greg MacSweeney
50%
50%
Greg MacSweeney,
User Rank: Apprentice
10/17/2013 | 11:55:38 AM
re: Check the Time: Majority of Firms have Time Synchronization Errors
No radioactive clocks, unfortunately. But a bunch of them had radioactive complex synthetic financial products a few years back. ;) And that didn't end well.
Nathan Golia
50%
50%
Nathan Golia,
User Rank: Author
10/17/2013 | 12:26:30 AM
re: Check the Time: Majority of Firms have Time Synchronization Errors
Kind of crazy to think about how fungible time can be considering the importance of these transactions. I assumed that big financial institutions had their own radioactive clocks or something.
Becca L
50%
50%
Becca L,
User Rank: Author
10/16/2013 | 11:59:46 PM
re: Check the Time: Majority of Firms have Time Synchronization Errors
Ha! Well, it's not exactly time traveling. More like if you change the time on your phone and wonder why your alarm isn't going off as usual and your e-mails and texts are out of order. Except with much much bigger consequences and confusion than being late for work.
Becca L
50%
50%
Becca L,
User Rank: Author
10/16/2013 | 11:58:01 PM
re: Check the Time: Majority of Firms have Time Synchronization Errors
I was shocked to hear that too! It's so interesting that something we all take for granted - time - is so subject to failure.
NJ_trader
50%
50%
NJ_trader,
User Rank: Apprentice
10/16/2013 | 9:06:35 PM
re: Check the Time: Majority of Firms have Time Synchronization Errors
Wall Street has accidentally invented algos that can time travel? how cool! It's only micro-jumps in the time-space continuum, but Marty McFly would be happy! JK
Zarna Patel
50%
50%
Zarna Patel,
User Rank: Author
10/16/2013 | 6:16:42 PM
re: Check the Time: Majority of Firms have Time Synchronization Errors
Whoa! Didn't expect to read that firms have time feeds that jump hours. Actually, I didn't expect to read that firms had any problem with setting the time!
Register for Wall Street & Technology Newsletters
White Papers
Current Issue
Wall Street & Technology - July 2014
In addition to regular audits, the SEC will start to scrutinize the cyber-security preparedness of market participants.
Video
5 Things to Look For Before Accepting Terms & Conditions
5 Things to Look For Before Accepting Terms & Conditions
Is your corporate data at risk? Before uploading sensitive information to cloud services be sure to review these terms.