Data Management

03:33 PM
Connect Directly
Google+
Twitter
Facebook
RSS
E-Mail
50%
50%

Big Data Overload is Reshaping the Technology Market

More and more organizations want to bring external data into their applications, multiplying the complexity of the equation.

Today we are clearly seeing a desire for businesses to access more data, which coincides with increased availability of data. New kinds of data, and a lot of the same. Data generated by machines, sensors, geological locations of individuals, etc. All of that transfers to a pool that needs to be pumped out, cleaned, and put together in a form digestible to a business and end user. There's a broad range of latencies and velocities of data, and preferably we all want it within an instant.

Traditionally, you have a limited set of data that requires some heavy duty processing, and by limited we don't necessarily mean smaller. Financial organizations have a a huge volume, but data was clearly identified and owned by the organization, explains Yves de Montcheuil, VP of marketing at Talend, a business process integration solutions provider.

"What we're seeing is a requirement to extend this use to more data types, so that's where big data comes into play. Even data within an organization that wasn't used, for example, the data that is generated by sensors on manufacturing equipment… By combining all this internal data, we can access more external data -- social data, sentiment analysis, predictive data. It's publicly available but it's about individuals, and you have this whole thing about open data, sensor data, spending by the government, data that is made more and more openly available due to legal requirements." Increasingly, organizations want to bring this external data into the equation.

But where to put it all? Organizations are increasingly relying on data warehousing companies, they are providing the space in which we dump the data we've aggregated and cleansed. At the Teradata PARTNERS conference in October it was announced that Teradata, the global leader in data warehousing and data analytics, launched their cloud and offering data warehousing as a service (DWaaS), managed by Teradata consulting professionals. Talend is the the only data integration service given the contract in advance, and to date the default option for uploading data into the DWaaS for Teradata customers.

"We've been partners with them forever," adds Montcheuil. "Their goal is for customers who don't want to deploy Teradata on premises to have the ability to farm it out to Teradata and manage the data warehouse for them. This does not alleviate the need to bring into the warehouse all the data needed for analytics, so the customer will have to provide to Teradata access to data sources."

Should firms be concerned about farming out their data? Perhaps, but DWaaS will take time to deploy, and in the end it won't be trust or security issues that hold back adoption. Organizations will go through obsolescent cycles of old systems before replacing them, testing the waters with ad hoc projects, helping to create proof of concepts and ramp up the technologies they need.

Big data is becoming more progressive, finding ways to more mainstream IT. People are no longer experimenting, they are full stream ahead and deriving value. "Big data is becoming the center of IT," explains Montcheuil, "and it's becoming the center on a ubiquitous platform. It's no longer the application at the center, instead data applications will plug into this data system." Becca Lipman is Senior Editor for Wall Street & Technology. She writes in-depth news articles with a focus on big data and compliance in the capital markets. She regularly meets with information technology leaders and innovators and writes about cloud computing, datacenters, ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
Jonathan_Camhi
50%
50%
Jonathan_Camhi,
User Rank: Author
11/14/2013 | 7:25:50 PM
re: Big Data Overload is Reshaping the Technology Market
With all of the focus on security when ti comes to cloud providers they can't afford not to devote a great deal of time and resources to those concerns. Their whole business goes out the window if they fail to secure their clients' data.
Jonathan_Camhi
50%
50%
Jonathan_Camhi,
User Rank: Author
11/14/2013 | 7:23:50 PM
re: Big Data Overload is Reshaping the Technology Market
Real time is definitely the holy grail for data and analytics right now, and I'm sure a lot of data and analytics providers are working towards that. Some have already reached that point, but it's still a nascent market. A ton of potential there though for companies that prove successful at it.
Byurcan
50%
50%
Byurcan,
User Rank: Author
11/14/2013 | 4:14:43 PM
re: Big Data Overload is Reshaping the Technology Market
That's a good point, plus there's various industry certificates you can get as a cloud provider that I imagine would enhance your credibility as well.
IvySchmerken
50%
50%
IvySchmerken,
User Rank: Author
11/14/2013 | 3:20:11 PM
re: Big Data Overload is Reshaping the Technology Market
A new crop of social data aggregators and analytics vendors have cropped up to clean the data and create content sets. One example is Connotate which"collects, delivers and transforms," according to its web site. It creates different packaged solutions Gă÷ news and content aggregation, compliance and risk management, and one specific to financial data aggregation. So in real time, it's gathering political events, corporate actions, financial filings and market moving information.
Greg MacSweeney
50%
50%
Greg MacSweeney,
User Rank: Apprentice
11/13/2013 | 9:13:02 PM
re: Big Data Overload is Reshaping the Technology Market
Most definitely. Social data is very "rough" and unrefined. A lot of work has to be done to decipher any sort of meaning from that type of data right now. I'm sure that we will see some new tools come to the market in the next few months that can help refine the data so it will be less prone to false positives, or biases.
PERACTON
50%
50%
PERACTON,
User Rank: Apprentice
11/13/2013 | 6:18:01 PM
re: Big Data Overload is Reshaping the Technology Market
Let's consider financial industry that has one of the best quality data: even with hard financial data (that is checked and validated by third party financial data providers) investors and traders struggle to separate real signals from noise, due to lack of proper analytic tools.

By contrast, Social Data is much less verifiable and much more prone to biases and social manipulations. For now we can look at it as just entertainment data. Some R&D effort must go into it, in a more inter-disciplinary form (mathematics/IT/sociology/psychology) and for sure, some interesting results could be obtained.
Becca L
50%
50%
Becca L,
User Rank: Author
11/5/2013 | 11:50:07 PM
re: Big Data Overload is Reshaping the Technology Market
What a complicated question, and even more complicated answer. Data hoarders could be the winners here. Imagine discarding data because there was no known use for it, only to find one the next week. (reminds me of throwing out a sock because I can't find the pair.. wait a week and there it is). Insurance companies are storing telematics data in the cloud so they can study it over time, getting a better idea of how to charge rates using info from all the sensor and geographic data in addition to the usual stuff. What a complex algorithm that I'm sure would utilize every bit of info, But when you think of all the cars, all the sensors - it's madness!
Nathan Golia
50%
50%
Nathan Golia,
User Rank: Author
11/5/2013 | 8:45:25 PM
re: Big Data Overload is Reshaping the Technology Market
In addition, some insurance CIOs have told me they prefer the cloud because the cloud provider's expertise is in security and data management, and they don't understaff those functions.
Greg MacSweeney
50%
50%
Greg MacSweeney,
User Rank: Apprentice
11/5/2013 | 12:25:55 PM
re: Big Data Overload is Reshaping the Technology Market
Nothing is 100% safe and secure nowadays, and if a provider tells you so, I would be wary. That said, with the proper safeguards in place, there is no reason why mining social data should be any more dangerous than other online apps.
IvySchmerken
50%
50%
IvySchmerken,
User Rank: Author
11/4/2013 | 2:11:57 PM
re: Big Data Overload is Reshaping the Technology Market
I'm not a data scientist, but social data that is predictive would seem to be more valuable. Customer comments in text form that can be quantified and turned into sentiment analysis are of interest to investment managers.
Page 1 / 2   >   >>
Register for Wall Street & Technology Newsletters
White Papers
Current Issue
Wall Street & Technology - Elite 8, October 2014
The in-depth profiles of this year's Elite 8 honorees focus on leadership, talent recruitment, big data, analytics, mobile, and more.
Video
5 Things to Look For Before Accepting Terms & Conditions
5 Things to Look For Before Accepting Terms & Conditions
Is your corporate data at risk? Before uploading sensitive information to cloud services be sure to review these terms.