Big Data FAQ: Separating Signal From Noise
Author Phil Simon answers the most frequently asked questions on the impact of big data on small businesses and individuals, the role of data scientists and the continuing importance of human intuition.
What is the role of intuition in the era of big data? Have machines and data supplanted the human mind?
Contrary to what some people believe, intuition is as important as ever. When looking at massive, unprecedented datasets, you need someplace to start. In Too Big to Ignore , I argue that intuition is more important than ever precisely because there's so much data now. We are entering an era in which more and more things can be tested.
Big data has not replaced intuition -- at least not yet; the latter merely complements the former. The relationship between the two is a continuum, not a binary.
A key piece of big data is its reliance on "unstructured" and "semi-structured" data. Can you explain what's going on here?
Roughly 80% of the information generated today is of an unstructured variety. Small data is still very important -- e.g., lists of customers, sales, employees and the like. Think Excel spreadsheets and database tables. However, tweets, blog posts, Facebook likes, YouTube videos, pictures and other forms of unstructured data have become too big to ignore.
Again, big data here serves as a complement to -- not a substitute for -- small data. When used right, big data can reduce uncertainty, not eliminate it. We can know more about previously unknowable things. We can solve previously vexing problems. And finally, there's the Holy Grail: Big data is helping organizations make better predictions and better business decisions.
NEXT: Data visualization is becoming more popular than ever.
... Read full story on InformationWeek
Post a comment to the original version of this story on InformationWeek