If your firm is still struggling with how to handle today's data deluge, it's not alone. "Although the business intelligence tools market exceeded $120 billion last year, our surveys show only 30 percent of potential users actually use them," notes Rita Sallam, research VP with Gartner. The challenge, she says, is "having self-service BI tools appropriate for business users. A big reason for lack of adoption is that the tools are just too hard to use."
But an evolving category of data discovery solutions may change all that. "Not only are there dramatic improvements to usability, but also increases in speed, more advanced analytics and gains in the ability for business users to integrate data from different sources," Sallam relates.
This new breed of business user-focused solutions -- offered by both established and emerging vendors -- seeks to improve the ability to access, visualize and make decisions based on larger and more diverse data sets, Sallam explains. "We see this category growing three times faster than the overall market for BI," she says.
Rosenblatt Securities is using just such a solution, a visual analytics platform from Seattle-based Tableau Software, to transform the way it does business. Although the New York-based boutique institutional brokerage and investment banker first adopted Tableau soon after the solution debuted in 2006, Rosenblatt's deployment has since undergone a significant metamorphosis, reports Scott Burrill, partner and managing director at the firm.
"Back then, the crisp PDFs it allowed us to create for client reports caused people to think we had a secret weapon," says Burrill. "But never in my wildest dreams did I think Tableau would give us the ability to explore data the way we do now."
Gaining Momentum During the Downturn
According to Burrill, for Rosenblatt, the big data journey really gained momentum while others were struggling to recover from the 2008 financial crisis. Recognizing the potential of extracting knowledge hidden within growing data volumes, he says, the firm became an early adopter of Seattle-based Amazon's cloud infrastructure. More recently, it moved to Amazon Elastic Compute Cloud (Amazon EC2). "Using the cloud saved us millions of dollars in capital expenses," Burrill points out.
With storage secured, an extensive evaluation of database technology followed. Finding available solutions cost prohibitive or too inflexible, Burrill adds, Rosenblatt engineered its own database and built a wrapper for it using Redwood Shores, Calif.-based Oracle's MySQL. "Then we started capturing every tick and quote coming off of the U.S. exchanges," he says.
"However, collecting data wasn't the end," Burrill continues. "Having a robust analytical infrastructure with the ability to retrieve, visualize and report that data was every bit as important."
Rosenblatt sought an analytics solution capable of examining hundreds of thousands of trades. But, Burrill notes, most vendors were too "high touch" when it came to requiring end users to "work with them to develop and customize their product," Burrill says. "We wanted tools that gave us instant productivity."
Ultimately, Rosenblatt decided to further leverage Tableau, pioneering new uses for the platform. "Most off-the-shelf solutions don't integrate data sources well," Burrill says.
"The ability to retrieve information in a timely manner is where many solutions fail," he continues. "We were looking for real-time data retrieval, which led to an entirely different framework of storing the data in-memory and computing it in-memory. That's why we chose to create a proprietary system."
On the hardware side, Rosenblatt built several custom analytics servers with massive amounts of graphic processing units (GPU), relative to CPUs, using Nvidia's compute unified device architecture (CUDA). "This gave us lightning-fast, HFT-like computational power," says Burrill. "We're processing data and doing analytics on the entire U.S. market, in some cases in under 4.2 seconds."