Trading Technology

02:59 PM
50%
50%

Is HCatalog The REST Of The Hadoop Capital Markets Story?

The necessary prerequisites for Hadoop to play a key role in the enterprise data architecture of Capital Markets firms are quickly coming into place.

The necessary prerequisites for Hadoop to play a key role in the enterprise data architecture of capital markets firms are quickly coming into place.

Jennifer L. Costley, Ashokan Advisors
Jennifer L. Costley, Ashokan Advisors

In a previous article, I discussed how new resource management features will allow multiple processing modes -- batch, interactive, online and streaming -- to run simultaneously with defined quality of service. Here, I address another key element in making Hadoop enterprise-ready, HCatalog.

A key component of Apache Hive, HCatalog is the metadata and table management system for the Hadoop platform which stores and shares information about data structure. Critically, HCatalog also enables sharing of data structure with external systems including traditional data management tools. As described by Jim Walker, director of product marketing at Hortonworks, "It is the glue that enables these systems to interact effectively and efficiently and is a key component in helping Hadoop fit into the enterprise."

[Is IT Irrelevant? William Murphy, Chief Technology Officer, Head of Blackstone Innovations and Infrastructure, is a keynote speaker at Interop, taking place in NYC September 30 through October 4.]

Hive, as the defacto SQL interface for Hadoop, provides a relational view through SQL-like language to data within Hadoop. HCatalog publishes the Hive interface as an abstraction, as well as a REST interface.

HCatalog includes: -- A shared schema and data type mechanism -- A table abstraction -- Interoperability across data processing tools in the Hadoop ecosystem such as Pig, Map Reduce, and Hive. -- A REST interface to allow language-independent access to Hive's metadata.

Data technology leaders are starting to use HCatalog to integrate Hadoop into their overall data architectures. Teradata recently announced their Teradata SQL-H product which leverages HCatalog to provide direct access to Hadoop data through standard ANSI SQL and enables that data to be run directly in-memory on Teradata.

Not bad for an Open Source project which was conceived (way back in 2011) not as an enterprise-enabler but a way to avoid having to contact the Hadoop data-producer to ask them where they write their data, what format it is in, and what its schema is.

About The Author: Jennifer L. Costley, Ph.D. is a scientifically-trained technologist with broad multidisciplinary experience in enterprise architecture, software development, line management and infrastructure operations, primarily (although not exclusively) in capital markets. She is also a non-profit board leader recognized for talent in building strong governance and process. Her current focus is in helping companies, organizations and individuals with opportunities related to data, analysis and sustainability. She can be reached at www.ashokanadvisors.com.

Jennifer L. Costley, Ph.D. is a scientifically-trained technologist with broad multidisciplinary experience in enterprise architecture, software development, line management and infrastructure operations, primarily (although not exclusively) in capital markets. She is also a ... View Full Bio
Comment  | 
Print  | 
More Insights
More Commentary
Could Intel Lose Data Center Market Share to ARM Chips?
ARM chips could be an alternative for certain purposes in the datacenter, but many questions have to be answered before they pose a threat to Intel's market dominance.
Cost to Trade: Hey, Banks, Itís Time to Face the Music
Why is calculating the cost to trade so difficult for banks? The answer is as complex as the calculations themselves.
M&A Activity Will Continue to Grow in 2015
Data shows that the M&A market continues to improve, and forecasts indicate deal making will be healthy in 2015.
Chief Data Officers: Organization Strategy & Cultural Change
Chief data officers are new to the financial services C-suite, but they are facing a number of challenges, including the need for new data governance and execution strategies, staffing, and new organizational structures to enable cultural change.
New York FinTech Innovation Lab Calls for New Entrepreneurial Applicants
Wells Fargo joins 14 other major financial institutions providing mentoring and guidance to the six chosen startups.
Register for Wall Street & Technology Newsletters
White Papers
Current Issue
Wall Street & Technology - Elite 8, October 2014
The in-depth profiles of this year's Elite 8 honorees focus on leadership, talent recruitment, big data, analytics, mobile, and more.
Video
Exclusive: Inside the GETCO Execution Services Trading Floor
Exclusive: Inside the GETCO Execution Services Trading Floor
Advanced Trading takes you on an exclusive tour of the New York trading floor of GETCO Execution Services, the solutions arm of GETCO.