Skip to main content

SQL on HBase with Apache Phoenix

By Sachin Ghai
Apache Phoenix is a powerful tool for implementing a relational layer on top of NoSQL HBase database. Low latency query model combined with SQL give the tool a strong edge over HBase API or shell commands. To discover more about the tool and find out how it works, HadoopSphere interacted with James Taylor, founder of Apache Phoenix.

How does Apache Phoenix fit in the crowded SQL on Hadoop space?


Apache Phoenix is specifically meant for accessing HBase data. It's not trying to compete with the Hive and Impalas of the world. We do very specific optimizations for HBase to ensure you get the best possible performance.

What are the complexities that Apache Phoenix abstracts away from the user?


Phoenix abstracts away the same complexities that a RDBMS does when you use SQL versus reading raw bytes from files that represent a page of data on disk. Sure, this is a bit of an exaggeration, but not by a lot. The HBase client APIs are very low level, at the level of byte arrays. There's no support for composite row keys in HBase, for example, so your left bit twiddling the start and stop row in your scans. Performance in HBase changes with each version, so in prior versions you'd get a 25% boost by avoiding the use of the ExplicitColumnTracker.

Above and beyond these lower level details are higher level features such as secondary indexes (now supporting functional indexes), user defined functions (available in 4.4), and just the overall capabilities for querying in SQL (with a parallel execution) versus writing lots and lots of Java code (with a serial execution). Try writing a N-way join with aggregation and union in the HBase client API. And then compare the performance. And then change your data model and see which one takes longer to adapt.
Apache Phoenix architecture

Can you tell us about internal architecture of Apache Phoenix? How does it work?


Apache Phoenix takes your SQL query, compiles it into a series of HBase scans, and orchestrates the running of those scans to produce regular JDBC result sets. The table metadata is stored in an HBase table and versioned, such that snapshot queries over prior versions will automatically use the correct schema. Phoenix has a pretty typical query engine architecture with a parser, compiler, planner, optimizer, and execution engine. We push as much computation as possible into the HBase server which provides plenty of hooks for us to leverage. This helps us achieve our stellar performance.

Taking an example, can you help us understand Apache Phoenix's execution plan?


Take a look at the video from my HBaseCon presentation here  to get a good overview (about 15mins in). Other good references can be found here on Apache Phoenix website.


James Taylor is an architect at salesforce.com in the Big Data Group. He founded the Apache Phoenix project and leads its on-going development efforts. Prior to Salesforce, James worked at BEA Systems on projects such as a federated query processing system and a SQL-based complex event programming platform, and has worked in the computer industry for the past 20+ years at various start-ups. He lives with his wife and two daughters in San Francisco.

Comments

Post a Comment

Popular articles

5 online tools in data visualization playground

By Sachin Ghai
While building up an analytics dashboard, one of the major decision points is regarding the type of charts and graphs that would provide better insight into the data. To avoid a lot of re-work later, it makes sense to try the various chart options during the requirement and design phase. It is probably a well known myth that existing tool options in any product can serve all the user requirements with just minor configuration changes. We all know and realize that code needs to be written to serve each customer’s individual needs.
To that effect, here are 5 tools that could empower your technical and business teams to decide on visualization options during the requirement phase. Listed below are online tools for you to add data and use as playground.
1)      Many Eyes: Many Eyes is a data visualization experiment by IBM Researchandthe IBM Cognos software group. This tool provides option to upload data sets and create visualizations including Scatter Plot, Tree Map, Tag/Word cloud and ge…
6 comments
Read more

Data deduplication tactics with HDFS and MapReduce

By Sachin Ghai
As the amount of data continues to grow exponentially, there has been increased focus on stored data reduction methods. Data compression, single instance store and data deduplication are among the common techniques employed for stored data reduction.
Deduplication often refers to elimination of redundant subfiles (also known as chunks, blocks, or extents). Unlike compression, data is not changed and eliminates storage capacity for identical data. Data deduplication offers significant advantage in terms of reduction in storage, network bandwidth and promises increased scalability.
From a simplistic use case perspective, we can see application in removing duplicates in Call Detail Record (CDR) for a Telecom carrier. Similarly, we may apply the technique to optimize on network traffic carrying the same data packets.
Some of the common methods for data deduplication in storage architecture include hashing, binary comparison and delta differencing. In this post, we focus on how MapReduce and…
2 comments
Read more

In-memory data model with Apache Gora

By Sachin Ghai
Open source in-memory data model and persistence for big data framework Apache Gora™ version 0.3, was released in May 2013. The 0.3 release offers significant improvements and changes to a number of modules including a number of bug fixes. However, what may be of significant interest to the DynamoDB community will be the addition of a gora-dynamodb datastore for mapping and persisting objects to Amazon's DynamoDB. Additionally the release includes various improvements to the gora-core and gora-cassandra modules as well as a new Web Services API implementation which enables users to extend Gora to any cloud storage platform of their choice. This 2-part post provides commentary on all of the above and a whole lot more, expanding to cover where Gora fits in within the NoSQL and Big Data space, the development challenges and features which have been baked into Gora 0.3 and finally what we have on the road map for the 0.4 development drive.
Introducing Apache Gora Although there are var…
4 comments
Read more