Skip to main content

bon appétit: hungry for easier Hadoop coordination?

Apache ZooKeeper is an invaluable tool for distributed coordination. However, when I began working for Netflix, I found a lot of interest in ZooKeeper but not a lot of experience. A few trials had been done but none of them made it out of the testing sandbox. For that reason, Curator was initially conceived as a way to make ZooKeeper easier to use for non-experts. The original versions of what would become Curator consisted of a small library meant for internal use, but it quickly became clear that it would be useful to others outside Netflix.

Over a period of time, ZooKeeper has become very popular at Netflix. Via Curator, some of the uses are:

- InterProcessMutex for ensuring unique values in various sequence ID generators
- Cassandra Backups
- LeaderSelector for various housekeeping tasks in Chukwa collector
- InterprocessSemaphore to manage third party services that allow only a limited number of concurrent users
- Managing various caches
- Much more all the time

Zookeeper challenges

Here’s a ZooKeeper pop quiz – what’s wrong with this code?


Answer: if the call to client.create() happens before the ZooKeeper client successfully connects to the server, the create() method will throw an exception.


New users of ZooKeeper are surprised to learn that a significant amount of connection management must be done manually. For example, when the ZooKeeper client connects to the ensemble it must negotiate a new session and so on. This takes some time. If you use a ZooKeeper client API before the connection process is complete, ZooKeeper will throw an exception. These types of exceptions are referred to as “recoverable” errors.


Alternate options

Prior to Curator, options for working with Curator were limited to the raw ZooKeeper class provided in the distribution and the open source ZKClient. With either of these options, most of the work of writing recipes would still have to be done manually. The ZooKeeper distribution includes some pre-built recipes but there is no generalized framework for creating new recipes and the provided implementations are missing some known best practices.

Today, it’s a much brighter picture for using ZooKeeper. If you are using a JVM-based language, Curator is the best way to connect to ZooKeeper. For Python, there’s Kazoo, etc.

Curator architecture

Curator is a platform for writing ZooKeeper Recipes. It’s built as a stack of three modules that make writing recipes much simpler – no need to worry about connection management, edge case, etc.

- At the bottom of the stack is Curator Client. This module is a wrapper around the native ZooKeeper class that adds connection management and a framework for retries.

- The next layer is Curator Framework. It takes advantage of Curator Client’s connection management and retry mechanism. It adds a nice fluent API for all ZooKeeper interactions and performs them inside of a retry loop. The beauty here is that when using ZooKeeper via the Curator Framework you no longer need to be concerned with managing the ZooKeeper connection or handling “recoverable” errors.

- On top of the Curator Framework, the Curator Recipes module provides implementations for all of the recipes listed on the ZooKeeper recipes page plus many additional recipes. Some of the most popular recipes are: LeaderSelector (a leader election recipe), InterProcessMutex (a distributed lock recipe) and PathChildrenCache (a local, updating cache of a ZooKeeper ZNode’s children).

Curator Recipes also serve as examples of how to use the Curator Framework for writing new recipes. However, writing ZooKeeper recipes, even with Curator, is still a challenge. I’m fond of saying that “friends don’t let friends write ZooKeeper recipes”. Always see if there’s a Curator recipe that meets your needs or, if necessary, build on an existing recipe.

Hungry for more?

Curator usage continues to grow and is now in the Apache Incubator. We invite you to participate and contribute via curator.incubator.apache.org.

About the author:

 Jordan Zimmerman is a Senior Software Engineer on the Platform Team at Netflix. 
He has contributed to open source software throughout his career. At Netflix he’s opened Curator, Exhibitor and Governator and maintains the Netflix Open Source hub at http://netflix.github.com In his spare time he plays drums (bebop and big band), listens to opera (mostly Puccini), drinks fine whisky (Laphroaig and Buffalo Creek) and watches soccer (ManU) and Epic Meal Time.



Comments

  1. Hi Jordan, we had the pleasure of hearing Adrian mentioning Curator within his talk @C*Summit a fortnight ago in SFO and great to see that i's gathering momentum within the Incubator.
    Can you please tell me/us more about Curator and how it manages/schedules your Cassandra backup. I was knocked for six when I saw the infrastructure you guys work with @Netflix so I am interested to know about this. Thanks for the post. Great work.

    ReplyDelete
  2. Jordan ZimmermanJuly 3, 2013 at 10:12 AM

    I don't have details on the Cassandra backup use of ZooKeeper. I'll ask around and try to get back to you.

    ReplyDelete

Post a Comment

Popular posts from this blog

Low latency SQL querying on HBase

HBase has emerged as one of the most popular NoSQL database offering distributed, versioned, non-relational tables hosted on commodity hardware. However, with a large set of users coming from a relational SQL world, it made sense to bring the SQL back in this NoSQL. With Apache Phoenix, database professionals get a convenient way to query HBase through SQL in a fast and efficient manner. Continuing our discussion with James Taylor, the founder of Apache Phoenix, we focus on the functional aspects of Phoenix in this second part of interaction.
Although Apache Phoenix started off with distinct low latency advantage, have the other options like Hive/Impala (integrated with HBase) caught up in terms of performance?
No, these other tools such as Hive and Impala have not invested in improving performance against HBase data, so if anything, Phoenix's advantage has only gotten bigger as our performance improves.  See this link for comparison of Apache Phoenix with Apache Hive and Cloudera Im…

Data deduplication tactics with HDFS and MapReduce

As the amount of data continues to grow exponentially, there has been increased focus on stored data reduction methods. Data compression, single instance store and data deduplication are among the common techniques employed for stored data reduction.
Deduplication often refers to elimination of redundant subfiles (also known as chunks, blocks, or extents). Unlike compression, data is not changed and eliminates storage capacity for identical data. Data deduplication offers significant advantage in terms of reduction in storage, network bandwidth and promises increased scalability.
From a simplistic use case perspective, we can see application in removing duplicates in Call Detail Record (CDR) for a Telecom carrier. Similarly, we may apply the technique to optimize on network traffic carrying the same data packets.
Some of the common methods for data deduplication in storage architecture include hashing, binary comparison and delta differencing. In this post, we focus on how MapReduce and…

5 online tools in data visualization playground

While building up an analytics dashboard, one of the major decision points is regarding the type of charts and graphs that would provide better insight into the data. To avoid a lot of re-work later, it makes sense to try the various chart options during the requirement and design phase. It is probably a well known myth that existing tool options in any product can serve all the user requirements with just minor configuration changes. We all know and realize that code needs to be written to serve each customer’s individual needs.
To that effect, here are 5 tools that could empower your technical and business teams to decide on visualization options during the requirement phase. Listed below are online tools for you to add data and use as playground.
1)      Many Eyes: Many Eyes is a data visualization experiment by IBM Researchandthe IBM Cognos software group. This tool provides option to upload data sets and create visualizations including Scatter Plot, Tree Map, Tag/Word cloud and ge…