As the social media giant Facebook celebrated it's 10th anniversary, let's take a look at how the company has been impacting Hadoop ecosystem. Listed below are 10 ecosystem projects in which Facebook has done significant open source contributions. Also, towards the end, listed below are some of the other Hadoop ecosystem projects which are not yet open source but occupy a position of prominence inside the company's technical environment.
Open Source Contributions
Developed at Facebook and later open sourced in 2008 for community contribution, Hive continues to be the leader in SQL on Hadoop product category.
Graph processing system built on top of Hadoop, Facebook uses this for social graph analysis and has made significant code additions to framework.
Scalable cross-language services development framework for communication across languages like C++, Python, PHP. Facebook open sourced Thrift in 2007.
Facebook remains one of the main patrons of HBase distributed, scalable, big data store. Implements it in Facebook Messages and Insights.
SQL on Hadoop offering which allows querying data in Hive, HBase, relational databases or even proprietary data stores.
A scheduling framework that helps to schedule MapReduce jobs more efficiently by separating cluster resource management from job coordination.
A server for aggregating log data that's streamed in real time from clients.
A two-node, highly available Namenode with manual failover which wraps the existing Namenode in a Zookeeper layer.
An embeddable, persistent key-value store for real time data queries. RocksDB can be used by applications that need low-latency database access.
Although Cassandra has now minimal usage at Facebook, it was developed to power its Inbox Search feature and open sourced in 2008.
Indicative list with select projects. The entire data enviroment consists of many other cool projects and utilities.
Utility to tail logs/events into Puma/HBase.
Real time analytics platform used by Facebook Insights.
Loads data from sharded DB into HDFS in realtime.
Reliable publish-subscribe system.
Online, in-memory inverted indexing system designed to search social graph. The architecture leverages Hadoop pipeline to convert Hive data results into Unicorn indices.