As the social media giant Facebook celebrated it's 10th anniversary, let's take a look at how the company has been impacting Hadoop ecosystem. Listed below are 10 ecosystem projects in which Facebook has done significant open source contributions. Also, towards the end, listed below are some of the other Hadoop ecosystem projects which are not yet open source but occupy a position of prominence inside the company's technical environment.
Open Source Contributions
Hive
Developed at Facebook and later open sourced in 2008 for community contribution, Hive continues to be the leader in SQL on Hadoop product category.
Giraph
Graph processing system built on top of Hadoop, Facebook uses this for social graph analysis and has made significant code additions to framework.
Thrift
Scalable cross-language services development framework for communication across languages like C++, Python, PHP. Facebook open sourced Thrift in 2007.
HBase
Facebook remains one of the main patrons of HBase distributed, scalable, big data store. Implements it in Facebook Messages and Insights.
Presto DB
SQL on Hadoop offering which allows querying data in Hive, HBase, relational databases or even proprietary data stores.
Corona
A scheduling framework that helps to schedule MapReduce jobs more efficiently by separating cluster resource management from job coordination.
Scribe/Calligraphus
A server for aggregating log data that's streamed in real time from clients.
Avatar Node
A two-node, highly available Namenode with manual failover which wraps the existing Namenode in a Zookeeper layer.
Rocks DB
An embeddable, persistent key-value store for real time data queries. RocksDB can be used by applications that need low-latency database access.
Cassandra
Although Cassandra has now minimal usage at Facebook, it was developed to power its Inbox Search feature and open sourced in 2008.
Facebook Internal
Indicative list with select projects. The entire data enviroment consists of many other cool projects and utilities.
PTail
Utility to tail logs/events into Puma/HBase.
Puma
Real time analytics platform used by Facebook Insights.
Morse
Loads data from sharded DB into HDFS in realtime.
Wormhole
Reliable publish-subscribe system.
Unicorn
Online, in-memory inverted indexing system designed to search social graph. The architecture leverages Hadoop pipeline to convert Hive data results into Unicorn indices.
Comments
Post a Comment