Let’s understand Hadoop, MapReduce with a few fun trivia analogies. They may not match perfectly with scientific definitions – but let the purists don’t fret and sweat. This is to give a simplistic idea to those who are still uninitiated in Hadoop. Enjoy and learn with this weekend read.
“As an analogy, you can think of map and reduce tasks as the way a census was conducted in Roman times, where the census bureau would dispatch its people to each city in the empire. Each census taker in each city would be tasked to count the number of people in that city and then return their results to the capital city. There, the results from each city would be reduced to a single count (sum of all cities) to determine the overall population of the empire. This mapping of people to cities, in parallel, and then combining the results (reducing) is much more efficient than sending a single person to count every person in the empire in a serial fashion.”
- Just like HDFS slices and distributes the chunk of data to individual nodes, each household submits the lunchbox to a Dabbawala.
- All the lunchboxes are collected at the common place for tagging them and to put them into carriages with unique codes. This is the job of the Mapper!
- Based on the code, carriages that need to go to the common destination are sorted and on-boarded to the respective trains. This is called Shuffle and Sort phase in MapReduce.
- At each railway station, the Dabbawala picks up the carriage and delivers each box in that to respective customers. This is the Reduce phase."