Skip to main content

Tools and Techniques for Matrix factorization



Matrix factorization as a technique of Collaborative filtering has been the preferred choice for recommendation systems ever since Netflix million $ competition was held a few years back. Further, with the advent of news personalization, advanced search and user analytics, the concept has gained prominence. In this post, let’s explore the tools and techniques in Hadoop ecosystem supporting Matrix factorization.

For purpose of understanding matrix factorization, we tend to simplify the computation to:
            p x r Matrix B
            r x q Matrix C
for achieving:
A ~= BC
and minimize L(A,B,C) computed over N training points
where, A is a p x q input matrix and L is a loss function.

Among the various algorithmic techniques available, the following are more popular:

-          Alternating Least Squares (ALS): as popularized in Netflix contest
Step 1: Initialize matrix C by assigning the average rating for that movie as
the first row, and small random numbers for the remaining entries.
Step 2: Fix C, Solve B by minimizing the objective function (the sum of
squared errors);
Step 3: Fix B, solve C by minimizing the objective function similarly;
Step 4: Repeat Steps 2 and 3 until a stopping criterion is satisfied.
           
- Expectation Maximization: used in large scale collaborative filtering and demonstrated experimentation on Google News.
The EM algorithm is an efficient iterative procedure to compute the Maximum Likelihood (ML) estimate in the presence of missing or hidden data…
Each iteration of the EM algorithm consists of two processes: The E-step, and the M-step. In the expectation, or E-step, the missing data are estimated
given the observed data and current estimate of the model parameters. This is achieved using the conditional expectation, explaining the choice of terminology. In the M-step, the likelihood function is maximized under the assumption that the missing data are known. The estimate of the missing data from the E-step are used in lieu of the actual missing data.

- Distributed non negative matrix factorization: a re-scaled version of gradient descent as tested on Internet Explorer ‘Suggested Sites’ function. In the distributed technique leveraging NMF, the data is partitioned along the long dimension for parallelism.
Assuming all matrices can be held in shared memory, multiplicative update rules are applied to determine step sizes for each parameter.

- Stochastic Gradient Descent (SGD): a well know technique which tends to compute direction of steepest descent and then takes a step in that direction. Among the variants include:
- Partitioned SGD: distribute without using stratification and run independently and in parallel on partitions
- Pipelined SGD: based on ‘delayed update’ scheme
- Decentralized SGD: computation in decentralized and distributed fashion


Beyond MapReduce as built in Apache Hadoop, there are a few other tools/frameworks/libraries which are available in Hadoop ecosystem for implementing matrix factorization techniques:

- Apache Mahout

Machine Learning library which contains matrix factorization algorithm for collaborative filtering implemented on top of Apache Hadoop using the map/reduce paradigm.




- Apache Giraph

Open source graph processing framework inspired by Google Pregel which follows the bulk synchronous parallel model and launched typically as Hadoop job.


- Ricardo

A scalable platform for deep analytics. Ricardo combines the data management capabilities of Hadoop and Jaql with the statistical functionality provided by R.

Source: Data Intensive Analytics with Hadoop: A Look Inside (refer slide 33 onwards)

There are further advances being made in the various projects in the Apache Hadoop ecosystem. A recent incubator proposal for MRQL project also aims to leverage the BSP model of Apache Hama to compute matrix functions.

Most, if not all, techniques listed above can be effectively run in a distributed environment using MapReduce and Hadoop only also. The same has been strongly argued in a paper by Jimmy Lin. ‘MapReduce only’ approach may suffer from a few limitations like launch overheads and may tend to be categorized batch only. However, it still may score in front of multiple tools architecture in terms of avoiding integration overheads. With a strong focus on low latency products and advances being made in analytic accelerators, there is a possibility we may see more on matrix factorization and collaborative filtering in coming months.

-----------------------------------------------------------------------

top image source: wikipedia

Comments

Popular posts from this blog

Technical deep dive into Apache Tajo

Over the past few months, a new Hadoop based warehouse called Apache Tajo has been making the buzz. We tried to do a technical deep dive in the topic and reached out to PMC chair Hyunsik Choi. Apache Tajo is an Apache top level project since March 2014 and supports SQL standards. It has a powerful distributed processing architecture which is not based on MapReduce. To get more sense on Tajo's claims for providing a distributed, fault-tolerant, low-latency and high throughput SQL engine, we asked a few questions to Choi. This Q&A is published in a two part article series on hadoopsphere.com. Read below to get a better idea on Apache Tajo.
How does Apache Tajo work? As you may already know, Tajo does not use MapReduce and has its own distributed processing framework which is flexible and specialized to relational processing. Since its storage manager is pluggable, Tajo can access data sets stored in various storages , such as HDFS, Amazon S3, Openstack Swift or local file system. …

In-memory data model with Apache Gora

Open source in-memory data model and persistence for big data framework Apache Gora™ version 0.3, was released in May 2013. The 0.3 release offers significant improvements and changes to a number of modules including a number of bug fixes. However, what may be of significant interest to the DynamoDB community will be the addition of a gora-dynamodb datastore for mapping and persisting objects to Amazon's DynamoDB. Additionally the release includes various improvements to the gora-core and gora-cassandra modules as well as a new Web Services API implementation which enables users to extend Gora to any cloud storage platform of their choice. This 2-part post provides commentary on all of the above and a whole lot more, expanding to cover where Gora fits in within the NoSQL and Big Data space, the development challenges and features which have been baked into Gora 0.3 and finally what we have on the road map for the 0.4 development drive.
Introducing Apache Gora Although there are var…

Data deduplication tactics with HDFS and MapReduce

As the amount of data continues to grow exponentially, there has been increased focus on stored data reduction methods. Data compression, single instance store and data deduplication are among the common techniques employed for stored data reduction.
Deduplication often refers to elimination of redundant subfiles (also known as chunks, blocks, or extents). Unlike compression, data is not changed and eliminates storage capacity for identical data. Data deduplication offers significant advantage in terms of reduction in storage, network bandwidth and promises increased scalability.
From a simplistic use case perspective, we can see application in removing duplicates in Call Detail Record (CDR) for a Telecom carrier. Similarly, we may apply the technique to optimize on network traffic carrying the same data packets.
Some of the common methods for data deduplication in storage architecture include hashing, binary comparison and delta differencing. In this post, we focus on how MapReduce and…