Skip to main content

Tools and Techniques for Matrix factorization

Matrix factorization as a technique of Collaborative filtering has been the preferred choice for recommendation systems ever since Netflix million $ competition was held a few years back. Further, with the advent of news personalization, advanced search and user analytics, the concept has gained prominence. In this post, let’s explore the tools and techniques in Hadoop ecosystem supporting Matrix factorization.

For purpose of understanding matrix factorization, we tend to simplify the computation to:
            p x r Matrix B
            r x q Matrix C
for achieving:
A ~= BC
and minimize L(A,B,C) computed over N training points
where, A is a p x q input matrix and L is a loss function.

Among the various algorithmic techniques available, the following are more popular:

-          Alternating Least Squares (ALS): as popularized in Netflix contest
Step 1: Initialize matrix C by assigning the average rating for that movie as
the first row, and small random numbers for the remaining entries.
Step 2: Fix C, Solve B by minimizing the objective function (the sum of
squared errors);
Step 3: Fix B, solve C by minimizing the objective function similarly;
Step 4: Repeat Steps 2 and 3 until a stopping criterion is satisfied.
- Expectation Maximization: used in large scale collaborative filtering and demonstrated experimentation on Google News.
The EM algorithm is an efficient iterative procedure to compute the Maximum Likelihood (ML) estimate in the presence of missing or hidden data…
Each iteration of the EM algorithm consists of two processes: The E-step, and the M-step. In the expectation, or E-step, the missing data are estimated
given the observed data and current estimate of the model parameters. This is achieved using the conditional expectation, explaining the choice of terminology. In the M-step, the likelihood function is maximized under the assumption that the missing data are known. The estimate of the missing data from the E-step are used in lieu of the actual missing data.

- Distributed non negative matrix factorization: a re-scaled version of gradient descent as tested on Internet Explorer ‘Suggested Sites’ function. In the distributed technique leveraging NMF, the data is partitioned along the long dimension for parallelism.
Assuming all matrices can be held in shared memory, multiplicative update rules are applied to determine step sizes for each parameter.

- Stochastic Gradient Descent (SGD): a well know technique which tends to compute direction of steepest descent and then takes a step in that direction. Among the variants include:
- Partitioned SGD: distribute without using stratification and run independently and in parallel on partitions
- Pipelined SGD: based on ‘delayed update’ scheme
- Decentralized SGD: computation in decentralized and distributed fashion

Beyond MapReduce as built in Apache Hadoop, there are a few other tools/frameworks/libraries which are available in Hadoop ecosystem for implementing matrix factorization techniques:

- Apache Mahout

Machine Learning library which contains matrix factorization algorithm for collaborative filtering implemented on top of Apache Hadoop using the map/reduce paradigm.

- Apache Giraph

Open source graph processing framework inspired by Google Pregel which follows the bulk synchronous parallel model and launched typically as Hadoop job.

- Ricardo

A scalable platform for deep analytics. Ricardo combines the data management capabilities of Hadoop and Jaql with the statistical functionality provided by R.

Source: Data Intensive Analytics with Hadoop: A Look Inside (refer slide 33 onwards)

There are further advances being made in the various projects in the Apache Hadoop ecosystem. A recent incubator proposal for MRQL project also aims to leverage the BSP model of Apache Hama to compute matrix functions.

Most, if not all, techniques listed above can be effectively run in a distributed environment using MapReduce and Hadoop only also. The same has been strongly argued in a paper by Jimmy Lin. ‘MapReduce only’ approach may suffer from a few limitations like launch overheads and may tend to be categorized batch only. However, it still may score in front of multiple tools architecture in terms of avoiding integration overheads. With a strong focus on low latency products and advances being made in analytic accelerators, there is a possibility we may see more on matrix factorization and collaborative filtering in coming months.


top image source: wikipedia


Popular articles

5 online tools in data visualization playground

While building up an analytics dashboard, one of the major decision points is regarding the type of charts and graphs that would provide better insight into the data. To avoid a lot of re-work later, it makes sense to try the various chart options during the requirement and design phase. It is probably a well known myth that existing tool options in any product can serve all the user requirements with just minor configuration changes. We all know and realize that code needs to be written to serve each customer’s individual needs. To that effect, here are 5 tools that could empower your technical and business teams to decide on visualization options during the requirement phase. Listed below are online tools for you to add data and use as playground. 1)      Many Eyes : Many Eyes is a data visualization experiment by IBM Research and the IBM Cognos software group. This tool provides option to upload data sets and create visualizations including Scatter Plot, Tree Ma

Data deduplication tactics with HDFS and MapReduce

As the amount of data continues to grow exponentially, there has been increased focus on stored data reduction methods. Data compression, single instance store and data deduplication are among the common techniques employed for stored data reduction. Deduplication often refers to elimination of redundant subfiles (also known as chunks, blocks, or extents). Unlike compression, data is not changed and eliminates storage capacity for identical data. Data deduplication offers significant advantage in terms of reduction in storage, network bandwidth and promises increased scalability. From a simplistic use case perspective, we can see application in removing duplicates in Call Detail Record (CDR) for a Telecom carrier. Similarly, we may apply the technique to optimize on network traffic carrying the same data packets. Some of the common methods for data deduplication in storage architecture include hashing, binary comparison and delta differencing. In this post, we focus o

In-memory data model with Apache Gora

Open source in-memory data model and persistence for big data framework Apache Gora™ version 0.3, was released in May 2013. The 0.3 release offers significant improvements and changes to a number of modules including a number of bug fixes. However, what may be of significant interest to the DynamoDB community will be the addition of a gora-dynamodb datastore for mapping and persisting objects to Amazon's DynamoDB . Additionally the release includes various improvements to the gora-core and gora-cassandra modules as well as a new Web Services API implementation which enables users to extend Gora to any cloud storage platform of their choice. This 2-part post provides commentary on all of the above and a whole lot more, expanding to cover where Gora fits in within the NoSQL and Big Data space, the development challenges and features which have been baked into Gora 0.3 and finally what we have on the road map for the 0.4 development drive. Introducing Apache Gora Although