Recently, the Apache Incubator accepted a new query engine for
Hadoop and Hama, called MRQL (pronounced miracle), which was initially
developed in 2011 by Leonidas Fegaras.
MRQL (MapReduce Query Language) is a
query processing and optimization system for large-scale, distributed data
analysis, built on top of Apache Hadoop and Hama. MRQL has some overlapping
functionality with Hive, Impala and Drill, but one major difference is that it
can capture many complex data analysis algorithms that can not be done easily
in those systems in declarative form. So, complex data analysis tasks, such as
PageRank, k-means clustering, and matrix multiplication and factorization, can
be expressed as short SQL-like queries, while the MRQL system is able to
evaluate these queries efficiently.
Another
difference from these systems is that the MRQL system can run these queries in
BSP (Bulk Synchronous Parallel) mode, in addition to the MapReduce mode. With BSP mode, it achieves lower
latency and higher speed. According to MRQL team, “In near future, MRQL will also be able to
process very large data effectively fast without memory limitation and
significant performance degradation in the BSP mode”.
As a simple
example, the MRQL query in Figure 1 calculates the k-means clustering
algorithm.
![]() |
Figure 1. K-means Clustering Expressed as an MRQL Query |
Figure 2
shows the results of evaluating the K-means query using MR and BSP modes for
limit (number of iterations) 10. We can see that the BSP evaluation outperforms
the MR evaluation by an order of magnitude.
MRQL team also has plans to support additional distributed processing frameworks, such
as Spark and OpenMPI in the future. Currently, a number of researchers and
developers from various organizations, such as UT Arlington, Oracle, and
Cloudera, are involved in the MRQL project. They are looking forward to your
contributions.
You can find
more information about MRQL at the website:
If you wish to write a post on Hadoop and want to share your experience/expertise, click here.
Comments
Post a Comment