Skip to main content

Let the Benchmarking contests begin


And, you thought your Big Data cluster was faster than your next door competitor. Oh, well… it’s time to prove it, contribute to an initiative and of course, get the honors. A new benchmarking initiative called BigData  Top100 was announced at the O’Reilly Strata Conference 2013 in a joint presentation by Chaitan Baru (SDSC) and Milind Bhandarkar (Greenplum). Other members of the BigData Top100 List steering group, include Dhruba Borthakur (Facebook), Eyal Gutkind (Mellanox), Jian Li (IBM), Raghunath Nambiar (Cisco), Ken Osterberg (Seagate), Scott Pearson (Brocade), Meikel Poess (Oracle), Tilmann Rabl (University of Toronto), Richard Treadway (NetApp), and Jerry Zhao (Google).

In the proposed benchmark, the group aims to arrive at a list of systems which can process the representative big data workload on a dataset of fixed size in the least amount of total time procured on a fixed budget, as specified by the benchmark.

The seeds of this initiative go back to late 2011 when the Center for Large-scale Data Systems Research (CLDS) at the San Diego Supercomputer Center, University of California San Diego initiated this activity. As part of this activity, 2 key workshops have been organized in May 2012 at San Jose (USA) and in December 2012 at Pune (India). The third workshop is planned in July 2013 at Xi’an (China).


“These meetings substantiated the initial ideas for a big data benchmark, which would include definitions of the data along with a data-generation procedure;
a workload representing common big data applications;
and a set of metrics, run rules,
and full-disclosure reports for fair comparisons of technologies and platforms
. These results would then be presented in the form of the BigData Top100 List, released on a regular basis at a predefined venue such as at the Strata Conferences.”

In a paper published in Big Data Journal, the group has established a workload specification to be used in first version of the benchmark. The various steps of proposed end-to-end entity-modeling pipeline include:
1-      Collect “user” interactions data and ingest them into the big data platform(s)
2-      Reorder the events according to the entity of interest, with secondary ordering according to timestamps
3-      Join the “fact tables” with various other “dimension tables.”
4-      Identify events of interest that one plans to correlate with other events in the same session for each entity
5-      Build a model for favorable/unfavorable target events based on the past session information
6-      Score the models built in the previous step with the hold-out data
7-      Apply the models to the initial entities, which did not result in the target event
8-      Publish and apply the model



From a critique perspective, the first evident pointer leads to similar initiatives in high performance computing. Queries have also been raised on the methodology to arrive at a fair comparison. Also, the steering group is loaded with corporate organizations. But, with this critique itself lie probably the strengths of this pioneering initiative. Firstly, if a robust list of Top 100 and in future Top 500 Big Data systems can come out, then not only it would stimulate competition, it would also ensure a standard to live by in non enterprise and enterprise clusters. Similarly, the steering group has made it clear that it would be a “concurrent benchmarking model, where one version of the benchmark is implemented while the next revision is concurrently being developed, incorporating more features and feedback from the first round of benchmarking”. Further, the presence of heavyweight corporate representation in the steering group as well as workshops indicates the interest is high enough and the competition also.

With regards to roll out, watch out for 3 Kaggle contests coming up this year for data collection, reference implementations and proposals respectively. Wishing the organizers and contestants a good one and hope to see the benchmark aligning  further as a self organizing initiative. 

------------------------------------------------------
image courtesy: freedigitalphotos.net

Comments

Popular posts from this blog

In-memory data model with Apache Gora

Open source in-memory data model and persistence for big data framework Apache Gora™ version 0.3, was released in May 2013. The 0.3 release offers significant improvements and changes to a number of modules including a number of bug fixes. However, what may be of significant interest to the DynamoDB community will be the addition of a gora-dynamodb datastore for mapping and persisting objects to Amazon's DynamoDB. Additionally the release includes various improvements to the gora-core and gora-cassandra modules as well as a new Web Services API implementation which enables users to extend Gora to any cloud storage platform of their choice. This 2-part post provides commentary on all of the above and a whole lot more, expanding to cover where Gora fits in within the NoSQL and Big Data space, the development challenges and features which have been baked into Gora 0.3 and finally what we have on the road map for the 0.4 development drive.
Introducing Apache Gora Although there are var…

Data deduplication tactics with HDFS and MapReduce

As the amount of data continues to grow exponentially, there has been increased focus on stored data reduction methods. Data compression, single instance store and data deduplication are among the common techniques employed for stored data reduction.
Deduplication often refers to elimination of redundant subfiles (also known as chunks, blocks, or extents). Unlike compression, data is not changed and eliminates storage capacity for identical data. Data deduplication offers significant advantage in terms of reduction in storage, network bandwidth and promises increased scalability.
From a simplistic use case perspective, we can see application in removing duplicates in Call Detail Record (CDR) for a Telecom carrier. Similarly, we may apply the technique to optimize on network traffic carrying the same data packets.
Some of the common methods for data deduplication in storage architecture include hashing, binary comparison and delta differencing. In this post, we focus on how MapReduce and…

Amazon DynamoDB datastore for Gora

What was initially suggested during causal conversation at ApacheCon2011 in November 2011 as a “neat idea”, would soon become prime ground for Gora's first taste of participation within Google's Summer of Code program. Initially, the project, titled Amazon DynamoDB datastore for Gora, merely aimed to extend the Gora framework to Amazon DynamoDB. However, it seem became obvious that the issue would include much more than that simple vision.

The Gora 0.3 Toolbox We briefly digress to discuss some other noticeable additions to Gora in 0.3, namely: Modification of the Query interface: The Query interface was amended from Query<K, T> to Query<K, T extends Persistent> to be more precise and explicit for developers. Consequently all implementors and users of the Query interface can only pass object's of Persistent type. Logging improvements for data store mappings: A key aspect of using Gora well is the establishment and accurate definitio…