“Hadoop is backend…it
is for batch…it is for Google, Facebook … how does a web developer
come in to picture…”
Lets dispel some perception… and demonstrate an
architectural flow for pulling data from HDFS right into a Yahoo Pipe, IFTTT
recipe or Zapier zap.
1- Setup Hadoop, HDFS with data coming in the nodes
2- Write the MapReduce algorithms to process the data you
want
3- Setup Hive or HBase or Voldemort or your preferred
DB/warehouse option depending on your interest.
4- Setup Struts/Spring based application in Eclipse/RAD/your
preferred IDE with supported web server like WAS 7.0/JBoss
5- Write a program to fetch the data from Hive (or other
variants mentioned in step 3 above), format it, render as RSS response and
provide servlet to serve it back as response
.
|
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean
id="hadoopFeedGenerationAction" class="com.hadoop.rss.action.hadoopFeedGenerationAction"
init-method="checkInitialization"
scope="prototype">
<property
name="generator" ref="hadoopFeedGenerator"/>
</bean>
<bean
id="hadoopItemDAO" class="com.hadoop.rss.dao.hadoopItemDAOImpl" init-method="checkInitialization">
<property name="jdbcTemplate"
ref="jdbcTemplate"/>
</bean>
<bean
id="hadoopFeedFormatter" class="com.hadoop.rss.formatter.hadoopFeedFormatterImpl"
init-method="checkInitialization">
<property
name="description" value="channel desc"/>
<property
name="link" value="www.test.com"/>
<property
name="title" value="channel title"/>
<property
name="contentTemplate">
<value><![CDATA[<table>
<tr><td>Request
Id</td><td>%requestId%</td></tr>
<tr><td>Title</td><td>%requestTitle%</td></tr>
<tr><td>Type
Code</td><td>%requestTypeCode%</td></tr>
<tr><td>Requester Web
Id</td><td>%requesterWebId%</td></tr>
<tr><td>Step start
time</td><td>%stepStartTime%</td></tr>
<tr><td>Status</td><td>%status%</td></tr>
<tr><td>Link to item</td><td>%requestLink%</td></tr>
</table>]]></value>
</property>
<property
name="rssFeedGenerator" ref="rssFeedGenerator"/>
<property
name="itemTitle" value="item title"/>
</bean>
|
Sample xml file excerpt for RSS generation – use case: transaction data fetch
6- Deploy and host the .war on a public server
7- Mash up RSS feed in the pipe or recipe or zap.

Use case examples:
1-
Need a live index of words in
documents uploaded by users
2-
Weather data served right from the
Hadoop crunchers to AJAX
renderers
Contact Sachin
Ghai for further details on this Hadoop hack.
Comments
Post a Comment