One of the major battles that the systems fight today is against
the bots. Be it search zombies, spam email bots or registration bots, there is
a big one upmanship battle between the bots and the systems.
To detect bots, a generic architecture approach utilizing MapReduce and
Hadoop is shown in the figure below. This architecture is adaptable for most algorithmic techniques where MapReduce is employed currently. Custom variations may be done based on individual tool sets that the architect may be adopting.
Many e-mail systems like Hotmail have successfully
implemented such techniques with Dryad as well. There are some commercial
service providers which utilize Cassandra, Hive etc. on top of HDFS. Further on
top of it, there is a presentation layer combined with analysis dashboard
utilizing the traditional portals and BI approach.
Most systems today employ techniques like CAPTCHAs. However,
there are organized attacks where the CAPTCHA have been manually or with
automated prior retrieval been compromised by bots. Similarly, spam constitutes
a major chunk of emails and most of these e-mails originate from Hotmail (Outlook),
Gmail and Yahoo Email – the biggest fighters against spam.
Algorithms have been tuned to counter the varying
refinements in spamming technology. The algorithmic approach may vary with each
implementation depending on use case. For example, one e-mail provider may tend
to analyse based on IP address and user ids. Another published approach
mentions use of PageRank to detect P2P botnets. Other algorithms may employ
Naïve Bayesian classifier or apriori rules. With computation a major bottleneck to analyze
massive data sets, MapReduce offers a significant reduction in processing time
with its parallel processing architecture.
Comments
Post a Comment