One of the major battles that the systems fight today is against the bots. Be it search zombies, spam email bots or registration bots, there is a big one upmanship battle between the bots and the systems.
To detect bots, a generic architecture approach utilizing MapReduce and Hadoop is shown in the figure below. This architecture is adaptable for most algorithmic techniques where MapReduce is employed currently. Custom variations may be done based on individual tool sets that the architect may be adopting.
Many e-mail systems like Hotmail have successfully implemented such techniques with Dryad as well. There are some commercial service providers which utilize Cassandra, Hive etc. on top of HDFS. Further on top of it, there is a presentation layer combined with analysis dashboard utilizing the traditional portals and BI approach.
Most systems today employ techniques like CAPTCHAs. However, there are organized attacks where the CAPTCHA have been manually or with automated prior retrieval been compromised by bots. Similarly, spam constitutes a major chunk of emails and most of these e-mails originate from Hotmail (Outlook), Gmail and Yahoo Email – the biggest fighters against spam.
Algorithms have been tuned to counter the varying refinements in spamming technology. The algorithmic approach may vary with each implementation depending on use case. For example, one e-mail provider may tend to analyse based on IP address and user ids. Another published approach mentions use of PageRank to detect P2P botnets. Other algorithms may employ Naïve Bayesian classifier or apriori rules. With computation a major bottleneck to analyze massive data sets, MapReduce offers a significant reduction in processing time with its parallel processing architecture.