My notes and thoughts on Hadoop Ecosystem from book Hadoop Operations ¹ . One of the major key take aways is emergence of the Hadoop cluster deployment and management tools such as hstack and Apache AMBARI. In our own setup we managed to deploy and scale the Hadoop clusters on AWS with few boto scripts and set of puppet recipies.

Notes plus thoughts from my recent read Cassandra: The Definitive Guide. Common ways to solve scalability bottleneck with relational databases, Throw More/better Hardware (memory And Cpu) Vertical scaling Faster disks (SSD vs RAID) Move To A Database Cluster With master-slave configuration: Master is now single point of failure. Update slave as master during failure. Separate read from writes - dedicated slaves for reads.

Polyglot persistence which takes a hybrid approach to persistence is getting a lot of traction these days. Currently I am reading NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence by Pramod J. Sadalage and Martin Fowler. Basic idea behind polyglot persistence is to use specialized databases (both NoSQL and Relational) for different purposes within the same web application.

This blog is reloaded- in terms of theme, content and target audience. For regular readers and visitors of this site/blog, this is a drastic change. If you came here looking for a bioinformatics and system biology blog then you will be fairly disappointed. Recently I made a career choice to move away from active research. In an attempt to reset everything I decided to remove most of the old content from this site.

Solely inspired by Atbrox’s list of academic papers for Mapreduce & Hadoop Algorithms. Unlike computer science where applications of Mapreduce/Hadoop are very much diversified, most of published implementations in bioinformatics are still focused on the analysis and/or assembly of biological sequences. As usual this list will be updated time to time.

Scala is a JVM languages designed by Martin Odersky in early 2001 and it brings features of object-oriented programming and functional programming together. Scala is competing again several new JVM languages including but not limited to Groovy, Clojure, JRuby and Jython. Scala resembles Python (and Ruby) in many ways and as a Pythonist I found it real easy to switch on Scala.

In Python/Django Celery with RabbitMQ is widely used for background processing or distributed task queue. Although Celery is really focused on being a distributed task queue, it can also be used as scheduler using it’s periodic tasks feature celerybeat which kicks off tasks at regular intervals. Installing Installing Celery and RabbitMQ is quite easy. First install RabbitMQ, add a new user and remove guest user.

It is good to know your data. But there is clear distinction between being data driven vs data informed. No matter which area you work, there is always an opportunity to make additional gains by closely observing the characteristic and quality of your data. By experimenting and looking carefully at the data you may identify some hidden patterns which may not be visible or obvious. But can we rely on peculiarities of the data?

The Bloom filter was originally developed by Burton H. Bloom back in the seventies and for long time it was there without any major application. Google is credited for making Bloom filter popular again. Only after the Google used Bloom filters for their BigTable database system, the idea started grabbing the attention of larger and diverse audience.

I often screen papers for my reading list based on their illustration appeal. I know this may be bit strange for the people who judge the papers simply based on their abstracts or conclusion section. But trust me it works because I know what I am looking for. I always avoid the sleep inducing papers full of creepy tables and bar graphs, in fact so lacking in interest as to cause mental weariness.

With great power comes great responsibility. One of my favorite quote from movie Spider-Man which was sourced by Stan Lee, famous American writer and editor. After working with visual analytics for three years I realized that this quote is utterly correct for visual analytics. Visual analytics is a new exploratory paradigm which combines visualization, human factors and data analysis. Human factors such as interaction, cognition, perception play central role in the final outcome of the analytics and decision making process.