Diary of a Tech Savant and Servant Leader - All things technology, product, and engineering leadership.

Abhishek Tiwari

A recent study published by researchers from Washington University in St. Louis and Uber Technologies reveals a critical but often overlooked pattern in microservices architecture: non-fatal errors. While these errors do not cause system failures, they introduce noticeable performance overhead that can impact user experience and operational costs.

The Hidden Cost of Success: Understanding Non-Fatal Errors in Microservices

Distributed traces are essential for understanding the behaviour, performance, and reliability of microservices architecture. They can be used to surface meaningful observations about service dependencies, call graphs, and runtime dynamics, enabling software engineers and scientists to develop new tools for optimisation and fault diagnosis.

Distributed Traces of Microservices Architecture: Publicly Available Datasets

Recent analyses of Meta and Alibaba’s production microservices architectures identified patterns of heavy-tail and power law distributions. These patterns manifest in service scale, and request patterns, providing a glimpse into the inherent characteristics of large-scale distributed systems. For details review of Meta and Alibaba’s microservices ecosystem please see here and here.

Power Laws and Heavy-Tail Distributions in Hyperscale Microservices Architectures

In contemporary technology environments, organisations are increasingly challenged with the complexities of privacy engineering. The evolving data governance and regulatory ecosystems demand not only technical ingenuity but also a deep understanding of legal frameworks and organizational dynamics.

Privacy Engineering: Why Standards Are Still Out of Reach?

Recently I migrated this website from Ghost to Hugo. This site is now generated by Hugo, stored by Github, deployed on Cloudflare Pages, and content managed via Decap CMS. Hugo, Decap CMS, Cloudflare Pages, and GitHub together create a powerful and efficient stack for building, managing, and deploying static websites.

Deploying Hugo Sites on Cloudflare Pages with Decap CMS and GitHub Backend

Since 2009, I have published over 125 articles on this blog, creating a space where my work intersects with the rigor of academic research and the applied challenges of industry. This platform has grown beyond a personal archive. It has become a resource for researchers, industry practitioners, and students alike. It currently attracts more than 18k monthly visitor from across the globe.

Archiving with The Rogue Scholar

Privacy in data systems has traditionally focused on protecting sensitive information as it enters a system - what we call input privacy. However, as systems become more complex and capable of inferring sensitive information from seemingly harmless data, the importance of output privacy has gained significant attention. Let’s explore these two crucial aspects of privacy protection and understand how different techniques address them.

Input vs Output Privacy

The promise of differential privacy is compelling. It offers a rigorous, provable guarantee of individual privacy, even in the face of arbitrary background knowledge. Rather than relying on anonymization techniques that can often be defeated, differential privacy works by injecting carefully calibrated noise into computations.

Managing Differential Privacy in Large Scale Systems

Safeguarding individual privacy inherently means data minimisation i.e. limiting the collection and disposal of data. This principle has been a cornerstone of privacy advocacy and is even enshrined in regulations like the EU’s General Data Protection Regulation (GDPR). However, a research published by Ponte et. al (see here) is challenging this fundamental assumption, introducing what they call the “Where’s Waldo effect.

The Where's Waldo Effect in Privacy

In last post we covered, Privacy Preserving Measurement (PPM) and discussed how Distributed Aggregation Protocol (DAP) works (see here). Today, we’ll explore how to implement a simplified version of the DAP using Python with Prio3 as our Verifiable Distributed Aggregation Function (VDAF). This implementation will support multiple clients, demonstrating how DAP can aggregate data from multiple sources while maintaining privacy.

Distributed Aggregation Protocol (DAP) Primer

A mathematical guarantee is a formal, provable assurance about the behavior, performance, or properties of a system, algorithm, or process, derived from rigorous mathematical analysis or proof. The guarantee is based on mathematical logic and can be demonstrated through mathematical reasoning. It provides a definitive statement about what can be expected under specified conditions.

Mathematical Guarantee

Differential Privacy (DP) is a mathematical framework that protects individual privacy in data analysis while allowing useful insights to be extracted. It works by adding carefully calibrated noise to data or query results, ensuring that including or excluding any single individual’s data doesn’t significantly change the analysis outcomes.

Differential Privacy: A Primer

Personal data processing forms the backbone of many big tech service providers. Tech giants like Netflix, Meta, and Amazon employ intricate networks of microservices to automatically process user data, creating complex data flows that span multiple layers of computation.

Modelling data flows as graphs to apply user privacy constraint

Anyone who has worked with advanced search systems knows the intricacy of boolean operators. AND, OR, NOT - these seemingly simple words become powerful tools in the hands of search specialists. Add in proximity operators like WITH and NEAR, throw in some parentheses for nesting, and you’ve got yourself a query that can pinpoint exactly what you’re looking for… or so we thought. In practice, these queries can become unwieldy beasts.

The funnel of relevance

Having a well-defined strategy is crucial for success of a modern business. A good strategy should address the core problems or challenges faced by a business, provide guiding principles on how to tackle them, and be supported by the coherent actions and right resources. In this article, we will explore the core elements of a good strategy.

Kernel of a good strategy - Why, How and What

For big data and complex data processing, data pipelines have emerged as a popular solution for managing and manipulating data. They provide a systematic approach to extract, transform, and load (ETL) data from various sources, enabling organizations to derive valuable insights. However, as with any technology trend, data pipelines have not been immune to misuse and overuse.

Data Pipelines: The Hammer for Every Nail

The ability to deploy applications quickly and efficiently is crucial for organizations. Agile methodologies and continuous integration/continuous deployment (CI/CD) pipelines have become the norm, enabling rapid release cycles and frequent updates. However, amidst the drive for speed, ensuring policy compliance is often overlooked, leading to potential security vulnerabilities and compliance risks.

Pre-Deployment Policy Compliance

In digital landscape user experience is paramount and organisations must ensure their applications and systems are functioning optimally at all times. Traditional testing and monitoring practices have long been relied upon to achieve this goal. However, the line between testing and monitoring is becoming increasingly blurred with the emergence of synthetic monitoring.

Sufficiently Advanced Monitoring is Indistinguishable from Testing

What can we learn from Cyberpunk 2077’s disastrous launch? Cyberpunk 2077 was extensively advertised yet highly buggy release. Sony has pulled Cyberpunk 2077 from its PlayStation store following widespread complaints about glitches. The game lost over 75% of players on Steam. The stock value of CD Projekt Red (CDPR) - Developer of Cyberpunk 2077 - is down by 50%. CDPR board and management are now defensive as they now face hostile staff (i.e.

Continuous Delivery of Games

Organizational debt is a sibling of technical debt. While technical debt is unhealthy and generally a well-understood problem, organizational debt can be killer and often treated as an afterthought.

Organizational debt

When it comes to organising engineering teams, a popular view has been to organise your teams based on either Spotify’s agile model (i.e. squads, chapters, tribes, and guilds) or simply follow Amazon’s two-pizza team model.

Organise your engineering teams around the work by reteaming

Recently I was asked about content management systems (CMS) of the future - more specifically how they are evolving in the era of microservices, APIs, and serverless computing. For enterprise customers either undergoing or planning the digital transformation, this is an important question to ask. Enterprise customers spend a large chunk of their digital and marketing budget on CMS and associated modules such as digital asset management (DAM).

Content Management Systems of the Future: Headless, JAMstack, ADN and Functions at the Edge

The assumption that engagement is somehow strongly correlated with performance is misplaced. Traditional engagement metrics such as satisfaction, happiness, well-being let alone cannot explain behaviours, actions and motivation of a high-performance team. Moreover, there is hardly any strong scientific evidence suggesting that there is a correlation between high-engagement and high-performance.

Why engaged teams are not always the best performers?

If you are running workloads in Kubernetes, your site reliability engineering (SRE) and operations (Ops) teams need right kind of tooling to ensure the high-reliability of the Kubernetes cluster and workloads running in it. Here we present a list of 10 open-source Kubernetes tools to make your SRE and Ops teams more effective to achieve their service level objectives.

10 open-source Kubernetes tools for highly effective SRE and Ops Teams

Kubernetes has emerged as go to container orchestration platform for data engineering teams. Kubernetes has a massive community support and momentum behind it. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads

 1


 2

and data workloads are up next.

Kubernetes for Big Data Workloads

Data integration generally requires in-depth domain knowledge, a strong understanding of data schemas and underlying relationships.

How machine learning is accelerating data integration?

A case for ELT (i.e. extract, load, and transform) and difference between ETL and ELT

Cheap storage and on-demand compute in the cloud coupled with the emergence of new big data frameworks and tools are forcing us to rethink the whole ETL and data warehousing architecture. There is a strong argument for ELT i.e. extract, load, and transform model. Classic ETL  In classic Extract, transform, and load (ETL) model, we store entities in their corresponding application databases i.e. as rows in the relational tables.

A case for ELT

Can machine learning-based data structures i.e.  learned data structures replace traditional data structures? This is a question recently asked and explored by a team of Google researchers led by Jeff Dean with a major focus on database indexes. Jeff is a Google Senior Fellow in the Google Brain team and widely known as a pioneer in artificial intelligence (AI) and deep learning community.

Upcoming of the learned data structures

At re:Invent 2017, Amazon Web Services (AWS) announced Amazon Time Sync Service which provides a highly accurate and reliable global time reference. This service is immediately available in all AWS regions. Amazon Time Sync Service  In many ways, Amazon Time Sync Service is truly inspired by Google’ TrueTime. TrueTime is a global reference clock with a bounded non-zero error. TrueTime utilizes satellite-connected GPS and atomic clocks.

Rise of TrueTime: Rationale behind Amazon Time Sync Service

If we start thinking containers in terms of objects, it abstracts away the low-level details but reveals higher-level patterns that are common to a variety of applications and algorithms. A recent paper, published by Google Researchers Brendan Burns and David Oppenheimer, suggests that

 containers are particularly well-suited as the fundamental “object” in distributed systems by virtue of the walls they erect at the container boundary

.

Object-inspired container design patterns

Over the weekend, I took a shot to build something new in React. React is a JavaScript library to build user interfaces. React was open sourced by Facebook and since then it has gained popularity over other frontend frameworks such as Angular, Vue, etc. To start, I have a good experience in JavaScript and I have built several applications using Node, Express, and Angular. Unfortunately, React landscape is way too complex.

What I've learned so far about React

In AI and machine learning, the future resembles the past and bias refers to prior information. There has been a growing interest in identifying the harmful biases in the machine learning. Often these harmful biases are just the reflection or amplification of human biases which algorithms learn from training data. Some training data sets such as text, medical, criminal, educational, financial etc.

Bias and Fairness in Machine Learning

Setting and running Kubernetes on Amazon Web Services (AWS) is a very involved process. AWS has decided to not implement Kubernetes as a Service but built something of its own - Amazon EC2 Container Service (ECS). Hence, you need to know and consider a lot of things before you can successfully roll out Kubernetes on AWS. These are the consideration which can make or break it your Kubernetes rollout.

Kubernetes on AWS: What you need to know and why

Functions as a service (FaaS) is an emerging pattern to build APIs and microservices at scale. You can use various FaaS implementations such as AWS Lambda, Azure Functions, and Google Cloud Functions to build APIs ecosystem for your organisation. If you remove serverless requirements, stored procedures can be considered as a variation of FaaS because they share similar traits and concerns.

Stored Procedure as a Service (SPaaS)

To unlock the true value of data, organisations will need internal data services. Data services provide streamlined and centralised data access to a diverse set of users which removes the friction in delivering faster insights, products and services. Data services promote innovation. In addition, effective implementation of data services can reduce data duplication hence reducing the cost of data management and storage.

For next wave of innovation organisations will need internal data services

I have been playing with Apache Drill for quite some time now. In layman’s terms, Apache Drill is SQL query engine which can perform queries against any type of data store - in particular any non-relational data store. Apache Drill’s ability to query against raw data files stored on the cloud storage platforms such as Amazon S3, Azure Blob Storage, Google Cloud Storage makes it a really attractive and cost effective query engine.

Reflections on Apache Drill

When using Amazon Redshift, distribution style plays an important role in optimising the table design for best performance. In a nutshell, table’s distribution style dictates how the data is distributed across Redshift node and slices. A key objective is to avoid the data redistribution during query execution or runtime. This is accomplished by locating or co-locating the data where it needs to be before the query is executed.

Amazon Redshift : Avoid data redistribution

DevOps is a cultural shift with immediate focus on maximising the business value by opting better communication, collaboration and feedback within and across IT development and operation teams. Infrastructure as Code (IaC) is a key element of DevOps philosophy with benefits for both development and operation teams. The term infrastructure as code is sometimes also referred to as programmable infrastructure.

How to Organise Your Infrastructure as Code

Ever since the Amazon introduced the AWS Lambda service at AWS re:Invent 2014, a variety of new applications for the service has emerged which highlights tremendous potential and traction for the AWS Lambda service. Over last one year, Amazon has been actively working towards integrating other AWS services with AWS Lambda.

AWS Lambda and Endless Serverless Possibilities

Currently the majority of cloud based database and data warehouse services are provisioned with fixed storage and compute resources. Resizing of resources cannot be performed without compromising availability and performance. This means service users typically end up with over-provisioned under-utilised expensive resources to accommodate possible peak demand. In the worst case, under-provisioned resources unable to handle sudden work overloads.

Rise of Elastic Data Warehouse and Database Services

If you are working with an e-commerce website, 5 to 10 e-commerce conversion pixels are quite common these days. Ad conversion pixels, affiliate conversion pixels, web analytics conversion pixels are most common type of tags typically deployed on an e-commerce website. Most of conversion pixels are designed to collect the transaction data such as id, total revenue, tax, products etc.

Google Tag Manager: Don’t Repeat Yourself Conversion Tracking

For those who are not familiar with MVC (Model–view–controller), MVC is a software architectural pattern primarily used for web development and to implement user interfaces. Ruby on Rails (RoR), Django, Spring MVC, ASP .net, Symfony, Zend, Express are some of the most popular MVC frameworks currently available for rapid web development. Over the time web development community has immensely benefited from these MVC frameworks.

MVC for data science is here

In a recent InfoWorld article, FoundationDB’s Stephen Pimentel explained the rise of the multimodel database. A multimodel database uses a single data store to support multiple data models. For instance, FoundationDB tries to map documents, graphs, and relational tables to a key-value data store.

Multimodel Database or Polyglot Persistence

Recently I discovered Looker, a relatively new Business Intelligence (BI) platform which utilises a YAML based modeling language LookML. LookML is core engine of Looker. LookML offers a very powerful, highly expressive yet lightweight modeling layer. LookML combines data modelling with on-demand data transformations, and data derivations at the same layer.

How Looker is unlocking Business Intelligence using YAML

A recent Google report highlights poor viewability of digital ads and offers some interesting statistics around various viewability factors. Report suggests that about fifty percent of ad impressions are never seen which means basically fifty percent of display spending are just wasted.

The current state of ad viewability

Sometime you want to generate UUID for visitors on your web property and set UUID as cookie. I want to show how easy it is to set your own visitor UUID cookie using Google Tag Manager(GTM). Once visitor UUID cookie is set you can pass this UUID to other tags. Normally most of tags set tag specific UUIDs but by setting your own UUID you can manage your own cookie match table with your own UUID as primary key. So lets get started.

Google Tag Manager: Visitor UUID Cookie Setter Tag

Few months back I managed to get Statamic working with Varnish. Setting up Varnish for Statamic was quite easy and my website performance improved drastically (you can see load and performance testing benchmarks here). One of the biggest challenges was Varnish cache invalidation on content update. Every time, Statamic content was updated I have to invalidate cache manually using curl requests.

Varnish cache buster add-on for Statamic

By default Google Tag Manager (GTM) provides several handy macros and lookup table macro is one of them. In GTM, by using lookup table macro you can reduce number of deployed tags as well as the number of rules you need to load the tags. When implementing tags in GTM always follow famous DRY principle (don’t repeat yourself). Use of lookup table avoids duplication and enables reusability.

Google Tag Manager: Lookup table and DRY Principle

SeeWhy

 1

a market leader in real-time behavioural marketing, has recently published an extensive and empirical research on ROI of 1-to-1 email remarketing programs.

Unreasonable Effectiveness of Email Remarketing

After working with Adobe AEM and several other JCR backed content management systems in last few years I have realised that there are a lot of mis-conceptions about Adobe AEM. As matter of fact, brain washing is quite common these days when Adobe AEM is on the table. Sometimes Adobe is on fault, on other occasions rhetoric of competitors, agencies and architects.

Adobe AEM: Good, Bad and Ugly

A recent report by Forrester Research on digital experience delivery platforms has named Adobe and Hybris as leading strong performers. Adobe was recognised for its web experience management suit which includes content management, digital assets management, targeting and personalisation capabilities.

Digital Experience  Delivery Platforms

Vagrant is a perfect choice for

 virtual development environment

. Vagrant was created and maintained by Mitchell Hashimoto. It benefits developers, operations engineer and designers. Vagrant really shines when working in a team of any size - small, medium or large.

Getting started with Vagrant

Today I want to talk about the performance testing. Automated web application testing is not very rare these days. This includes functional, unit and regression tests at minimum. A good web application delivery approach normally includes a continuous integration setup to run automated tests.

Performance testing as a first-class citizen

Term “marketing middleware” was recently pioneered by marketing technologist and blogger Scott Brinker (at least in context of marketing technology).  Search for term “marketing middleware” on web leads to Scott Brinker’s posts on Marketing Technology Landscape Supergraphic and Tag management software as marketing middleware.

Marketing Middleware and Data Layer

So you got your website up and running using Statamic. For better performance you have enabled, Statamic rendered-HTML caching. Cache-Control header for all static assets Gzip compression at web server level  Basically, Statamic rendered-HTML caching  stores the entire rendered HTML page and by enabling it you can cut  down page-load time by up to 90%~85% on average.

Statamic with Varnish for a high-performance website

This is a quick rundown on how to install, configure and tune the PageSpeed on Ubuntu with Apache as web server. Installing  First get the appropriate mod_pagespeed (64-bit or 32-bit version) .DEB package for your Ubuntu server.

How to PageSpeed with Apache

Recently I have been playing with Web Components (WC). W3C specification drafts describe Web Components as the component model for the web

 1

. This component model consists of five new constructs:

 HTML Templates

,

 Decorators

,

 Custom Elements

,

 Shadow DOM

and

 HTML Imports

. These constructs will be used as new primitives in the browser.

Web Components: Web's Polymer Future

Apache Mesos is a popular open source cluster manager which enables building resource-efficient distributed systems. Mesos provides efficient dynamic resources isolation and sharing across multiple distributed applications such as Hadoop, Spark, Memcache, MySQL etc on a dynamic shared pool of resources nodes. This means with Mesos you can run any distributed application which requires clustered resources.

Building Distributed Systems with Mesos

Anjali Yakkundi  from Forester offers some interesting insights about current digital asset management (DAM) technologies in the market. Most of DAM vendors are either struggling to adopt new digital trends or unable to respond to new type of requirements. Poor Usability  Surprisingly, usability remains biggest DAM challenge for existing as well as new users and buyers.

Why are current DAM offerings so uninspiring?

Migration of this blog from Jekyll to Statamic is now complete. To my surprise migration process was very smooth and quick (it took less than 5 hours). As part of migration I also ported design of existing Jekyll website to Statamic. In addition, actual content migration was quite straightforward.

Migrating to Statamic

Two days back Scott Brinker published  Marketing Technology Landscape for 2014. This is the third edition of his widely appreciated and popular marketing technology landscapes. This version includes about 950 marketing technology companies into 43 categories across 6 major classes. These companies mainly provide software and tools for marketers. This is 600 more than year 2012 and ~9 times compared to year 2011.

Marketing Technology Landscape 2014

Recently I discussed at length about Responsive Images and Responsive Image as Service. In this post I will discuss how we can use some effective templating techniques to implement highly maintainable responsive image markup. Templates express presentation logic of a web application. Templates are quite similar to plain HTML markup but they are re-usable and modular.

Templating for Responsive Images

A very happy new year to all my readers. The excitement of making resolution in new year’s week is practically irresistible. So here I am. This post is for myself more than anyone else. 2013  It feels like only yesterday when I wrote a similar resolution post last year. What I can say? Time flies very quickly, often so quickly that there is not enough time to accomplish all resolutions. That was story of last year.

2014 - The Year Ahead, 2013 - The Year Lookback

A Responsive Image as Service (RIaS) offers on-the-fly responsive image generation and delivery using REST APIs or REST like image paths. RIaS is a key component of any responsive image solution. RIaS acts as image proxy. To generate responsive images it uses master images typically stored in an image repository. RIaS can be deployed as standalone service as well as embedded service.

Responsive Image as Service (RIaS)

A lot has been written about current state of responsive images. Things like why we need responsive images

 1


 2

, use-cases

 3

, requirements

 3

and how to choose a responsive image solution

 4


 5


 6


 7

are well documented elsewhere. This article is my attempt to distill signal from noise.

Responsive Image Distilled

Typically my Vagrant workflow starts with a official base box provided by Ubuntu. With help of a Vagrant providervagrant up will create a virtual machine (VM) using specified base box.

Creating a new Vagrant base box from an existing VM

When it comes to select a Content Delivery Networks (CDN) there are many factors you need to consider such as performance, availability, scalability, reliability, pricing model, features, SLA etc. In my opinion performance and pricing are key in the decision making process. Performance  CDN performance is measured in terms of speed of DNS resolution and content delivery.

Measuring CDN Performance, Availability and Scalability

In this world there are all kind of people. But in context of this post I am interested in only two type of people. On one side there are people who are not afraid of change and embrace the change in life. On the other side there are human beings who are less open to change. Change can be personal or professional.

Embracing change when on peak

Polyglot persistence which takes a hybrid approach to persistence is getting a lot of traction these days. Currently I am reading NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence by Pramod J. Sadalage and Martin Fowler. Basic idea behind polyglot persistence is to use specialized databases (both NoSQL and Relational) for different purposes within the same web application.

Polyglot Persistence Patterns

CloudFront is a content delivery service offered by Amazon web services(AWS). CloudFront serves static contents (images, audio, video etc) using a global network of more than 28+ edge locations. Using these edge locations, CloudFront accelerates delivery of content by serving the cached copies of the content objects from a nearest edge location. Service is highly reliable and designed to be used with Amazon S3 or any other custom origin server.

CloudFront Design Patterns And Best Practices

Celery is an asynchronous distributed task queue. RabbitMQ is a message broker which implements the Advanced Message Queuing Protocol (AMQP). Before we describe relationship between RabbitMQ and Celery, a quick overview of AMQP will be helpful

 1


 2

. Amqp Key Terms Message Or Task A message or task consists of attributes (headers) and payload (body). Some attributes are used by broker but most are used by consumers.

AMQP, RabbitMQ and Celery - A Visual Guide For Dummies

Recently I moved my blog from Scriptogram to static site generated by Jekyll and hosted on Amazon S3. I am a big fan of Vagrant and I really like the way it keeps clean separation between various development environments without affecting host operating system. This post will explain how you can set-up a Vagrant instance for Jekyll based blogging supported by S3 hosting.

Jekyll, Amazon S3, Vagrant And Puppet

My notes and thoughts on Hadoop Ecosystem from book Hadoop Operations

 1

. One of the major key take aways is emergence of the Hadoop cluster deployment and management tools such as hstack and Apache AMBARI. In our own setup we managed to deploy and scale the Hadoop clusters on AWS with few boto scripts and set of puppet recipies.

Hadoop Ecosystem- Deployment And Management

This blog is reloaded- in terms of theme, content and target audience. For regular readers and visitors of this site/blog, this is a drastic change. If you came here looking for a bioinformatics and system biology blog then you will be fairly disappointed. Recently I made a career choice to move away from active research. In an attempt to reset everything I decided to remove most of the old content from this site.

Blog Reloaded

Solely inspired by Atbrox’s list of academic papers for Mapreduce &amp; Hadoop Algorithms. Unlike computer science where applications of Mapreduce/Hadoop are very much diversified, most of published implementations in bioinformatics are still focused on the analysis and/or assembly of biological sequences. As usual this list will be updated time to time.

Mapreduce and Hadoop Algorithms in Bioinformatics Papers

Scala is a JVM languages designed by  Martin Odersky in early 2001 and it brings features of object-oriented programming and functional programming together. Scala is competing again several new JVM languages including but not limited to Groovy, Clojure, JRuby and Jython. Scala resembles Python (and Ruby) in many ways and as a Pythonist I found it real easy to switch on Scala.

Scala for the Python Geeks

In Python/Django Celery with RabbitMQ is widely used for background processing  or distributed task queue. Although Celery is really focused on being a distributed task queue, it can also be used as scheduler using it’s  periodic tasks feature celerybeat which kicks off tasks at regular intervals. Installing  Installing Celery and RabbitMQ is quite easy. First install RabbitMQ, add a new user and remove guest user.

Periodic Tasks and Queue Processing in Django

It is good to know your data. But there is clear distinction between being data driven vs data informed. No matter which area you work, there is always an opportunity to make additional gains by closely observing the characteristic and quality of your data. By experimenting and looking carefully at the data you may identify some hidden patterns which may not be visible or obvious. But can we rely on peculiarities of the data?

Knowing your data vs relying on it

The Bloom filter was originally developed by Burton H. Bloom back in the seventies and for long time it was there without any major application. Google is credited for making Bloom filter popular again. Only after the Google used Bloom filters for their BigTable database system, the idea started grabbing the attention of larger and diverse audience.

Bloom filters for bioinformatics

I often screen papers for my reading list based on their illustration appeal. I know this may be bit strange for the people who judge the papers simply based on their abstracts or conclusion section. But trust me it works because I know what I am looking for. I always avoid the sleep inducing papers full of creepy tables and bar graphs, in fact so lacking in interest as to cause mental weariness.

Put some breathe life in your papers with clever visualization

With great power comes great responsibility.
One of my favorite quote from movie Spider-Man which was sourced by Stan Lee, famous American writer and editor. After working with visual analytics for three years I realized that this quote is utterly correct for visual analytics. Visual analytics is a new exploratory paradigm which combines visualization, human factors and data analysis. Human factors such as interaction, cognition, perception play central role in the final outcome of the analytics and decision making process.

One of my favorite quote from movie Spider-Man which was sourced by Stan Lee, famous American writer and  editor. After working with visual analytics for three years I realized that  this quote is utterly correct for visual analytics. Visual analytics is a new exploratory paradigm which combines visualization, human factors and data analysis.

Visual analytics: Aimless wandering or Systematic data exploration

Information visualization is in the core of bioinformatics, and over the time variety of interesting information visualization approaches have been developed to visualize the different type of biological information. This post is specifically targeted on circular/radial visualization in biological domain.

Circular/Radial Visualizations in Bioinformatics

The rise of service-oriented architecture (SOA) and microservices architecture has led to a major shift in software development, enabling the creation of complex, distributed systems composed of independent, loosely coupled services. These architectures offer numerous benefits, including scalability, flexibility, and resilience.

Unveiling Graph Structures in Microservices: Service Dependency Graph, Call Graph, and Causal Graph

Alibaba has built a complex system of microservices to support its large user base and manage its diverse business operations. This article explores key learning from Alibaba’s microservices architecture, presenting critical observations from its design, scalability, performance optimisation, and resource allocation model.

Microservice Architecture of Alibaba

Differential Privacy is a powerful framework for ensuring privacy in data analysis by adding controlled noise to computations. Its mathematical foundation guarantees that the presence or absence of any individual’s data in a dataset does not significantly affect the outcome of an analysis. Here are six key equations that capture the essence of differential privacy and its mechanisms, along with references to their origins and explanations.

Differential Privacy: 6 Key Equations Explained

We’ve all experienced the frustration of our smartphones crashing or freezing at the worst possible moment. So how is it that NASA can land a rover on Mars, millions of miles away, with software that works flawlessly? The answer lies in a combination of factors that set space-grade software apart from your average app. Let’s dive deep into the world of aerospace software engineering and uncover the secrets behind NASA’s incredible reliability.

Engineering excellence is how NASA flawlessly landed Curiosity on Mars

Jsonnet is a new domain specific configuration language from Google which allows you to define data templates. These data templates are transformed into JSON objects using Jsonnet library or command line tool. As a language, Jsonnet is extension of JSON - a valid JSON object is always valid Jsonnet template. In terms of use cases, initial focus seems to be on configuration management, package definition, application configuration, etc.

Jsonnet: A data templating language from Google

When it comes to agile estimation, quite often agile teams spend a big chunk of their time in heavyweight processes like detailed story point estimation to improve the

 predictability

or

 accuracy

of their estimates. In other words, agile teams invest more time in estimation to reduce that variance (i.e. time estimated vs. time taken) which otherwise can be used to create features and add business value.

Agile: Optimise for velocity over predictability

On this blog from very early on, we have advocated the concept of service mesh. In fact, our post a sidecar for your service mesh is one of the most viewed posts this year. When Buoyant announced the Conduit - their next-generation lightweight service mesh for Kubernetes - we were really thrilled. Buoyant is also the creator of Linkerd which is one of the most widely used service mesh currently available to the microservices community.

Getting started with Conduit - lightweight service mesh for Kubernetes

ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. ETL is a product of the relational database era and it has not evolved much in last decade. With the arrival of new cloud-native tools and platform, ETL is becoming obsolete. There are several emerging data trends that will define the future of ETL in 2018.

5 data integration trends that will define the future of ETL in 2018

Leadership as a service is not about extracting work or anything from teams but it’s about serving them and inspiring them by service.

Best leaders are admired by their teams. They lead the team without positional authority. They command the unparalleled deep respect of team which is generally quite evident. They are driven by a purpose to serve and not by the power which comes with positional authority. But this blog post is not about leaders. I am here to tell you that leadership does not reside solely within individuals themselves i.e leaders.

Leadership as a service

When hiring for senior roles, companies put too much stress on stakeholder management skills i.e. ability to navigate through complex stakeholder landscape and manage expectations. I find the ability to form meaningful and impactful partnership a lot more important skill than stakeholdership.

Partnership over Stakeholdership

Hundreds of marketing technologies, thousands of tags - effective tag management requires more than just an enterprise tag management solution. In this post, I present you my 10 commandments for effective tag management. I think these 10 simple rules are quite obvious and useful for anyone interested in implementing a more robust and effective tagging regime.

The 10 Commandments of Tag Management

A recent white paper from Ghostery provides some interesting best practices for managing your site’s marketing cloud. Following is a rundown of key take aways from the report including my own comments.

Tag management: Focus on SSL- Security, Structure and Latency

If you work with AWS CloudFormation on a day-to-day basis, I think you might be interested in visualising your AWS CloudFormation templates. CloudFormation stack visualization can be quite handy for lifecycle management of your AWS deployment. In addition, visualization can be immensely helpful in change management and governance.

Visualising CloudFormation Stack

In 2005 Stonebraker et al. published a paper that outlined 8 key requirements for stream processing architecture. These key requirements can be easily translated into building blocks of stream processing architecture. Although, this article dates before systems such as Apache Kafka, Amazon Kinesis, Apache Spark, Apache Storm, etc. most of the requirements are still relevant.

Requirements for stream processing architecture

A lot have been said and written about growing competition between major marketing cloud vendors and how they are more different than similar. We can include Adobe, Oracle, Salesforce, Google, IBM and may be Microsoft in this game of marketing cloud.

Marketing cloud: Buy, Curate or Build

A new breed of tools have started appearing on our radar supporting the very idea of automated visual regression.