Diary of a Tech Savant and Servant Leader - All things technology, product, and engineering leadership.

Abhishek Tiwari

Distributed traces are essential for understanding the behaviour, performance, and reliability of microservices architecture. They can be used to surface meaningful observations about service dependencies, call graphs, and runtime dynamics, enabling software engineers and scientists to develop new tools for optimisation and fault diagnosis.

Distributed Traces of Microservices Architecture: Publicly Available Datasets

A recent study published by researchers from Washington University in St. Louis and Uber Technologies reveals a critical but often overlooked pattern in microservices architecture: non-fatal errors. While these errors do not cause system failures, they introduce noticeable performance overhead that can impact user experience and operational costs.

The Hidden Cost of Success: Understanding Non-Fatal Errors in Microservices

Recent analyses of Meta and Alibaba’s production microservices architectures identified patterns of heavy-tail and power law distributions. These patterns manifest in service scale, and request patterns, providing a glimpse into the inherent characteristics of large-scale distributed systems. For details review of Meta and Alibaba’s microservices ecosystem please see here and here.

Power Laws and Heavy-Tail Distributions in Hyperscale Microservices Architectures

The rise of service-oriented architecture (SOA) and microservices architecture has led to a major shift in software development, enabling the creation of complex, distributed systems composed of independent, loosely coupled services. These architectures offer numerous benefits, including scalability, flexibility, and resilience.

Unveiling Graph Structures in Microservices: Service Dependency Graph, Call Graph, and Causal Graph

A recent Google report highlights poor viewability of digital ads and offers some interesting statistics around various viewability factors. Report suggests that about fifty percent of ad impressions are never seen which means basically fifty percent of display spending are just wasted.

The current state of ad viewability

Recently I migrated this website from Ghost to Hugo. This site is now generated by Hugo, stored by Github, deployed on Cloudflare Pages, and content managed via Decap CMS. Hugo, Decap CMS, Cloudflare Pages, and GitHub together create a powerful and efficient stack for building, managing, and deploying static websites.

Deploying Hugo Sites on Cloudflare Pages with Decap CMS and GitHub Backend

Differential Privacy is a powerful framework for ensuring privacy in data analysis by adding controlled noise to computations. Its mathematical foundation guarantees that the presence or absence of any individual’s data in a dataset does not significantly affect the outcome of an analysis. Here are six key equations that capture the essence of differential privacy and its mechanisms, along with references to their origins and explanations.

Differential Privacy: 6 Key Equations Explained

Since 2009, I have published over 125 articles on this blog, creating a space where my work intersects with the rigor of academic research and the applied challenges of industry. This platform has grown beyond a personal archive. It has become a resource for researchers, industry practitioners, and students alike. It currently attracts more than 18k monthly visitor from across the globe.

Archiving with The Rogue Scholar

Privacy in data systems has traditionally focused on protecting sensitive information as it enters a system - what we call input privacy. However, as systems become more complex and capable of inferring sensitive information from seemingly harmless data, the importance of output privacy has gained significant attention. Let’s explore these two crucial aspects of privacy protection and understand how different techniques address them.

Input vs Output Privacy

Safeguarding individual privacy inherently means data minimisation i.e. limiting the collection and disposal of data. This principle has been a cornerstone of privacy advocacy and is even enshrined in regulations like the EU’s General Data Protection Regulation (GDPR). However, a research published by Ponte et. al (see here) is challenging this fundamental assumption, introducing what they call the “Where’s Waldo effect.

The Where's Waldo Effect in Privacy

In last post we covered, Privacy Preserving Measurement (PPM) and discussed how Distributed Aggregation Protocol (DAP) works (see here). Today, we’ll explore how to implement a simplified version of the DAP using Python with Prio3 as our Verifiable Distributed Aggregation Function (VDAF). This implementation will support multiple clients, demonstrating how DAP can aggregate data from multiple sources while maintaining privacy.

Distributed Aggregation Protocol (DAP) Primer

Differential Privacy (DP) is a mathematical framework that protects individual privacy in data analysis while allowing useful insights to be extracted. It works by adding carefully calibrated noise to data or query results, ensuring that including or excluding any single individual’s data doesn’t significantly change the analysis outcomes.

Differential Privacy: A Primer

We’ve all experienced the frustration of our smartphones crashing or freezing at the worst possible moment. So how is it that NASA can land a rover on Mars, millions of miles away, with software that works flawlessly? The answer lies in a combination of factors that set space-grade software apart from your average app. Let’s dive deep into the world of aerospace software engineering and uncover the secrets behind NASA’s incredible reliability.

Engineering excellence is how NASA flawlessly landed Curiosity on Mars

Anyone who has worked with advanced search systems knows the intricacy of boolean operators. AND, OR, NOT - these seemingly simple words become powerful tools in the hands of search specialists. Add in proximity operators like WITH and NEAR, throw in some parentheses for nesting, and you’ve got yourself a query that can pinpoint exactly what you’re looking for… or so we thought. In practice, these queries can become unwieldy beasts.

The funnel of relevance

Personal data processing forms the backbone of many big tech service providers. Tech giants like Netflix, Meta, and Amazon employ intricate networks of microservices to automatically process user data, creating complex data flows that span multiple layers of computation.

Modelling data flows as graphs to apply user privacy constraint

Jsonnet is a new domain specific configuration language from Google which allows you to define data templates. These data templates are transformed into JSON objects using Jsonnet library or command line tool. As a language, Jsonnet is extension of JSON - a valid JSON object is always valid Jsonnet template. In terms of use cases, initial focus seems to be on configuration management, package definition, application configuration, etc.

Jsonnet: A data templating language from Google

Organizational debt is a sibling of technical debt. While technical debt is unhealthy and generally a well-understood problem, organizational debt can be killer and often treated as an afterthought.

Organizational debt

Having a well-defined strategy is crucial for success of a modern business. A good strategy should address the core problems or challenges faced by a business, provide guiding principles on how to tackle them, and be supported by the coherent actions and right resources. In this article, we will explore the core elements of a good strategy.

Kernel of a good strategy - Why, How and What

Term “marketing middleware” was recently pioneered by marketing technologist and blogger Scott Brinker (at least in context of marketing technology).  Search for term “marketing middleware” on web leads to Scott Brinker’s posts on Marketing Technology Landscape Supergraphic and Tag management software as marketing middleware.

Marketing Middleware and Data Layer

In digital landscape user experience is paramount and organisations must ensure their applications and systems are functioning optimally at all times. Traditional testing and monitoring practices have long been relied upon to achieve this goal. However, the line between testing and monitoring is becoming increasingly blurred with the emergence of synthetic monitoring.

Sufficiently Advanced Monitoring is Indistinguishable from Testing

Anjali Yakkundi  from Forester offers some interesting insights about current digital asset management (DAM) technologies in the market. Most of DAM vendors are either struggling to adopt new digital trends or unable to respond to new type of requirements. Poor Usability  Surprisingly, usability remains biggest DAM challenge for existing as well as new users and buyers.

Why are current DAM offerings so uninspiring?

A very happy new year to all my readers. The excitement of making resolution in new year’s week is practically irresistible. So here I am. This post is for myself more than anyone else. 2013  It feels like only yesterday when I wrote a similar resolution post last year. What I can say? Time flies very quickly, often so quickly that there is not enough time to accomplish all resolutions. That was story of last year.

2014 - The Year Ahead, 2013 - The Year Lookback

When it comes to agile estimation, quite often agile teams spend a big chunk of their time in heavyweight processes like detailed story point estimation to improve the

 predictability

or

 accuracy

of their estimates. In other words, agile teams invest more time in estimation to reduce that variance (i.e. time estimated vs. time taken) which otherwise can be used to create features and add business value.

Agile: Optimise for velocity over predictability

A lot has been written about current state of responsive images. Things like why we need responsive images

 1


 2

, use-cases

 3

, requirements

 3

and how to choose a responsive image solution

 4


 5


 6


 7

are well documented elsewhere. This article is my attempt to distill signal from noise.

Responsive Image Distilled

The assumption that engagement is somehow strongly correlated with performance is misplaced. Traditional engagement metrics such as satisfaction, happiness, well-being let alone cannot explain behaviours, actions and motivation of a high-performance team. Moreover, there is hardly any strong scientific evidence suggesting that there is a correlation between high-engagement and high-performance.

Why engaged teams are not always the best performers?

On this blog from very early on, we have advocated the concept of service mesh. In fact, our post a sidecar for your service mesh is one of the most viewed posts this year. When Buoyant announced the Conduit - their next-generation lightweight service mesh for Kubernetes - we were really thrilled. Buoyant is also the creator of Linkerd which is one of the most widely used service mesh currently available to the microservices community.

Getting started with Conduit - lightweight service mesh for Kubernetes

ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. ETL is a product of the relational database era and it has not evolved much in last decade. With the arrival of new cloud-native tools and platform, ETL is becoming obsolete. There are several emerging data trends that will define the future of ETL in 2018.

5 data integration trends that will define the future of ETL in 2018

Can machine learning-based data structures i.e.  learned data structures replace traditional data structures? This is a question recently asked and explored by a team of Google researchers led by Jeff Dean with a major focus on database indexes. Jeff is a Google Senior Fellow in the Google Brain team and widely known as a pioneer in artificial intelligence (AI) and deep learning community.

Upcoming of the learned data structures

At re:Invent 2017, Amazon Web Services (AWS) announced Amazon Time Sync Service which provides a highly accurate and reliable global time reference. This service is immediately available in all AWS regions. Amazon Time Sync Service  In many ways, Amazon Time Sync Service is truly inspired by Google’ TrueTime. TrueTime is a global reference clock with a bounded non-zero error. TrueTime utilizes satellite-connected GPS and atomic clocks.

Rise of TrueTime: Rationale behind Amazon Time Sync Service

Leadership as a service is not about extracting work or anything from teams but it’s about serving them and inspiring them by service.

Best leaders are admired by their teams. They lead the team without positional authority. They command the unparalleled deep respect of team which is generally quite evident. They are driven by a purpose to serve and not by the power which comes with positional authority. But this blog post is not about leaders. I am here to tell you that leadership does not reside solely within individuals themselves i.e leaders.

Leadership as a service

Setting and running Kubernetes on Amazon Web Services (AWS) is a very involved process. AWS has decided to not implement Kubernetes as a Service but built something of its own - Amazon EC2 Container Service (ECS). Hence, you need to know and consider a lot of things before you can successfully roll out Kubernetes on AWS. These are the consideration which can make or break it your Kubernetes rollout.

Kubernetes on AWS: What you need to know and why

Alibaba has built a complex system of microservices to support its large user base and manage its diverse business operations. This article explores key learning from Alibaba’s microservices architecture, presenting critical observations from its design, scalability, performance optimisation, and resource allocation model.

Microservice Architecture of Alibaba

To unlock the true value of data, organisations will need internal data services. Data services provide streamlined and centralised data access to a diverse set of users which removes the friction in delivering faster insights, products and services. Data services promote innovation. In addition, effective implementation of data services can reduce data duplication hence reducing the cost of data management and storage.

For next wave of innovation organisations will need internal data services

CloudFront is a content delivery service offered by Amazon web services(AWS). CloudFront serves static contents (images, audio, video etc) using a global network of more than 28+ edge locations. Using these edge locations, CloudFront accelerates delivery of content by serving the cached copies of the content objects from a nearest edge location. Service is highly reliable and designed to be used with Amazon S3 or any other custom origin server.

CloudFront Design Patterns And Best Practices

Recently I moved my blog from Scriptogram to static site generated by Jekyll and hosted on Amazon S3. I am a big fan of Vagrant and I really like the way it keeps clean separation between various development environments without affecting host operating system. This post will explain how you can set-up a Vagrant instance for Jekyll based blogging supported by S3 hosting.

Jekyll, Amazon S3, Vagrant And Puppet

For those who are not familiar with MVC (Model–view–controller), MVC is a software architectural pattern primarily used for web development and to implement user interfaces. Ruby on Rails (RoR), Django, Spring MVC, ASP .net, Symfony, Zend, Express are some of the most popular MVC frameworks currently available for rapid web development. Over the time web development community has immensely benefited from these MVC frameworks.

MVC for data science is here

In a recent InfoWorld article, FoundationDB’s Stephen Pimentel explained the rise of the multimodel database. A multimodel database uses a single data store to support multiple data models. For instance, FoundationDB tries to map documents, graphs, and relational tables to a key-value data store.

Multimodel Database or Polyglot Persistence

A recent report by Forrester Research on digital experience delivery platforms has named Adobe and Hybris as leading strong performers. Adobe was recognised for its web experience management suit which includes content management, digital assets management, targeting and personalisation capabilities.

Digital Experience  Delivery Platforms

What can we learn from Cyberpunk 2077’s disastrous launch? Cyberpunk 2077 was extensively advertised yet highly buggy release. Sony has pulled Cyberpunk 2077 from its PlayStation store following widespread complaints about glitches. The game lost over 75% of players on Steam. The stock value of CD Projekt Red (CDPR) - Developer of Cyberpunk 2077 - is down by 50%. CDPR board and management are now defensive as they now face hostile staff (i.e.

Continuous Delivery of Games

When hiring for senior roles, companies put too much stress on stakeholder management skills i.e. ability to navigate through complex stakeholder landscape and manage expectations. I find the ability to form meaningful and impactful partnership a lot more important skill than stakeholdership.

Partnership over Stakeholdership

Hundreds of marketing technologies, thousands of tags - effective tag management requires more than just an enterprise tag management solution. In this post, I present you my 10 commandments for effective tag management. I think these 10 simple rules are quite obvious and useful for anyone interested in implementing a more robust and effective tagging regime.

The 10 Commandments of Tag Management

Functions as a service (FaaS) is an emerging pattern to build APIs and microservices at scale. You can use various FaaS implementations such as AWS Lambda, Azure Functions, and Google Cloud Functions to build APIs ecosystem for your organisation. If you remove serverless requirements, stored procedures can be considered as a variation of FaaS because they share similar traits and concerns.

Stored Procedure as a Service (SPaaS)

A recent white paper from Ghostery provides some interesting best practices for managing your site’s marketing cloud. Following is a rundown of key take aways from the report including my own comments.

Tag management: Focus on SSL- Security, Structure and Latency

When using Amazon Redshift, distribution style plays an important role in optimising the table design for best performance. In a nutshell, table’s distribution style dictates how the data is distributed across Redshift node and slices. A key objective is to avoid the data redistribution during query execution or runtime. This is accomplished by locating or co-locating the data where it needs to be before the query is executed.

Amazon Redshift : Avoid data redistribution

If you work with AWS CloudFormation on a day-to-day basis, I think you might be interested in visualising your AWS CloudFormation templates. CloudFormation stack visualization can be quite handy for lifecycle management of your AWS deployment. In addition, visualization can be immensely helpful in change management and governance.

Visualising CloudFormation Stack

In 2005 Stonebraker et al. published a paper that outlined 8 key requirements for stream processing architecture. These key requirements can be easily translated into building blocks of stream processing architecture. Although, this article dates before systems such as Apache Kafka, Amazon Kinesis, Apache Spark, Apache Storm, etc. most of the requirements are still relevant.

Requirements for stream processing architecture

This is a quick rundown on how to install, configure and tune the PageSpeed on Ubuntu with Apache as web server. Installing  First get the appropriate mod_pagespeed (64-bit or 32-bit version) .DEB package for your Ubuntu server.

How to PageSpeed with Apache

Migration of this blog from Jekyll to Statamic is now complete. To my surprise migration process was very smooth and quick (it took less than 5 hours). As part of migration I also ported design of existing Jekyll website to Statamic. In addition, actual content migration was quite straightforward.

Migrating to Statamic

A lot have been said and written about growing competition between major marketing cloud vendors and how they are more different than similar. We can include Adobe, Oracle, Salesforce, Google, IBM and may be Microsoft in this game of marketing cloud.

Marketing cloud: Buy, Curate or Build

Typically my Vagrant workflow starts with a official base box provided by Ubuntu. With help of a Vagrant providervagrant up will create a virtual machine (VM) using specified base box.

Creating a new Vagrant base box from an existing VM

A Responsive Image as Service (RIaS) offers on-the-fly responsive image generation and delivery using REST APIs or REST like image paths. RIaS is a key component of any responsive image solution. RIaS acts as image proxy. To generate responsive images it uses master images typically stored in an image repository. RIaS can be deployed as standalone service as well as embedded service.

Responsive Image as Service (RIaS)

A new breed of tools have started appearing on our radar supporting the very idea of automated visual regression.

Visual Regression

In Python/Django Celery with RabbitMQ is widely used for background processing  or distributed task queue. Although Celery is really focused on being a distributed task queue, it can also be used as scheduler using it’s  periodic tasks feature celerybeat which kicks off tasks at regular intervals. Installing  Installing Celery and RabbitMQ is quite easy. First install RabbitMQ, add a new user and remove guest user.

Periodic Tasks and Queue Processing in Django

This blog is reloaded- in terms of theme, content and target audience. For regular readers and visitors of this site/blog, this is a drastic change. If you came here looking for a bioinformatics and system biology blog then you will be fairly disappointed. Recently I made a career choice to move away from active research. In an attempt to reset everything I decided to remove most of the old content from this site.

Blog Reloaded

My notes and thoughts on Hadoop Ecosystem from book Hadoop Operations

 1

. One of the major key take aways is emergence of the Hadoop cluster deployment and management tools such as hstack and Apache AMBARI. In our own setup we managed to deploy and scale the Hadoop clusters on AWS with few boto scripts and set of puppet recipies.

Hadoop Ecosystem- Deployment And Management

The Bloom filter was originally developed by Burton H. Bloom back in the seventies and for long time it was there without any major application. Google is credited for making Bloom filter popular again. Only after the Google used Bloom filters for their BigTable database system, the idea started grabbing the attention of larger and diverse audience.

Bloom filters for bioinformatics

Scala is a JVM languages designed by  Martin Odersky in early 2001 and it brings features of object-oriented programming and functional programming together. Scala is competing again several new JVM languages including but not limited to Groovy, Clojure, JRuby and Jython. Scala resembles Python (and Ruby) in many ways and as a Pythonist I found it real easy to switch on Scala.

Scala for the Python Geeks

It is good to know your data. But there is clear distinction between being data driven vs data informed. No matter which area you work, there is always an opportunity to make additional gains by closely observing the characteristic and quality of your data. By experimenting and looking carefully at the data you may identify some hidden patterns which may not be visible or obvious. But can we rely on peculiarities of the data?

Knowing your data vs relying on it

Over the weekend, I took a shot to build something new in React. React is a JavaScript library to build user interfaces. React was open sourced by Facebook and since then it has gained popularity over other frontend frameworks such as Angular, Vue, etc. To start, I have a good experience in JavaScript and I have built several applications using Node, Express, and Angular. Unfortunately, React landscape is way too complex.

What I've learned so far about React

In a recent blog post, we discussed object-inspired container design patterns in detail and the sidecar pattern was one of them. In a sidecar pattern, the functionality of the main container is extended or enhanced by a sidecar container without strong coupling between two. This pattern is particularly useful when using Kubernetes as container orchestration platform. Kubernetes uses Pods. A Pod is composed of one or more application containers.

A sidecar for your service mesh

I have been playing with Apache Drill for quite some time now. In layman’s terms, Apache Drill is SQL query engine which can perform queries against any type of data store - in particular any non-relational data store. Apache Drill’s ability to query against raw data files stored on the cloud storage platforms such as Amazon S3, Azure Blob Storage, Google Cloud Storage makes it a really attractive and cost effective query engine.

Reflections on Apache Drill

Currently the majority of cloud based database and data warehouse services are provisioned with fixed storage and compute resources. Resizing of resources cannot be performed without compromising availability and performance. This means service users typically end up with over-provisioned under-utilised expensive resources to accommodate possible peak demand. In the worst case, under-provisioned resources unable to handle sudden work overloads.

Rise of Elastic Data Warehouse and Database Services

After much deliberation, recently W3C Customer Experience Digital Data Community Group released version 1.0 of Customer Experience Digital Data Layer (CEDDL). This specification is designed to communicate customer experience digital data to analytics and reporting services. What is a data layer? A data layer is a standard way to format data within a web page.

A Standard Data Layer for Customer Experience Data

Recently I wrote about a standard data layer for customer experience data. Extending our last discussion, in this post I am going to describe how you can use custom data attributes (data-*) and data layer for event tracking. As an example and to demonstrate use of custom data attributes and data layer, we are going to use Google Analytics.

Event Tracking with Data Layer and Data Attributes

In last 5 years or so, content management systems have evolved. New breed of content management systems (CMS) are powered by content repositories. Behind the scenes content repositories use familiar persistence managers such as relational databases, file systems, in-memory etc. That said, content repositories are more than just plain persistence storage.

Decoupling content management using content repositories

First of all Happy New Year to you all. Honestly speaking this post was due in first week of this month but it’s not too late for new year’s resolution. As matter of fact I desperately wanted to write this post - to share my excitement for year 2013 and also humbling experience in 2012. 2012  2012 was phenomenal, both personally and professionally especially considering a rather depressing 2011.

2013 - The Year Ahead, 2012 - The Year Lookback

I often screen papers for my reading list based on their illustration appeal. I know this may be bit strange for the people who judge the papers simply based on their abstracts or conclusion section. But trust me it works because I know what I am looking for. I always avoid the sleep inducing papers full of creepy tables and bar graphs, in fact so lacking in interest as to cause mental weariness.

Put some breathe life in your papers with clever visualization

In 1982, Andrew Yao proposed the Millionaire Problem which discusses how two millionaires can learn who is richest one without disclosing their actual wealth. They solve this problem by comparing their wealth using secure two party computation to ensure that they learn only the richest one and nothing else is revealed. The problem was later generalised for  secure multiparty computation by Goldreich et al in 1987.

Privacy Preserving Measurement

Recently I have been reading Art of Leadership, The: Small Things, Done Well by Michael Lopp. This book is an excellent read and it covers small things that you can do to build trust and to become an authentic and true leader to your team at different stages of your leadership journey.

Engineering excellence: the art of doing small things really well

Ever since the Amazon introduced the AWS Lambda service at AWS re:Invent 2014, a variety of new applications for the service has emerged which highlights tremendous potential and traction for the AWS Lambda service. Over last one year, Amazon has been actively working towards integrating other AWS services with AWS Lambda.

AWS Lambda and Endless Serverless Possibilities

In simple words, a Google Tag Manager (GTM) container holds other tags. GTM container is also know as universal container tag or site-wide tag. A GTM container is managed via GTM dashboard and controls tag dependencies and tag firing order. On your web page, you can have multiple GTM containers. GTM container code looks like following - embedded either header or footer of your web page template.

Google Tag Manager: How it works

There are key differences between deployment and release. Deployment means putting the latest stable version of code on your web application servers. Latest version of code often includes a range of features or improvements. Release means making all or selected features available to the public or end-users. In addition, one main difference between deployment and release is the ability to roll back.

Decoupling Deployment and Release- Feature Toggles

AWS allows tagging of EC2 resources using your own metadata (key-value pair). Tagging of EC2 resources is quite helpful in managing instances, images and organise billing by tag names. An EC2 instance can be tagged by CLI tools, API wrappers or using AWS Console when launching the instance as well as after launching the instance.

Interacting with Tagged EC2 Instances using Fabric

Homomorphic encryption is a powerful cryptographic technique that allows computations to be performed on encrypted data without decrypting it first. This blog post will introduce the concept of homomorphic encryption and demonstrate implementations using Python. What is Homomorphic Encryption? Homomorphic encryption is a form of encryption that allows specific types of computations to be carried out on ciphertext.

Exploring Homomorphic Encryption with Python

At the heart of privacy lies the principle of purpose limitation, dictating that data should only be processed for explicitly stated purposes. This principle presents a considerable challenge, especially for organisations operating at the scale of Meta, which handles vast amounts of data from billions of users.

Policy Zones: Purpose Limitation at Scale using Information Flow Control

As microservices architectures have become increasingly common in modern software systems, they have brought benefits and challenges. One of the most pressing challenges has been maintaining performance at scale while dealing with complex service dependencies and network communication overhead.

Cache Me If You Can: Taming the Caching Complexity of Microservice Call Graphs

Multi-touch attribution is considered as holy grail in advertising industry. As advertisers are targeting users with multiple advertisements across different platforms and publishers, understanding how each of these touch points contributes to conversion is crucial—but this understanding has traditionally come at the cost of user privacy.

Privacy-Preserving Multi-Touch Attribution at TikTok

Secure multi-party computation (SMPC) enables organisations to collaborate on sensitive data analysis without directly sharing raw information. However, seemingly harmless aggregate outputs, particularly private set intersection (PSI), can leak individual-level information when analysed strategically over time.

Birds of a Feather Leak Together: The Set Bias Privacy Problem

When a machine learning model is trained on a dataset, not all data points contribute equally to the model’s performance. Some are more valuable and influential than others. Unfortunately value of data for training purposes is often nebulous and difficult to quantify. Applying data valuation to large language models (LLMs) like GPT-3, Claude 3, Llama 3.1 and their vast training datasets has faced significant scalability challenges to date.

What is worth of your data to LLM?

For big data and complex data processing, data pipelines have emerged as a popular solution for managing and manipulating data. They provide a systematic approach to extract, transform, and load (ETL) data from various sources, enabling organizations to derive valuable insights. However, as with any technology trend, data pipelines have not been immune to misuse and overuse.

Data Pipelines: The Hammer for Every Nail

When it comes to organising engineering teams, a popular view has been to organise your teams based on either Spotify’s agile model (i.e. squads, chapters, tribes, and guilds) or simply follow Amazon’s two-pizza team model.

Organise your engineering teams around the work by reteaming

Tech companies and large consumer businesses are grappling with how best to protect end-user data while maintaining pace of innovation and competitive edge. Two distinct approaches have emerged: top-down and bottom-up privacy. Understanding these approaches is essential for anyone involved in privacy engineering, product development, or driving tech policy decisions.

Top-down vs. Bottom up Privacy

By default Google Tag Manager (GTM) provides several handy macros and lookup table macro is one of them. In GTM, by using lookup table macro you can reduce number of deployed tags as well as the number of rules you need to load the tags. When implementing tags in GTM always follow famous DRY principle (don’t repeat yourself). Use of lookup table avoids duplication and enables reusability.

Google Tag Manager: Lookup table and DRY Principle

Believe it or not, on average an US ecommerce website has between 15-20 marketing tags. Implementing and maintaining these many marketing tags on your website is often a cumbersome job. Google Tag Manager (GTM) is a great tool for marketer as well as developers. Like any other Tag management solution (TMS), GTM not only decouples deployment of these marketing tags but also introduces an efficient way of sharing data to these tags via data layer.

Google Tag Manager: Ecommerce Integration

Few months back I managed to get Statamic working with Varnish. Setting up Varnish for Statamic was quite easy and my website performance improved drastically (you can see load and performance testing benchmarks here). One of the biggest challenges was Varnish cache invalidation on content update. Every time, Statamic content was updated I have to invalidate cache manually using curl requests.

Varnish cache buster add-on for Statamic

SeeWhy

 1

a market leader in real-time behavioural marketing, has recently published an extensive and empirical research on ROI of 1-to-1 email remarketing programs.

Unreasonable Effectiveness of Email Remarketing

After working with Adobe AEM and several other JCR backed content management systems in last few years I have realised that there are a lot of mis-conceptions about Adobe AEM. As matter of fact, brain washing is quite common these days when Adobe AEM is on the table. Sometimes Adobe is on fault, on other occasions rhetoric of competitors, agencies and architects.

Adobe AEM: Good, Bad and Ugly

A mathematical guarantee is a formal, provable assurance about the behavior, performance, or properties of a system, algorithm, or process, derived from rigorous mathematical analysis or proof. The guarantee is based on mathematical logic and can be demonstrated through mathematical reasoning. It provides a definitive statement about what can be expected under specified conditions.

Mathematical Guarantee

I am thrilled to announce the early availability of Axioms - a financial-grade identity and trust management platform. Axioms enables organizations to protect their apps, devices, and APIs against unauthorized and untrusted access using our fully-managed SaaS offering.

Introducing Axioms

Recently I was asked about content management systems (CMS) of the future - more specifically how they are evolving in the era of microservices, APIs, and serverless computing. For enterprise customers either undergoing or planning the digital transformation, this is an important question to ask. Enterprise customers spend a large chunk of their digital and marketing budget on CMS and associated modules such as digital asset management (DAM).

Content Management Systems of the Future: Headless, JAMstack, ADN and Functions at the Edge

If you are running workloads in Kubernetes, your site reliability engineering (SRE) and operations (Ops) teams need right kind of tooling to ensure the high-reliability of the Kubernetes cluster and workloads running in it. Here we present a list of 10 open-source Kubernetes tools to make your SRE and Ops teams more effective to achieve their service level objectives.

10 open-source Kubernetes tools for highly effective SRE and Ops Teams

Ability to think critically and anticipate potential problems is essential for success for product management. One mental model that can greatly aid product managers in this regard is inversion. Inversion, a concept popularized by the German mathematician Carl Jacobi, involves thinking about a problem in reverse order or considering the opposite of what is desired.

Using Inversion as a Mental Model for Product Management

Recently I discussed at length about Responsive Images and Responsive Image as Service. In this post I will discuss how we can use some effective templating techniques to implement highly maintainable responsive image markup. Templates express presentation logic of a web application. Templates are quite similar to plain HTML markup but they are re-usable and modular.

Templating for Responsive Images

Statamic is a new PHP CMS with relatively small community. This website currently runs on Statamic. When it comes to setup,  Statamic documentation provides very brief instructions on how you can install Statamic. Normally installing Statamic is quite easy. This is a quick rundown on how you can install Statamic on Ubuntu with Apache as web server.

Installing Statamic

The ability to deploy applications quickly and efficiently is crucial for organizations. Agile methodologies and continuous integration/continuous deployment (CI/CD) pipelines have become the norm, enabling rapid release cycles and frequent updates. However, amidst the drive for speed, ensuring policy compliance is often overlooked, leading to potential security vulnerabilities and compliance risks.

Pre-Deployment Policy Compliance

It seriously turns me off when organisations are unable to differentiate between a role and a job. Talking about this confusion, Scrum Product Owner probably one of the most misunderstood roles these days which unfortunately many organisations advertise as a job.  A job describes what one does daily. A role encompasses how our actions help an organisation to succeed. Next time you are not sure, ask yourself how you add value to the organisation.

Roles vs Jobs

The evolution of your technology architecture should depend on the size, culture, and skill set of your engineering organization. In fact, they should coevolve. There are no hard-and-fast rules to figure out interdependency between technology architecture and engineering organization but below is what I think can really work well for a typical product startup.

Your technology architecture and engineering organization should coevolve as your startup grows

The promise of differential privacy is compelling. It offers a rigorous, provable guarantee of individual privacy, even in the face of arbitrary background knowledge. Rather than relying on anonymization techniques that can often be defeated, differential privacy works by injecting carefully calibrated noise into computations.

Managing Differential Privacy in Large Scale Systems

In contemporary technology environments, organisations are increasingly challenged with the complexities of privacy engineering. The evolving data governance and regulatory ecosystems demand not only technical ingenuity but also a deep understanding of legal frameworks and organizational dynamics.