When it comes to agile estimation, quite often agile teams spend a big chunk of their time in heavyweight processes like detailed story point estimation to improve the predictability or accuracy of their estimates. In other words, agile teams invest more time in estimation to reduce that variance (i.e. time estimated vs. time taken) which otherwise can be used to create features and add business value.

Recently I was asked about content management systems (CMS) of the future - more specifically how they are evolving in the era of microservices, APIs, and serverless computing. For enterprise customers either undergoing or planning the digital transformation, this is an important question to ask. Enterprise customers spend a large chunk of their digital and marketing budget on CMS and associated modules such as digital asset management (DAM).

Building data pipelines can offer strategic advantages to the business. It can be used to power new analytics, insight, and product features. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines. Data pipeline initiatives are generally unfinished projects. In this post, we will discuss why you should avoid building data pipelines in first place.

The assumption that engagement is somehow strongly correlated with performance is misplaced. Traditional engagement metrics such as satisfaction, happiness, well-being let alone cannot explain behaviours, actions and motivation of a high-performance team. Moreover, there is hardly any strong scientific evidence suggesting that there is a correlation between high-engagement and high-performance.

Developing Extract–transform–load (ETL) workflow is a time-consuming activity yet a very important component of data warehousing process. The process to develop ETL workflow is often ad-hoc, complex, trial and error based. It has been suggested that formal modeling of ETL process can alleviate most of these pain points.

If you are running workloads in Kubernetes, your site reliability engineering (SRE) and operations (Ops) teams need right kind of tooling to ensure the high-reliability of the Kubernetes cluster and workloads running in it. Here we present a list of 10 open-source Kubernetes tools to make your SRE and Ops teams more effective to achieve their service level objectives.

ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. ETL is a product of the relational database era and it has not evolved much in last decade. With the arrival of new cloud-native tools and platform, ETL is becoming obsolete. There are several emerging data trends that will define the future of ETL in 2018.

Kubernetes has emerged as go to container orchestration platform for data engineering teams. Kubernetes has a massive community support and momentum behind it. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads ^{1 2} and data workloads are up next.

On this blog from very early on, we have advocated the concept of service mesh. In fact, our post a sidecar for your service mesh is one of the most viewed posts this year. When Buoyant announced the Conduit - their next-generation lightweight service mesh for Kubernetes - we were really thrilled. Buoyant is also the creator of Linkerd which is one of the most widely used service mesh currently available to the microservices community.

Data integration generally requires in-depth domain knowledge, a strong understanding of data schemas and underlying relationships.

A case for ELT (i.e. extract, load, and transform) and difference between ETL and ELT