The evolution of your technology architecture should depend on the size, culture, and skill set of your engineering organization. In fact, they should coevolve. There are no hard-and-fast rules to figure out interdependency between technology architecture and engineering organization but below is what I think can really work well for a typical product startup.
When hiring for senior roles, companies put too much stress on stakeholder management skills i.e. ability to navigate through complex stakeholder landscape and manage expectations. I find the ability to form meaningful and impactful partnership a lot more important skill than stakeholdership.
When it comes to organising engineering teams, a popular view has been to organise your teams based on either Spotify’s agile model (i.e. squads, chapters, tribes, and guilds) or simply follow Amazon’s two-pizza team model.
When it comes to agile estimation, quite often agile teams spend a big chunk of their time in heavyweight processes like detailed story point estimation to improve the predictability or accuracy of their estimates. In other words, agile teams invest more time in estimation to reduce that variance (i.e. time estimated vs. time taken) which otherwise can be used to create features and add business value.
Recently I was asked about content management systems (CMS) of the future - more specifically how they are evolving in the era of microservices, APIs, and serverless computing. For enterprise customers either undergoing or planning the digital transformation, this is an important question to ask. Enterprise customers spend a large chunk of their digital and marketing budget on CMS and associated modules such as digital asset management (DAM).
Building data pipelines can offer strategic advantages to the business. It can be used to power new analytics, insight, and product features. Often companies underestimate the necessary effort and cost involved to build and maintain data pipelines. Data pipeline initiatives are generally unfinished projects. In this post, we will discuss why you should avoid building data pipelines in first place.
The assumption that engagement is somehow strongly correlated with performance is misplaced. Traditional engagement metrics such as satisfaction, happiness, well-being let alone cannot explain behaviours, actions and motivation of a high-performance team. Moreover, there is hardly any strong scientific evidence suggesting that there is a correlation between high-engagement and high-performance.
Developing Extract–transform–load (ETL) workflow is a time-consuming activity yet a very important component of data warehousing process. The process to develop ETL workflow is often ad-hoc, complex, trial and error based. It has been suggested that formal modeling of ETL process can alleviate most of these pain points.
If you are running workloads in Kubernetes, your site reliability engineering (SRE) and operations (Ops) teams need right kind of tooling to ensure the high-reliability of the Kubernetes cluster and workloads running in it. Here we present a list of 10 open-source Kubernetes tools to make your SRE and Ops teams more effective to achieve their service level objectives.
ETL refers to extract, transform, load and it is generally used for data warehousing and data integration. ETL is a product of the relational database era and it has not evolved much in last decade. With the arrival of new cloud-native tools and platform, ETL is becoming obsolete. There are several emerging data trends that will define the future of ETL in 2018.
Kubernetes has emerged as go to container orchestration platform for data engineering teams. Kubernetes has a massive community support and momentum behind it. In 2018, a widespread adaptation of Kubernetes for big data processing is anitcipated. Organisations are already using Kubernetes for a variety of workloads 1 2 and data workloads are up next.