Data Engineering
Designing Data Products
Working backwards from the end goal is a core principle of software development, and we've found it to be highly effective in modelling data products. In this article we'll explore a step-by-step, methodical approach to identifying data products that avoids overdesign while providing just enough clarity for teams to begin implementation.
ReadGoverning Data Products Using Fitness Functions
Decentralized data management requires automation to scale governance effectively. Fitness functions are a powerful automated governance technique we've applied to data products within the context of a Data Mesh. Since data products serve as the foundational building blocks of a data mesh, ensuring robust governance around them significantly increases the chances of a successful data mesh transformation.
ReadData Mesh in practice - Getting off to the right start (Part I)
In this series of articles, we'll share practical learnings from our recent Data Mesh implementation engagement at Roche. As the world's largest global provider of healthcare, Roche generates, manages, and processes vast quantities of data across a wide range of deeply specialized domains. We lay out what it really takes to successfully apply the principles of Data Mesh in a large corporate environment.
ReadThe Curse of the data lake monster
Artificial intelligence and machine learning are currently all the rage. Every organisation is trying to jump on this bandwagon and cash in on their data reserves. Organisations are investing a lot in building fancy data lakes but often without considering the use cases. In this essay, Lucy and I explain a better approach.
Read