Central Data Team Often Becomes Blockers

I have been working as a Data Engineer for many years and have helped migrate, transform, and modernize data teams for many companies. But at the end of the day, the data team often ends up becoming the blocker and slows down stakeholders. To understand how to solve this, I recommend reading the book Data Mesh by O’Reilly to learn how to modernize data architecture.

Data Engineering teams often do not understand the business well enough to implement the BI layer or perform analytics. As a result, engineers become dependent on analysts, and analysts become dependent on engineers to bring data. At the same time, data analytics teams want to perform analytics quickly instead of waiting for data engineering work, which creates dependencies and delays.

Central Data Team Dependency

So, how do we solve this problem?

Using Data Mesh as a data architecture promotes decentralized data ownership and aligns technology and business through a product-centric approach. It addresses the challenges of traditional data architecture models, which often result in data silos, lack of data governance, and slow decision-making due to centralized data ownership.

In a Data Mesh architecture, data ownership is decentralized, with each product team owning its own data. This enables faster decision-making, more efficient data management, improved data quality, and easier data access. The architecture is based on domain-driven design, event-driven architecture, and microservices, and it encourages collaboration between teams.

Data Mesh Pillars

Data Ownership: Decentralize the ownership of analytical data to business domains closest to the data.

Data as a Product: Domain-oriented data is shared as a product directly with data users, following guidelines that make it shareable and reusable.

Self-serve Data Platform: The data core team empowers domain cross-functional teams to share and consume data without dependency by providing a self-serve data platform.

Federated Governance: This model works on federated decision-making and accountability, with a team composed of domain representatives, data platform teams, and subject matter experts.

What Is Data Mesh?

Data Mesh is a decentralized approach to sharing, accessing, and managing analytical data in complex and large-scale environments. It enables domain teams to perform cross-platform analytics without being dependent on data engineers.

Domain teams ingest operational data and build analytical data models as data products for their own analysis. They may also publish these data products with data contracts to serve the needs of other domains.

Domain teams agree on global policies such as interoperability, security, and documentation through federated governance. A domain-agnostic self-serve data platform enables teams to discover, understand, and use data products effectively. An enabling team guides domains on modeling analytical data, using the platform, and maintaining interoperable data products.