
Data Mesh / Data Fabric
What is it?
Data Mesh is an organisational approach to developing and operating data platforms that overcomes the pitfalls of large, monolithic platforms. Unlike warehouses, lakes, and lakehouses, Data Meshes are federated models for the data platform, delivery, and management.
A Data Mesh isn’t tied to any specific tool or platform and instead refers to a design strategy for a decentralised data platform. There are four key principles:
- Decentralised domain-oriented data ownership: In a data mesh setup, all data is owned and managed by the business. However, different departments or business areas will take ownership of the different types of data.
- Data as a product: This means treating data as reusable assets, called Data Products. Each Data Product has a clear purpose or business goal, alongside an owner responsible for its delivery, management, and alignment with business needs.
- Self-service platform: To enable agility and independence, Data Products should be usable by non-technical business users through mediums like low-code and no-code platforms.
- Federated data governance: To maintain global standards for security, data quality, and legal compliance, each team that handles data should follow a pragmatic governance framework.
A Data Mesh enables organisations to build business-focused data products but often requires a significant business change. Experts in business domains must lead government, management and access of specific data sets.
A Data Fabric integrates data across different enterprise infrastructures, including cloud, hybrid, and on-premise deployments. This integration, coupled with putting data in a common format, gives organisations a single view of their data estate and how datasets relate to metadata. It often requires significant technology investments, though this depends on existing infrastructure, systems and processes.