Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Mesh Architecture

datamesh-architecture.com

Post reply on HN
Print view View original

21–30 of 46 posts

Re: Data Mesh Architecture

#21

Post by shoo »

kevinsundar wrote: post #19

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses: 1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen…

I believe the data mesh claim is that as well as the operational API exposed and supported by some given domain team, the domain team also gets a new goal of exposing and supporting an analytical API to deliver a data product to potential consumers. For this to happen, an organisation would need to value this new objective -- perhaps comparably to operational objectives -- and fund and resource it adequately. Arguably throwing enough budget and appropriately skilled headcount at it might address points 1, 2, 4, 5.

But, it's not obvious it actually makes sense for an organisation to value analytical concerns at a similar level to operational concerns, with similar resourcing. The value to the organisation of building and maintaining analytical APIs to serve data products is likely to be considerably less -- perhaps by one or two orders of magnitude -- than the value produced by actually performing and maintaining the core operational function.

If the stars align, maybe in future the analytical data could be used as input to an optimisation project that improves some key metric of the operational function by 10% or 1%. How much is it worth paying to have the possibility of an outcome like that in future? Not obvious, really comes down to how valuable a 10% or 1% lift would be, and how much it would cost, needs some kind of business case. Not obvious that resourcing analytical data APIs owned by arbitrary domain teams everywhere is a sound investment for the business.

Re: Data Mesh Architecture

#22

Post by shoo »

skrrr wrote: post #17
i_like_waiting wrote: post #10

So if I understand this correctly, data mesh is just data mart, that doesn't bring data in database as a table, but uses S3 storage instead (I assume because thats cheaper in the cloud?)

That + a central data platform team that provides infra, quality monitors, data lineage and catalogue capabilities + a central team that provides guidelines on SLAs, metadata standards etc. Sounds good in theory, I am eager to see how it fails in practice

many points of internet karma (and perhaps a profitable career as a consultant) awaits anyone who spills the beans on how their grand data mesh rearchitecture actually turned out a few years down the track, and if the exciting new problems caused by the data mesh were easier or harder to deal with than the boring old problems caused by the organisational and IT architecture it replaced.

Re: Data Mesh Architecture

#23

Post by timwis »

It sounds almost entirely about team responsibility and governance, rather than technical architecture. What’s the difference from a data lake on a technical level?

Re: Data Mesh Architecture

#24

Post by KmVFIz »

skrrr wrote: post #17
i_like_waiting wrote: post #10

So if I understand this correctly, data mesh is just data mart, that doesn't bring data in database as a table, but uses S3 storage instead (I assume because thats cheaper in the cloud?)

That + a central data platform team that provides infra, quality monitors, data lineage and catalogue capabilities + a central team that provides guidelines on SLAs, metadata standards etc. Sounds good in theory, I am eager to see how it fails in practice

I can chime in as part of the central team for SLAs, etc. We offer a platform to produce datasets given some inputs, SQL, and pushes to downstream systems. Standardized jobs are ran after the user’s SQL to produce standardized outputs.

It works well, but has many issues too. User’s SQLs and input data can differ, often in unpredictable ways, because they bring their own and expect the central team to handle the rest. Those edge cases break the standardization rules, fails the workflow, confuses the user because the platform is a black box, and they ask about changing it or adding a new feature. Now your standardization asks are bottle-necked by this central team, and the options are:

- to wait for the central team to fix/improve it

- find some hack around the platform

- don’t use the platform and its associated toolings, so you build it yourself and have another disjoint system for a specific use case

- central team might build a feature that one team asked for 1 years ago, but now nobody needs it anymore and nobody knows why it’s in code. Repeat many many times for various asks over the years and your code base is likely a foreign mess.

- give your resources/funding to the central team to prioritize your ask. When built and a few years later, the central team owns something they themselves never wanted.

Re: Data Mesh Architecture

#25

Post by cgio »

kevinsundar wrote: post #19

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses: 1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen…

Just on point 1, maybe in the context of said FAANG data quality was by choice if at all. In other industries, e.g. Finance, it can be regulated and audited, so operational teams care in terms of not being able to continue operations. That addresses point 2 partially, it is a complex topic, though.

Re: Data Mesh Architecture

#26

Post by cgio »

politelemon wrote: post #2

Is there an underlying assumption here that all of the datasets' domains are perfectly in sync with each other in the context of domain metadata? As an example, a Team1 might define the manufacturer of a Sprocket as the company that assembled it, whereas a Team2 might define the manufacturer as the company that built the Sprocket's engine. Since the purpose of a datamesh is to enable other teams to perform cross-doma…

Data mesh is not a complete framework, more sociopolitical rather than technical at the moment. When tested in practice, I think you already allured to a key technical component that will need to be more central, I.e. reconciliation. What that means in terms of domain ownership of reconciliation that is an open question.

Re: Data Mesh Architecture

#27

Post by pklee »

The concept of a data-mesh is more of a business concept as opposed to tech. IMHO the idea being proposed is that of a conceptual data-server (not to be confused with database server) much like a HTTP server / Mail Server where people can engage with data as a first class citizen and create "data" products. This is especially true as we move from HTML to somewhat HDML (Hyper data markup).

By making data as the product (abstracting all the gory details), you are fundamentally engaging with data through a UI or an API. As you expose these products they become accretive while fundamentally encapsulating the domain expertise within them.

Re: Data Mesh Architecture

#28

Post by d--b »

kevinsundar wrote: post #19

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses: 1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen…

At some point one has to ask, what are you guys doing ingesting 5pb of data per day?!

Unless this is google, that doesn’t make any sense. That’s an average of 7.5mb per human on the planet, every day.

Re: Data Mesh Architecture

#29

Post by jerglingu »

robertlagrant wrote: post #4

It really feels like data mesh is a fairly half baked concept born out of short term consulting gigs and a desire to become a technical thought leader.

I got same this feeling when reading the original white paper linked on the page[1]. It's filled with the kind of bloated abstract "consultant speak" chosen to mask relatively straightforward ideas. And then there is this casual claim at the end of the paper that IMO discredits everything preceding it[2]: "Luckily, building common infrastructure as a platform is a well understood and solved problem;"

[1]: https://martinfowler.com/articles/data-monolith-to-mesh.html

[2]: https://martinfowler.com/articles/data-monolith-to-mesh.html...

Re: Data Mesh Architecture

#30

Post by jerglingu »

kevinsundar wrote: post #19

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses: 1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen…

On point 1: "That's not gonna happen in a large company."

This doesn't happen in small or mid companies, either. Or if it does happen, it happens begrudgingly. SWE's have too much to do.

Post reply on HN