Live data from Hacker News

Data Mesh Architecture

datamesh-architecture.com

41–46 of 46 posts

Re: Data Mesh Architecture

#41

Isn't this usually called a "data mart" as opposed to "data mesh"? Or is the "mesh" term intended to point to something more unstructured, like team- or business division-level equivalent to a data lake? But isn't that just a data pond?

Mesh implies that there are clearly defined join keys on each dataset that allows you do join across domains.

E-commerce example: the warehouse team might produce a batch of datasets, and the web site time as well. A data lake approach would have a single data team owning both sets of datasets. A data mesh would have each team be responsible for maintaining their own, and making sure they’re interoperable (like having a shared order ID concept).

Re: Data Mesh Architecture

#42
post #23

It sounds almost entirely about team responsibility and governance, rather than technical architecture. What’s the difference from a data lake on a technical level?

You get a data lake per team/service.

what's a good word for a region with a bunch of small lakes? the fens?

"Data Mesh" is much trendier branding than "Data Fenlands".

Re: Data Mesh Architecture

#43
post #42

Earlier quoted context omitted.

You get a data lake per team/service.

what's a good word for a region with a bunch of small lakes? the fens? "Data Mesh" is much trendier branding than "Data Fenlands".

Sounds like a Silicon Fen[1] startup waiting to happen.

[1] https://en.wikipedia.org/wiki/Silicon_Fen

Re: Data Mesh Architecture

#44
post #42

Earlier quoted context omitted.

You get a data lake per team/service.

what's a good word for a region with a bunch of small lakes? the fens? "Data Mesh" is much trendier branding than "Data Fenlands".

A "water garden". No joke.

https://martinfowler.com/bliki/DataLake.html

Re: Data Mesh Architecture

#45
post #28

Earlier quoted context omitted.

At some point one has to ask, what are you guys doing ingesting 5pb of data per day?! Unless this is google, that doesn’t make any sense. That’s an average of 7.5mb per human on the planet, every day.

I did say FAANG :) Believe it or not this is just for security data. But that fact combined with SOA leads to lots of logs. (5Pbs is the uncompressed amount, but we decompress incoming data, then ETL).

Would love to know some of the answers to the questions you posed

Re: Data Mesh Architecture

#46

Earlier quoted context omitted.

What are the downsides of OLAP cubes, and how were they fixed? Curious to level up my understanding.

I guess they had their place in some point and time, but I still vividly remember my old manager speaking about building OLAP cube in 2018. https://www.holistics.io/blog/the-rise-and-fall-of-the-olap-...

What made them obsolete, Snowflake?
Post reply on HN