Data Mesh Architecture • Hacker News Board

Re: Data Mesh Architecture

#31

Post by Foobar8568 » Sun, Mar 20, 2022, 6:20 AM UTC

shoo wrote: post #22

skrrr wrote: post #17

Earlier quoted context omitted.

That + a central data platform team that provides infra, quality monitors, data lineage and catalogue capabilities + a central team that provides guidelines on SLAs, metadata standards etc. Sounds good in theory, I am eager to see how it fails in practice

many points of internet karma (and perhaps a profitable career as a consultant) awaits anyone who spills the beans on how their grand data mesh rearchitecture actually turned out a few years down the track, and if the exciting new problems caused by the data mesh were easier or harder to deal with than the boring old problems caused by the organisational and IT architecture it replaced.

You mean like the shit show that is Data Vault? https://danlinstedt.com/allposts/datavaultcat/datavault-issu...

>>data vault 2.0 brings with it methodology, architecture, modeling, and implementation – best practices, standards, automation and more. the ability to encompass and leverage disciplined agile delivery, and sei/cmmi, six sigma, lean initiatives, cycle time reduction, and proper build practices lead us to one day sprint cycles.

And let's not forget that shit show of a "book" https://www.amazon.com/Data-Architecture-Primer-Scientist-Wa...

Re: Data Mesh Architecture

#32

Post by ceeplusplus » Sun, Mar 20, 2022, 6:22 AM UTC

cgio wrote: post #25

kevinsundar wrote: post #19

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses: 1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen…

Just on point 1, maybe in the context of said FAANG data quality was by choice if at all. In other industries, e.g. Finance, it can be regulated and audited, so operational teams care in terms of not being able to continue operations. That addresses point 2 partially, it is a complex topic, though.

In my experience data quality in finance is much worse than FAANG. It's common to have just the raw data feed from markets/trades/network dumped into a OLAP DB and whoever is using it has to sort through it whereas FAANG have data engineers to clean stuff up.

Re: Data Mesh Architecture

#33

Post by beckingz » Sun, Mar 20, 2022, 7:23 AM UTC

timwis wrote: post #23

It sounds almost entirely about team responsibility and governance, rather than technical architecture. What’s the difference from a data lake on a technical level?

You get a data lake per team/service.

Re: Data Mesh Architecture

#34

Post by fn1 » Sun, Mar 20, 2022, 7:44 AM UTC

d--b wrote: post #28

kevinsundar wrote: post #19

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses: 1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen…

At some point one has to ask, what are you guys doing ingesting 5pb of data per day?! Unless this is google, that doesn’t make any sense. That’s an average of 7.5mb per human on the planet, every day.

Why the concept "data lake" emerged in the first place:

Thinking about what to store (and what to log) is not trivial and takes careful consideration. Plus, there's always the argument: "But what if we need something that we forgot to store or log?".

Answering that takes time and risk-acceptance that most developers in most projects don't get.

"Datalake" was just a pseudo-solution to gain peace-of-mind: We throw everything into a big bucket and figure it out later.

Re: Data Mesh Architecture

#35

Post by ako » Sun, Mar 20, 2022, 7:58 AM UTC

Lots of concerns and scepticism in the discussions here. Any suggestions about good, achievable data strategies and data architecture that work at enterprise level?

Re: Data Mesh Architecture

#36

Post by cgio » Sun, Mar 20, 2022, 8:49 AM UTC

ceeplusplus wrote: post #32

cgio wrote: post #25

Earlier quoted context omitted.

Just on point 1, maybe in the context of said FAANG data quality was by choice if at all. In other industries, e.g. Finance, it can be regulated and audited, so operational teams care in terms of not being able to continue operations. That addresses point 2 partially, it is a complex topic, though.

In my experience data quality in finance is much worse than FAANG. It's common to have just the raw data feed from markets/trades/network dumped into a OLAP DB and whoever is using it has to sort through it whereas FAANG have data engineers to clean stuff up.

I am in Finance and while I don’t question that may be the case for some, the reality is that with regulatory requirements getting more sophisticated, anyone who does not focus on DQ end to end is making a very costly decision. When the quality of data pipelines ends up impacting the capital requirements, the cost of bad data quality is hitting the PL straight away.

Re: Data Mesh Architecture

#37

Post by ricardobeat » Sun, Mar 20, 2022, 11:20 AM UTC

d--b wrote: post #28

kevinsundar wrote: post #19

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses: 1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen…

At some point one has to ask, what are you guys doing ingesting 5pb of data per day?! Unless this is google, that doesn’t make any sense. That’s an average of 7.5mb per human on the planet, every day.

Worked in a similar environment. Events and logs. When a single page view does 200+ database queries, triggers another hundred requests, ML services, analytics and other tracking, and a transaction kicks off a chain of a hundred events in your stream, it's pretty easy to reach those insane numbers. That one page view can easily add 1MB of data.

Just like performance optimization, it's cheaper to buy more hardware than to pay for humans to think about it and coordinate.

Re: Data Mesh Architecture

#38

Post by zbentley » Sun, Mar 20, 2022, 1:17 PM UTC

ako wrote: post #35

Lots of concerns and scepticism in the discussions here. Any suggestions about good, achievable data strategies and data architecture that work at enterprise level?

Require domain teams' code to communicate (with other domain teams and with the outside world) using the same pathways, schemas, and contracts that are used when extracting a domain team's data into a data lake.

Whether or not that data lake is semi-operated by the team (as proposed in the article) or operated centrally, requiring the lake's ETL process to use at least some of the APIs and tools used for transactional interaction goes a long way towards making data architecture tend towards sanity.

Resist the temptation of things like RDBMS-level CDC/log stream capture or database snapshots for populating data lakes (RDS Aurora's snapshot export/restore is like methamphetamine in this area: incredibly fast and powerful, has a very severe long term cost for data lake uniformity and usability).

I'm not saying "every row in the data lake must be extracted by making the exact same API hit that an internet user would make, with all of the overhead incurred by that". You can tap into the stack at a lower level than that (e.g. use the same DAOs that user APIs use when populating the data lake, but skip the whole web layer). Just don't tap into the lowest possible layer of the stack for data lake ETL--even though that lowest layer is probably the quickest to get working and most performant, it results in poor data hygiene over the medium and long term.

Re: Data Mesh Architecture

#39

Post by zozbot234 » Sun, Mar 20, 2022, 3:34 PM UTC

Isn't this usually called a "data mart" as opposed to "data mesh"? Or is the "mesh" term intended to point to something more unstructured, like team- or business division-level equivalent to a data lake? But isn't that just a data pond?

Re: Data Mesh Architecture

#40

Post by kevinsundar » Sun, Mar 20, 2022, 5:01 PM UTC

d--b wrote: post #28

kevinsundar wrote: post #19

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses: 1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen…

At some point one has to ask, what are you guys doing ingesting 5pb of data per day?! Unless this is google, that doesn’t make any sense. That’s an average of 7.5mb per human on the planet, every day.

I did say FAANG :)

Believe it or not this is just for security data. But that fact combined with SOA leads to lots of logs. (5Pbs is the uncompressed amount, but we decompress incoming data, then ETL).