Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Mesh Architecture

datamesh-architecture.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Data Mesh Architecture

#11

Post by gxt »

i_like_waiting wrote: post #8
gxt wrote: post #3

Earlier quoted context omitted.

The chief data officer in close collaboration with the chief data engineering officer must elaborate automated normalization guidelines backed with implementations used across all data streams to insure any skew in the data model is limited to non production environments and all data entities are materialized consistently across the whole data model.

what type of company you are working for? Usually there is not even CIO, I haven't even heard about company with both CDO and CDEO (or even CDEO itself). I thought big portion of need that data mesh fills is the organizations who are missing resources in their core BI team.

There's no magic. you need a core team that pivots from writing code at O(n) cost enterprise wide to more or less amortized O(1) where n is the amount of work required to process a new data stream - ie having to write code once per stream vs once for a standardized stream that gets reused. With only datamesh I don't think it's going to work but with standardized tools that allow your teams to write transformations and code as data then every team effectively gets access to a self-service data warehouse with only access to pre-approved happy paths that can be automatically monitored for the most part. That's where you gain in efficiency and can let your BI teams focus on BI and not boilerplate code, infrastructure, conformity, etc.

Re: Data Mesh Architecture

#12

Post by MikeDelta »

rad_gruchalski wrote: post #7
sdze wrote: post #6

If you need so many "slides" to persuade your clients of something, I think you lost already.

Considering how many big companies go about implementing this right now, I don’t agree. C line likes slides.

Indeed, the Future State Architecture documentation from the central architects that I have seen were all powerpoint presentations with at least 100 slides.

Re: Data Mesh Architecture

#13

Post by edmundsauto »

i_like_waiting wrote: post #9
robertlagrant wrote: post #4

It really feels like data mesh is a fairly half baked concept born out of short term consulting gigs and a desire to become a technical thought leader.

Reminds me of first OLAP cubes a lot, something that consultant online praise as much as possible, just so then 3-4 years later they are contracted by the company to fix the mess it created.

What are the downsides of OLAP cubes, and how were they fixed? Curious to level up my understanding.

Re: Data Mesh Architecture

#14

Post by LaserToy »

I looks like a weird attempt to build a consulting business around a simple idea.

Treat data assets like micro services and pipelines like network. Period.

Prescribing everything else rubs me wrong way.

So, data mesh is: architecture in which data in the company organized in loosely coupled data assets.

Re: Data Mesh Architecture

#15

Post by i_like_waiting »

gxt wrote: post #11
i_like_waiting wrote: post #8

Earlier quoted context omitted.

what type of company you are working for? Usually there is not even CIO, I haven't even heard about company with both CDO and CDEO (or even CDEO itself). I thought big portion of need that data mesh fills is the organizations who are missing resources in their core BI team.

There's no magic. you need a core team that pivots from writing code at O(n) cost enterprise wide to more or less amortized O(1) where n is the amount of work required to process a new data stream - ie having to write code once per stream vs once for a standardized stream that gets reused. With only datamesh I don't think it's going to work but with standardized tools that allow your teams to write transformations an…

Yes, its similar path that I am taking (while leading BI in my org.) Having first sights of self-service from analysis perspective is super easy thanks to tools like metabase.

For bringing data in, thats completely different story, especially in non-tech organizations. The gap between how power user from specific department and somebody from my team brings and transforms data is still too big and somehow hard to enforce (following naming conventions, keeping same data formats for same columns, lowercasing certain columns, so joins are done correctly...). They usually have their "playground schemas" they use, but its very far from saying that they "own" data quality there.

Re: Data Mesh Architecture

#16

Post by tremoloqui »

i_like_waiting wrote: post #8
gxt wrote: post #3

Earlier quoted context omitted.

The chief data officer in close collaboration with the chief data engineering officer must elaborate automated normalization guidelines backed with implementations used across all data streams to insure any skew in the data model is limited to non production environments and all data entities are materialized consistently across the whole data model.

what type of company you are working for? Usually there is not even CIO, I haven't even heard about company with both CDO and CDEO (or even CDEO itself). I thought big portion of need that data mesh fills is the organizations who are missing resources in their core BI team.

A data mesh approach probably wouldn't work in the sort of organization you describe.

IMO - To make it work you need a consistent taxonomy or way of translating from a particular domain to some sort of interchange format.

If you have that then a set of centralized tools can pull from the separate domains using a core set of protocols to produce reports etc.

Re: Data Mesh Architecture

#17

Post by skrrr »

i_like_waiting wrote: post #10

So if I understand this correctly, data mesh is just data mart, that doesn't bring data in database as a table, but uses S3 storage instead (I assume because thats cheaper in the cloud?)

That + a central data platform team that provides infra, quality monitors, data lineage and catalogue capabilities + a central team that provides guidelines on SLAs, metadata standards etc. Sounds good in theory, I am eager to see how it fails in practice

Re: Data Mesh Architecture

#18

Post by i_like_waiting »

edmundsauto wrote: post #13
i_like_waiting wrote: post #9

Earlier quoted context omitted.

Reminds me of first OLAP cubes a lot, something that consultant online praise as much as possible, just so then 3-4 years later they are contracted by the company to fix the mess it created.

What are the downsides of OLAP cubes, and how were they fixed? Curious to level up my understanding.

I guess they had their place in some point and time, but I still vividly remember my old manager speaking about building OLAP cube in 2018. https://www.holistics.io/blog/the-rise-and-fall-of-the-olap-...

Re: Data Mesh Architecture

#19

Post by kevinsundar »

Wow this is an oversimplification. I've had years of experience working in a data lake within a FAANG handling > 5 PBs of data per day ingest. There's so many things this misses:

1. What if the domain teams don't actually care to maintain data quality or even care about sharing data in the first place? This model requires every data producer to maintain a relationship with every data consumer. That's not gonna happen in a large company.

2. Who pays for query compute and data storage when you're dealing with petabytes and petabytes of data from different domains? If you (the data platform team) bill the domain teams then see above, they'll just stop sending data.

3. Just figuring out what data exists in the data mart (which this essentially is describing) is a hassle and slows down business use cases, especially when you have 1000s of datasets. You need a team to act as sort of a "reference librarian" to help those querying data. You can't easily decentralize this.

4. How do you get domain teams to produce data in a form that is easy to query? Like what if they write lots of small files that are computationally difficult to query, whose gonna advise them? Data production is very related to data query performance at TBs scale. The domain team is not gonna become experts or care.

5. What do you do when a domain team has a lot of important data but no engineering resources? Do you just say "oh well, we're just a self-service data platform so no one gets to access the data"?

Re: Data Mesh Architecture

#20

Post by cwp »

Dunno about the merits of this, but it does seem to be part of the overall effort to rethink how to organize large groups of people working together. With the internet we can afford peer-to-peer communication, and we don't have to organize into hierarchies. But we can't just do full-mesh communication either, because that's overwhelming to individuals, as anyone who lived through the initial slack-and-zoom remote work of early 2020 can tell you. (Though lots of people are still living through it, unfortunately)

So what kind of communication structures are good, and in what circumstances? How do we structure work so that we don't have to communicate about everything? When do we fall back to ad-hoc video chat or even in-person meetings? These are the kinds of questions that 21st-century management has to answer. It's fascinating to watch people grapple with them.

Post reply on HN