[deleted]
Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
61–70 of 108 posts
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#62Main challenge with this approach is change management of models scheme. Apart from Consensus for updating schema, maintaining versioned models across services becomes a challenge. Let’s say someone deprecates a field in schema, all services needs to update the business logic based on that which is challenging and against the ethos of distributed services.
It's still distributed services in the service of one entity. So why is something deprecated without a clear plan what existing users will do?
To me feels related to the monorepo or not discussions?
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#63When translating from French to English, find someone that speaks both fluently and had domain expertise over the content being translated.
Don't find a linguist who understands grammatical structure and claims to be able to map the source language to some neutral intermediate structure and map that to the target language.
This is a fallacy I notice everywhere but I dont know how to name. Maybe the "Linguist translator" fallacy?
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#64For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…
Reminds me of my experience trying to understand what SAP actually is. For decades I wondered what sort of magic tech must be in there that allowed their software to be used by thousands of different businesses. Then someone who knew about SAP told me: "oh, no that's not how it works -- what they do is have a fixed schema and tell the customer that they must adopt it".
Epic EMR is the same. But then some hospitals insist on customizing it which causes no end of problems.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#65Earlier quoted context omitted.
> A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. Shouldn’t it be?
No, because context and use defines the meaning. To the data team, a "Movie" might mean a file on disk. To the finance team, a "Movie" might mean a contract to a studio. To the Customer, a "Movie" is something they watch. That each of these contexts can use the term "Movie" does not actually mean they share anything in common. We could have called them "Files", "Contracts" and "Watchables" instead. When people embark…
Alternatively, the process of defining these global definitions exposes exactly this conflict and leads to common definitions of "Files", "Contracts" and "Watchables" instead of 3 conflicting definitions of "Movies"?
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#66For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…
> It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. Yes it is a "fundamentally a business problem" but we believe it can be solved with technology. We think we have a more systematic way to adopt and deploy model-first knowledge graphs in the enterprise. > But think about the red tape that introduces. We are very i…
How can it be universal if everything isn't in UDA?
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#67For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…
Data drift is real! I’ve recently restored sanity in a medium sized enterprise where there were three concurrent financial data flows. Including people not understanding each other, projects to find out ground truth and triple the workload in maintaining the dataflows. I’ve quipped to the team that endless summer is near. What if we only work on business relevant development. I would dream that the bigcorp we are par…
Data drift is real, and the yoke of governance chafes enough that new people insist on redoing your work in excel until the problem gets bad enough that a new data governance push is needed.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#68Reminds me of the work done at Uber with Dragon
https://www.uber.com/blog/dragon-schema-integration-at-uber-...
Unfortunately it never got open sourced but Joshua left for LinkedIn and started working on the LambdaGraph project and the Hydra language that are open sourced.
You can find more information on this fascinating work here:
https://github.com/CategoricalData/hydra
I think these approaches, including all the semantic web stuff from 10+ years ago, suffered from the added overhead of agreeing and formalising semantics and then of course maintaining them.
I wonder if LLMs can help with that part today.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#69Earlier quoted context omitted.
No, because context and use defines the meaning. To the data team, a "Movie" might mean a file on disk. To the finance team, a "Movie" might mean a contract to a studio. To the Customer, a "Movie" is something they watch. That each of these contexts can use the term "Movie" does not actually mean they share anything in common. We could have called them "Files", "Contracts" and "Watchables" instead. When people embark…
Alternatively, the process of defining these global definitions exposes exactly this conflict and leads to common definitions of "Files", "Contracts" and "Watchables" instead of 3 conflicting definitions of "Movies"?
The conflict will definitely help define the terms. Maybe they will all choose "Movie", maybe not. Just there is no universally ideal term that represents a concept for all users for all time. It's a common error to seek such universal definitions.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#70It's been so long since the Semantic web and RDF and OWL and SKOS. I'm so glad they stuck with W3C and didn't reinvent those wheels. Will this UDA approach catch on? I don't know, but I hope so. It seems like it is trying to move the frontier of the difficulties of applying Domain Driven Design and semantic concepts to an enterprise company of significant scale.
If we can get compound interest across development teams by giving them a common toolset and skillset that covers different applications but the same data semantics, maybe not every data contract will have to be reduced to DTOs that can be POSTed or otherwise forced to be a least common denominator just so it can fit past a network or other IPC barrier.
For that, I'm grateful Netflix is working on this and publicizing the interesting work.