Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

netflixtechblog.com

Post reply on HN
Print view View original

51–60 of 108 posts

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#51

Post by twodave »

cletus wrote: post #44

I realize scale makes everything more difficult but at the end of the day, Netflix is encoding and serving several thousand videos via a CDN. It can't be this hard. There are a few statements in this that gave me pause. The core problem seems to be development in isolation. Put another way: microservices. This post hints at microservices having complete autonomy over their data storage and developing their own GraphQ…

I totally agree. Especially with Fusion it’s very easy to establish core types in self-contained subgraphs and then extend those types in domain-specific subgraphs. IMO the hardest part about this approach is just namespacing all the things, because GraphQL doesn’t have any real conventions for organizing service- (or product-) specific types.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#52

Post by bertails »

Keyframe wrote: post #45

Having dealt with same problems for years now (we call our UDM - Unified Data Model, heh), I was under the impression this was an over-engineered Datamart++; It's not though. Calling UDA a datamart would be like calling K8S a bash script, which might be related but wildly different in scope. I am definitely interested to read more and implement it myself as well. Would also be more than happy to skip the whole GraphQ…

> Would also be more than happy to skip the whole GraphQL end of it.

Netflix benefits from a large GraphQL ecosystem with federation, which is why it's so central in UDA from day 1. But adding a projection to "REST" would be very easy.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#53

Post by bertails »

bertylicious wrote: post #22

How does this relate to domain-driven design? It seems to be at odds with it, because in DDD it's kind of expected that the same concept will be represented in a different way by each system? But to be honest, I didn't read the whole blog post because of the UML vibes.

> How does this relate to domain-driven design?

The "Domain" in `upper:DomainModel` is the same D as in DDD (Domain-Driven Design) as the D in DGS (Domain Graph Service).

> in DDD it's kind of expected that the same concept will be represented in a different way by each system

In UDA, those concepts would explicitly co-exist in different domains. "Being the same" becomes a subjective thing.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#54

Post by cush »

cletus wrote: post #44

I realize scale makes everything more difficult but at the end of the day, Netflix is encoding and serving several thousand videos via a CDN. It can't be this hard. There are a few statements in this that gave me pause. The core problem seems to be development in isolation. Put another way: microservices. This post hints at microservices having complete autonomy over their data storage and developing their own GraphQ…

>at the end of the day, Netflix is encoding and serving several thousand videos via a CDN. It can't be this hard

Yeah maybe 10 years ago, but today Netflix is one of the top production companies on the planet. In the article, they even point to how this addresses their issues in content engineering

https://netflixtechblog.com/netflix-studio-engineering-overv...

https://netflixtechblog.com/globalizing-productions-with-net...

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#55

Post by detaro »

jaakl wrote: post #42

It seems to be based on very common naive belief that things which are named same or similar in different domains are conceptually same, so "lets deduplicate" ? There can be rare moments when they really are, but then the moment passes and then you only have troubles.

To me the motivation seems more along the lines of "we build lots of different systems that deal in the same domains" (because they are deep in microservice land, have apps for all kinds of platforms, ...) "lets make sure they all use the same definition of the things". Do you think that doesn't make sense (because each of those should be considered their own domain?) or does something else give you your impression?

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#57

Post by mkoubaa »

tomrod wrote: post #18
jawns wrote: post #13

For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…

Corolloray to Hyrum's Law then. Perhaps we call it "Orange is the New Model" Law

Love it

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#58

Post by Keyframe »

bertails wrote: post #52
Keyframe wrote: post #45

Having dealt with same problems for years now (we call our UDM - Unified Data Model, heh), I was under the impression this was an over-engineered Datamart++; It's not though. Calling UDA a datamart would be like calling K8S a bash script, which might be related but wildly different in scope. I am definitely interested to read more and implement it myself as well. Would also be more than happy to skip the whole GraphQ…

> Would also be more than happy to skip the whole GraphQL end of it. Netflix benefits from a large GraphQL ecosystem with federation, which is why it's so central in UDA from day 1. But adding a projection to "REST" would be very easy.

I don't doubt their yield out of GraphQL is great. Not something I'm having a need for though. I'm at the helm of the tech group at one part of dun&bradstreet so we have different challenges, unification across different borders being primary one. We manage, but the going gets tough sometimes. Described architecture of UDA certainly seems to be what it was designed to solve. I think our system is even at a perfect inflection point to adopt at least some of the principles described to provide a clear path forward to resolve some of those challenges we face; Not as a replacement, but more of as a control plane over our system. I can already see how we could avoid at least schema bloat, lowest common denominator fields and overall rigidity.

Of course, details on "Upper", PDM, and Sphere are well - missing, but at least I have concepts to focus on :)

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#59

Post by mkoubaa »

rorylaitila wrote: post #32

Good luck. This is not new. Back in the Enterprise OOP era, there was a fad of developing universal data entities. Everyone eventually learned that there is no such thing as a universal entity. The semantic meaning of the data model depends on the user context, not the producer context. A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. There is not even always a…

Wittgenstein sends his regards

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#60

Post by mkoubaa »

andsoitis wrote: post #41
rorylaitila wrote: post #32

Good luck. This is not new. Back in the Enterprise OOP era, there was a fad of developing universal data entities. Everyone eventually learned that there is no such thing as a universal entity. The semantic meaning of the data model depends on the user context, not the producer context. A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. There is not even always a…

> A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. Shouldn’t it be?

A unique identifier for a movie is the same thing, like an ISBN number. What the label means in each area is going to be different. That said, some things like "director", "budget" are immutable properties of a movie but are absolutely irrelevant for the business areas and the duplication of these properties in different domains is fundamentally not that big of a deal
Post reply on HN