Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

netflixtechblog.com

Post reply on HN
Print view View original

41–50 of 108 posts

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#41

Post by andsoitis »

rorylaitila wrote: post #32

Good luck. This is not new. Back in the Enterprise OOP era, there was a fad of developing universal data entities. Everyone eventually learned that there is no such thing as a universal entity. The semantic meaning of the data model depends on the user context, not the producer context. A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. There is not even always a…

> A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer.

Shouldn’t it be?

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#42

Post by jaakl »

It seems to be based on very common naive belief that things which are named same or similar in different domains are conceptually same, so "lets deduplicate" ? There can be rare moments when they really are, but then the moment passes and then you only have troubles.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#43

Post by bertails »

jawns wrote: post #13

For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…

> It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem.

Yes it is a "fundamentally a business problem" but we believe it can be solved with technology. We think we have a more systematic way to adopt and deploy model-first knowledge graphs in the enterprise.

> But think about the red tape that introduces.

We are very intentional about UDA not becoming more red tape. UDA lives alongside all the other systems. There will never be a mandate for everything to be in UDA.

But we sure want to make it easy for those teams who wants their business models to exist everywhere, to be connected to the business, and to make it easy to be discovered, extended, and linked to.

(I'm one of UDA's architects.)

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#44

Post by cletus »

I realize scale makes everything more difficult but at the end of the day, Netflix is encoding and serving several thousand videos via a CDN. It can't be this hard. There are a few statements in this that gave me pause.

The core problem seems to be development in isolation. Put another way: microservices. This post hints at microservices having complete autonomy over their data storage and developing their own GraphQL models. The first is normal for microservices (but an indictment at the same time). The second is... weird.

The whole point of GraphQL is to create a unified view of something, not to have 23 different versions of "Movie". Attributes are optional. Pull what you need. Common subsets of data can be organized in fragments. If you're not doing that, why are you using GraphQL?

So I worked at Facebook and may be a bit biased here because I encountered a couple of ex-Netflix engineers in my time who basically wanted to throw away FB's internal infrastructure and reinvent Netflix microservices.

Anyway, at FB there a Video GraphQL object. There aren't 23 or 7 or even 2.

Data storage for most things was via write-through in-memory graph database called TAO that persisted things to sharded MySQL servers. On top of this, you'd use EntQL to add a bunch of behavior to TAO like permissions, privacy policies, observers and such. And again, there was one Video entity. There were offline data pipelines that would generally process logging data (ie outside TAO).

Maybe someone more experienced with microservices can speak to this: does UDA make sense? Is it solving an actual problem? Or just a self-created problem?

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#45

Post by Keyframe »

Having dealt with same problems for years now (we call our UDM - Unified Data Model, heh), I was under the impression this was an over-engineered Datamart++; It's not though. Calling UDA a datamart would be like calling K8S a bash script, which might be related but wildly different in scope.

I am definitely interested to read more and implement it myself as well. Would also be more than happy to skip the whole GraphQL end of it.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#47

Post by cush »

jawns wrote: post #13

For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…

> It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem.

It doesn't read from the article that they are denying that it's a business problem. The models they're defining seem to span all roles, engineering being only one.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#48

Post by rorylaitila »

andsoitis wrote: post #41
rorylaitila wrote: post #32

Good luck. This is not new. Back in the Enterprise OOP era, there was a fad of developing universal data entities. Everyone eventually learned that there is no such thing as a universal entity. The semantic meaning of the data model depends on the user context, not the producer context. A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. There is not even always a…

> A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. Shouldn’t it be?

No, because context and use defines the meaning. To the data team, a "Movie" might mean a file on disk. To the finance team, a "Movie" might mean a contract to a studio. To the Customer, a "Movie" is something they watch. That each of these contexts can use the term "Movie" does not actually mean they share anything in common. We could have called them "Files", "Contracts" and "Watchables" instead.

When people embark on 'universal' data definitions, conversations of the type "But is it reaaalllly a Movie??" are an endless source of confusion.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#49

Post by buster »

andsoitis wrote: post #41
rorylaitila wrote: post #32

Good luck. This is not new. Back in the Enterprise OOP era, there was a fad of developing universal data entities. Everyone eventually learned that there is no such thing as a universal entity. The semantic meaning of the data model depends on the user context, not the producer context. A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. There is not even always a…

> A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. Shouldn’t it be?

No, why would the finance team care for the cover of a movie or the available subtitles? If everyone would have the same definition, changing some thing about a movie will need a change in every consumer who doesn't actually care.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#50

Post by bertails »

jfengel wrote: post #26
jawns wrote: post #13

For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…

I tried, for some time, to develop a product designed to solve this. It would have made it easier to specialize models locally while complying with the corporate one. (Basically, beefing up the data definition language to something like prolog, and putting real thought into making the corporate model reality-based rather than just what suits your current requirements.) Unfortunately it came about at exactly the same…

UDA embraces the duplication of models: it's a fact of life in the enterprise. That is why "domains" are first-class citizen. We believe that good discovery capabilities will increase reusability of the domain models. Our next article will dive more into the extensibility capabilities of the metamodel Upper.
Post reply on HN