Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

netflixtechblog.com

Post reply on HN
Print view View original

101–108 of 108 posts

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#101

Post by eggsby »

regularfry wrote: post #39
bertylicious wrote: post #22

How does this relate to domain-driven design? It seems to be at odds with it, because in DDD it's kind of expected that the same concept will be represented in a different way by each system? But to be honest, I didn't read the whole blog post because of the UML vibes.

It doesn't. It's a blessing that they avoided the term "ubiquitous language" because that's almost exactly the dual of this concept, although people who have only ever heard the words and not dug any deeper won't know what the difference is.

Seems to be enforcing ‘ubiquitous language’ at the machine level - not some kind of mathematical dual where one is invertible to the other - but enforcing soft skills as hard skills.

  ‘protobuf specs dont have enough information for us to codegen iceberg tables so we will write a new codegen spec language’
what makes a duck a duck? when we know which tables we can find it in

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#102

Post by bertails »

b0a04gl wrote: post #99
bertails wrote: post #80

Earlier quoted context omitted.

Great question. It really depends on the projection. For example, the projections to GraphQL and Java are mostly limited to what can be expressed there. But the projection to SHACL has access to all of SPARQL Constraints, which is what's used for the bootstrapping knowledge graph. We are looking into being able to do more runtime validation for data in the warehouse.

got it, thanks. makes sense that it depends on the projection target. SHACL+SPARQL seems like the strongest runtime check layer then. for projections like graphql or java where enforcement is weaker, is there any way to inject runtime guards or contract tests as part of the generated code? or is the idea to keep enforcement external and just let uda define the schema canonically?

We have a little bit of runtime checks in the Java projections but it's still tied to SHACL/SPARQL through the Jena library. We are exploring ways to keep using SHACL at scale for advanced data profiling, and also try to identify a subset of SHACL constraints which would be able to compile down to SQL constraints for validating directly against the data warehouse.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#103

Post by kiitos »

thethimble wrote: post #95
kiitos wrote: post #91

> Core business concepts like ‘actor’ or ‘movie’ are modeled in many places ... teams re-model the same business entities in different systems, leading to conflicting definitions that are hard to reconcile. There is no singular universal consistent definition of any concept like "actor" or "movie" or whatever else. These are all concepts that are well-defined only within a specific domain. The business domain concept…

They’re not pushing a single canonical schema. Each team still owns its local RDF graph for concepts like actor or movie. What UDA adds is a shared graph of mappings that translate between those local models whenever another team needs them. Traditionally such translations live in scattered adapter code, which hides lineage and adds opacity - particularly as systems proliferate. By expressing the mappings as RDF trip…

> What UDA adds is a shared graph of mappings that translate between those local models whenever another team needs them

Every such mapping is a composite definition, composed over two discrete concepts, each of which is owned/managed by discrete teams. Who gets to define/own/maintain these composite mappings? They're separate concepts, "on top of" the things they map, meaning they only exist in terms of specific versions of the underlying entities. It's just not as simple as it seems, or as is being implied here...

> As a result, discrete teams/domains remain independent while the interconnections/relationships become first-class and introspectable. This enables coordination without centralization.

What's being described here is for sure a centralized collection of definitions of mappings between interconnections/relationships.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#104

Post by regularfry »

eggsby wrote: post #101
regularfry wrote: post #39

Earlier quoted context omitted.

It doesn't. It's a blessing that they avoided the term "ubiquitous language" because that's almost exactly the dual of this concept, although people who have only ever heard the words and not dug any deeper won't know what the difference is.

Seems to be enforcing ‘ubiquitous language’ at the machine level - not some kind of mathematical dual where one is invertible to the other - but enforcing soft skills as hard skills. ‘protobuf specs dont have enough information for us to codegen iceberg tables so we will write a new codegen spec language’ what makes a duck a duck? when we know which tables we can find it in

Except that "Ubiquitous Language" is supposed to refer to terminology within a specific Bounded Context. In DDD it is desirable and expected that there is a mapping between them. This proposal tries to entirely erase Bounded Contexts. This is what I mean about people not understanding the words.

So in the sense of "what do we do about terminology not matching across an organisation" this and DDD are literal opposite solutions: one says "erase differences with a central definition (and bear the coordination costs)" while the other says "encourage differences with local definitions (and bear the mapping costs)".

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#105

Post by bitwize »

In Milt and Tim Bryce's PRIDE system, one of the first steps in designing a system for an organization is adopting organization-wide standards and vocabulary. This includes cataloguing every piece of data recorded by the system, including its name, type, and relevance in a data dictionary.

In PRIDE/ASDM, literally every field in every table is catalogued online in this way, where it's searchable and discoverable by anyone working on the system.

PRIDE has been around since the seventies. And Netflix is discovering unification of data representations across the company just now? That's what happens when you let programmers design your systems, instead of systems analysts.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#106

Post by osi »

regularfry wrote: post #104
eggsby wrote: post #101

Earlier quoted context omitted.

Seems to be enforcing ‘ubiquitous language’ at the machine level - not some kind of mathematical dual where one is invertible to the other - but enforcing soft skills as hard skills. ‘protobuf specs dont have enough information for us to codegen iceberg tables so we will write a new codegen spec language’ what makes a duck a duck? when we know which tables we can find it in

Except that "Ubiquitous Language" is supposed to refer to terminology within a specific Bounded Context. In DDD it is desirable and expected that there is a mapping between them. This proposal tries to entirely erase Bounded Contexts. This is what I mean about people not understanding the words. So in the sense of "what do we do about terminology not matching across an organisation" this and DDD are literal opposite…

UDA enables both approaches. each has its place.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#107

Post by hadjian »

Thank you bertails for the post. I work in IoT and we are also moving in that direction. I was of course hesitant, because of the criticism of semantic web on the ...web ;-), but this approach is used very successfully in our market for many years:

- https://haxall.io/ basically a reinvented copy of sem-web, but so much nicer from a dev perspective

- Used by https://skyfoundry.com/product

- https://brickschema.org/ the new star, built on sem-web

- Used by https://www.mapped.com/ where one of the founders worked on Brick

- https://docs.open223.info also a model that has caught interest in the market

- https://qudt.org/ de-facto standard for units and conversions

We started just a couple of months ago, but I have not regretted the decision, yet.

The downside of a lesser known tech-stack is always the missing OSS tools, e.g. for authoring, ownership management, review workflows etc. Counting on you guys to open source some good stuff ;-)

When we are further down the road, I am interested to get in touch with you guys, if you are up for it?

edit: forgot the ontology part of the haxall project.

- https://www.project-haystack.org/doc/index

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#108

Post by regularfry »

osi wrote: post #106
regularfry wrote: post #104

Earlier quoted context omitted.

Except that "Ubiquitous Language" is supposed to refer to terminology within a specific Bounded Context. In DDD it is desirable and expected that there is a mapping between them. This proposal tries to entirely erase Bounded Contexts. This is what I mean about people not understanding the words. So in the sense of "what do we do about terminology not matching across an organisation" this and DDD are literal opposite…

UDA enables both approaches. each has its place.

It may enable both approaches, but the article positions only one. I mean, the title is "Model Once, Represent Everywhere".
Post reply on HN