Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

netflixtechblog.com

Post reply on HN
Print view View original

91–100 of 108 posts

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#91

Post by kiitos »

> Core business concepts like ‘actor’ or ‘movie’ are modeled in many places ... teams re-model the same business entities in different systems, leading to conflicting definitions that are hard to reconcile.

There is no singular universal consistent definition of any concept like "actor" or "movie" or whatever else. These are all concepts that are well-defined only within a specific domain. The business domain concept of an "actor" is well and good and probably the most important and top-level user-facing concept of that term -- that doesn't mean that this business-domain definition is somehow authoritative, or comprehensive, or in any way some kind of superset-composite description of any/all other domain definitions of that same term.

Reconciliation of domain-specific concepts like these requires higher-level coordination across separate domains, it's not something you can do within individual domains or domain-specific services. If discrete domains/services needed to abide the same business-defined concept of whatever concepts, then that would subvert the main purpose of having separate domains/teams in the first place.. !

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#92

Post by twodave »

bertails wrote: post #78
twodave wrote: post #21

I wonder how they deal with versioning or breaking changes to the model. One advantage of keeping things more segregated is that when you decide to change a model you can do it in much smaller pieces. I guess in their world they’d add a new model for whatever they want to change and then phase out use of the old one before removing it.

> I wonder how they deal with versioning or breaking changes to the model. Versioning is permission to break things. Although it is not currently implemented in UDA yet, the plan is to embrace the same model as Federated GraphQL, which has proved to work very well for us (think 500+ federated GraphQL schemas). In a nutshell, UDA will actively manage deprecation cycles, as we have the ability to track the consumers of…

That is a lot of subgraphs. Am I understanding correctly then that under UDA developers fulfill the UDA spec in whatever language they’re using, and then there’s some kind of middleware that will handle serving GraphQL queries? How are mutations represented? And how are other GraphQL-specific idioms expressed (like input parameters, nodes/edges/connections/etc.)? Is it just a subset of GraphQL that is supported?

I manage a much smaller federation where I work, and we have a lot of the same ideals I think in terms of having some centralized types that the rest of the business recognizes across the board. Right now we accomplish that within a set of “core” subgraphs that define these types, while our more product-focused ones implement their own sets of types, queries and mutations and can extend the core ones as it makes sense to.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#93

Post by zellyn »

This is so meta it’s hard to ground it enough to understand any of it. I’d love to see a tiny worked example of, say, a billing department and an app that represent “Customer” in different ways, and how to model each, and most importantly, what practical benefits result. Ideally, the ideas of what a “customer” is would be different enough to differ in cardinality: e.g. in Billing, a customer is a business, and in the app, a customer is something that started out 1:1 with business but that broke down over time with co-ownership, assistants performing work for owners, etc.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#95

Post by thethimble »

kiitos wrote: post #91

> Core business concepts like ‘actor’ or ‘movie’ are modeled in many places ... teams re-model the same business entities in different systems, leading to conflicting definitions that are hard to reconcile. There is no singular universal consistent definition of any concept like "actor" or "movie" or whatever else. These are all concepts that are well-defined only within a specific domain. The business domain concept…

They’re not pushing a single canonical schema.

Each team still owns its local RDF graph for concepts like actor or movie. What UDA adds is a shared graph of mappings that translate between those local models whenever another team needs them.

Traditionally such translations live in scattered adapter code, which hides lineage and adds opacity - particularly as systems proliferate. By expressing the mappings as RDF triples inside UDA’s knowledge graph, they become versioned, queryable, and reusable. No more spelunking through layers of service code to understand how one team’s actor becomes another’s.

As a result, discrete teams/domains remain independent while the interconnections/relationships become first-class and introspectable. This enables coordination without centralization.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#96

Post by bertails »

borromakot wrote: post #88
bertails wrote: post #86

Earlier quoted context omitted.

This does look interesting. Does the Ash Framework yield a knowledge graph? How good is it a cataloging existing data containers?

The concept is you model the core of your application and build it at the same time, using declarative tools, and project additions layers from this definition. The underlying data model is extendable via, well, extensions. These extend the DSL schema. It's not conceptually a knowledge graph in the same way, but you can introspect essentially everything about your application. However, resources can be given data lay…

It is important in UDA for the data models to be part of the same knowledge graph as the data container representations and the mappings, and eventually the instance data too. Our metamodel Upper is strongly inspired from RDFS, SHACL, and OWL in that respect.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#97

Post by bravesoul2 »

tough wrote: post #94
bravesoul2 wrote: post #4

Why would Netflix engineering host on Medium? Very odd. And you just lose readers to the popups but you don't benefit from their discovery much either.

Marketing department can own it and SEO

Makes sense for say a 30 person consultancy, but Netflix?!

Part of marketing is knowing your audience. And plenty of marketing people exist with deep tech experience.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#98

Post by bravesoul2 »

jawns wrote: post #13

For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…

Spitballing. Another way to deal with the problem is like what would you do if you had billions of pieces of unstructured data (except for maybe the data being somewhat XML like) and you don't control any of it but you need to make sense of it as (ignoring rounding errors) your only business concern. That company is Google of course.

Maybe let the business units be loose but make the sense making central. Any individual unit can eventually tidy things up (SEO!) but everything will work regardless. The UX effect might be you can't find something decent to watch but that is an entirely different problem solved by not using Netflix and going to the theatre!

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#99

Post by b0a04gl »

bertails wrote: post #80
b0a04gl wrote: post #37

how much of upper is actually enforced at runtime vs just used for schema generation? like if a downstream system silently breaks a semantic assumption (say, infers enum incorrectly or drops a type constraint), does uda catch that anywhere or is this trust-based across projections?

Great question. It really depends on the projection. For example, the projections to GraphQL and Java are mostly limited to what can be expressed there. But the projection to SHACL has access to all of SPARQL Constraints, which is what's used for the bootstrapping knowledge graph. We are looking into being able to do more runtime validation for data in the warehouse.

got it, thanks. makes sense that it depends on the projection target. SHACL+SPARQL seems like the strongest runtime check layer then. for projections like graphql or java where enforcement is weaker, is there any way to inject runtime guards or contract tests as part of the generated code? or is the idea to keep enforcement external and just let uda define the schema canonically?

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#100

Post by bytefish »

To me it's a business problem, a communication problem and a technical problem. You would have to solve all for a unified graph to succeed. And that's super hard.

For the communication problem, you'll need to break the "sweet isolation within autonomous teams". You'll need someone to hop between the teams and analyze their data models. It's usually not enough to share a data schema. You'll need to actually sit down with humans and ask questions.

The technical side is by far the easiest one. Share a fat schema, like Microsoft does with Microsoft Graph, and dictate all of your teams to use it. Problem solved. Yes, this requires a lot of empathy and frustration tolerance, because in my experience all software developers (me included) are drama queens.

The technical side can only be solved by dictating and enforcing it. You need management buy-in and authority for this, if you don't have the authority and all teams are still free to roam... you can come up with the best solutions, it won't work.

The hardest part is the business-side. If you have highly optimized processes people have been trained on for 20+ years, it's usually not possible to change processes and associated terminology.

So... You need 100% buy-in from decision makers, that it's useful and the cost of this monumental work is going to pay off in a lifetime.

Post reply on HN