Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
81–90 of 108 posts
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#82I realize scale makes everything more difficult but at the end of the day, Netflix is encoding and serving several thousand videos via a CDN. It can't be this hard. There are a few statements in this that gave me pause. The core problem seems to be development in isolation. Put another way: microservices. This post hints at microservices having complete autonomy over their data storage and developing their own GraphQ…
> The whole point of GraphQL is to create a unified view of something, not to have 23 different versions of "Movie".
GraphQL is great at federating APIs, and is a standardized API protocol. It is not a data modeling language. We actually tried really hard with GraphQL first.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#83> ... RDF ... SPARQL ... OWL ... I want to believe. (really! I think that's hugely underestimated tech).
We joke internally that Upper is like "RDF: The Good Parts".
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#84Good luck. This is not new. Back in the Enterprise OOP era, there was a fad of developing universal data entities. Everyone eventually learned that there is no such thing as a universal entity. The semantic meaning of the data model depends on the user context, not the producer context. A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. There is not even always a…
UDA does not believe in the existence of universal data entities. We embrace the idea that 2+ teams may have different opinions on how to represent the world. We are focused on the discovery of existing entities across systems and their reusability through extensibility. We believe that automation of the projections will be key for teams to align on defining some entities, where it makes sense.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#85I'm curious if anyone has seen business improvements along the lines of "this let us discover something that led to 5%+ or >$5M improvements" (percent or absolute depending on how big the company is) from these kinds of efforts? I've been in a couple of the "we need to unify the data tables to serve everyone" exercises before decided to focus on other parts of the software stack and a lot of it just seemed like "the…
> "the video game people model it differently because they're doing different analysis, and if you unify the base layer to support everybody's type of analysis, it's not going to change that there's still a bunch of independent, not-talking-to-each-other analysis going on"
This resonates. Moreover, it's very easy for architects to assume that because different areas of the business use data about the 'same' thing, the thing must be the same.
But often the analysis requires a slightly different thing. Like: we want a master list of prisons. But is a prison a building, a collection of prisoners (such that the male prison and the female prison on the same site are different prisons), or the institution with that name managed under a particular contract?
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#86https://ash-hq.org > Model your domain, derive the rest Been doing this for 5+ years.
This does look interesting. Does the Ash Framework yield a knowledge graph? How good is it a cataloging existing data containers?
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#87Earlier quoted context omitted.
Alternatively, the process of defining these global definitions exposes exactly this conflict and leads to common definitions of "Files", "Contracts" and "Watchables" instead of 3 conflicting definitions of "Movies"?
The conflict will definitely help define the terms. Maybe they will all choose "Movie", maybe not. Just there is no universally ideal term that represents a concept for all users for all time. It's a common error to seek such universal definitions.
Exactly. In UDA, each Movie entity belongs to a specific business domain. Universality isn't an inherent truth, it's a social alignment within a group, useful only to the extent that it helps solve shared problems.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#88https://ash-hq.org > Model your domain, derive the rest Been doing this for 5+ years.
This does look interesting. Does the Ash Framework yield a knowledge graph? How good is it a cataloging existing data containers?
The concept is you model the core of your application and build it at the same time, using declarative tools, and project additions layers from this definition. The underlying data model is extendable via, well, extensions. These extend the DSL schema.
It's not conceptually a knowledge graph in the same way, but you can introspect essentially everything about your application. However, resources can be given data layers which define how they map to underlying storage, and you could use all of this information only as static information to derive additional things from, or you could just...well, use it. i.e `Ash.read(Resource)` yielding the table data. Our query engine has the same semantics they describe where you don't explicitly join etc.
```elixir MyApp.Post |> Ash.Query.filter(author.type == :admin) |> Ash.read!() ```
You can generate charts and graphs, including things like policy flow charts.
---
Ultimately I've found that modeling tools like UML that can't simultaneously actually execute that model (i.e act as the application itself) are always insufficient and/or have massive impedance mismatches once rubber meets the road. The point is to effectively reimagine this as "what if we use these modeling principles, declaratively, from the ground up".
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#89Earlier quoted context omitted.
This does look interesting. Does the Ash Framework yield a knowledge graph? How good is it a cataloging existing data containers?
The concept is you model the core of your application and build it at the same time, using declarative tools, and project additions layers from this definition. The underlying data model is extendable via, well, extensions. These extend the DSL schema. It's not conceptually a knowledge graph in the same way, but you can introspect essentially everything about your application. However, resources can be given data lay…
Factor in that building essentially any server-side tooling without Elixir (BEAM) is a bad idea in my view, you end up with "lets just make this the way we build apps, and do it in Elixir". It's been very powerful and we're continuing to progress on it.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#90[deleted]