Why would Netflix engineering host on Medium? Very odd. And you just lose readers to the popups but you don't benefit from their discovery much either.
So they don’t have to maintain it themselves
Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
71–80 of 108 posts
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#72I'm curious if anyone has seen business improvements along the lines of "this let us discover something that led to 5%+ or >$5M improvements" (percent or absolute depending on how big the company is) from these kinds of efforts?
I've been in a couple of the "we need to unify the data tables to serve everyone" exercises before decided to focus on other parts of the software stack and a lot of it just seemed like "the video game people model it differently because they're doing different analysis, and if you unify the base layer to support everybody's type of analysis, it's not going to change that there's still a bunch of independent, not-talking-to-each-other analysis going on." (This is specifically different from the much LARGER sort of problem which is more a copypasta one - Finance's accounting doesn't agree with Legal's accounting and nobody knows who's right, which is one dataset needed in multiple places, vs multiple datasets needed in different places.)
I think this mostly sidesteps that - they aren't forcing everyone to migrate to the same things, AFAICT - and is just about making it easy to access more broadly. Is that right?
And confusion-reducing definition things - "everyone uses the same official definitions for business concepts" - I'm all for. Seen a lot of that pain for sure.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#73Why would Netflix engineering host on Medium? Very odd. And you just lose readers to the popups but you don't benefit from their discovery much either.
Every time I see that hex-encoded URL, I enjoy plugging scribe.rip https://news.ycombinator.com/item?id=28838053>
https://scribe.rip/uda-unified-data-architecture-6a6aee261d8...
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#74> Once concepts are selected, Sphere walks the knowledge graph and generates SQL queries to retrieve data from the warehouse, no manual joins or technical mediation required.
If I had to guess this is how eng pitched it to the business to carve out the time to build this tooling. As with all these internally built schemas, ui’s, tooling, etc… they’re never gonna post how much this is actually used relative to the work arounds ds and eng use in their day to day.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#75I realize scale makes everything more difficult but at the end of the day, Netflix is encoding and serving several thousand videos via a CDN. It can't be this hard. There are a few statements in this that gave me pause. The core problem seems to be development in isolation. Put another way: microservices. This post hints at microservices having complete autonomy over their data storage and developing their own GraphQ…
I think they are just trying to put in place the common data model that, as you point out, they need.
(So their micro services can work together usefully and efficiently -- I would guess that currently the communication burden between microservice teams is high and still is not that effective.)
> The whole point of GraphQL is to create a unified view of something
It can do that, but that's not really the point of GraphQL.. I suppose you're saying that's how it was used as FB. That's fine, IMO, but it sounds like this NF team decided to use something more abstract for the same purpose.
I can't comment on their choices without doing a bunch more analysis, but in my own experience I've found off-the-shelf data modeling formats have too much flexibility in some places (forcing you to add additional custom controls or require certain usage patterns) and not enough in others (forcing you to add custom extensions). The nice thing about your own format is you can make it able to express everything you want and nothing you don't. And have a well-defined projection to Graphql (and sqlite and oracle and protobufs and xml and/or whatever other thing you're using).
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#76> Once concepts are selected, Sphere walks the knowledge graph and generates SQL queries to retrieve data from the warehouse, no manual joins or technical mediation required. If I had to guess this is how eng pitched it to the business to carve out the time to build this tooling. As with all these internally built schemas, ui’s, tooling, etc… they’re never gonna post how much this is actually used relative to the wor…
The price is in the 500+ domain graph services federated into our GraphQL enterprise gateway, which will all be exposed to Sphere through UDA. That's real.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#77Earlier quoted context omitted.
> Would also be more than happy to skip the whole GraphQL end of it. Netflix benefits from a large GraphQL ecosystem with federation, which is why it's so central in UDA from day 1. But adding a projection to "REST" would be very easy.
I don't doubt their yield out of GraphQL is great. Not something I'm having a need for though. I'm at the helm of the tech group at one part of dun&bradstreet so we have different challenges, unification across different borders being primary one. We manage, but the going gets tough sometimes. Described architecture of UDA certainly seems to be what it was designed to solve. I think our system is even at a perfect in…
> Of course, details on "Upper", PDM, and Sphere are well - missing, but at least I have concepts to focus on :)
Definitely coming soon ;-)
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#78I wonder how they deal with versioning or breaking changes to the model. One advantage of keeping things more segregated is that when you decide to change a model you can do it in much smaller pieces. I guess in their world they’d add a new model for whatever they want to change and then phase out use of the old one before removing it.
> I wonder how they deal with versioning or breaking changes to the model.
Versioning is permission to break things.
Although it is not currently implemented in UDA yet, the plan is to embrace the same model as Federated GraphQL, which has proved to work very well for us (think 500+ federated GraphQL schemas). In a nutshell, UDA will actively manage deprecation cycles, as we have the ability to track the consumers of the projected models.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#79For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…
IME it often comes down to "big men" issues where someone important wants the data in a certain way that is not logical or consistent so they won't let the "tech people" simple take the data and present it in a way that is logically consistent and follows best practices. They want to sit in meetings and create their own mental model monstrosity and force the devs to make it. Once that happens one time there is zero chance of the company ever having a consistent data model at any point in the future ever.
Not really a problem that can be overcome in probably 99% of companies. Lots of consultancy money to be made for the sake of ego and inflexibility though.
Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix
#80how much of upper is actually enforced at runtime vs just used for schema generation? like if a downstream system silently breaks a semantic assumption (say, infers enum incorrectly or drops a type constraint), does uda catch that anywhere or is this trust-based across projections?
Great question. It really depends on the projection. For example, the projections to GraphQL and Java are mostly limited to what can be expressed there. But the projection to SHACL has access to all of SPARQL Constraints, which is what's used for the bootstrapping knowledge graph. We are looking into being able to do more runtime validation for data in the warehouse.