Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

netflixtechblog.com

Post reply on HN
Print view View original

31–40 of 108 posts

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#31

Post by bob1029 »

This kind of problem could be made a lot more straightforward if we separate the schema owner (i.e., the business) from the rest of the stack. Some major SQL engines have this role built-in. Whatever you want to call it - "premature" optimization, etc. - the act of simultaneously trying to optimize while you build is perhaps important, but otherwise very disruptive to the creative exercise of naming things and relating them together (domain modeling).

When your brain is constantly locked into big-O notation and you are only worrying about N being larger than a billion, it becomes really easy to justify running a high quality representation of the domain into the dirt over arbitrary performance concerns. E.g., storing a bunch of tiny fields in one JSON blob column is going to be faster for many cases, but it totally screws up downstream use cases by making custom views of the data more expensive. The query of concern might only hit once a day, but the developers probably aren't thinking at that level of detail.

The really tragic part is that the modern RDBMS is typically capable of figuring out acceptable query plans even given the most pathetically naive domain models. I think in general there is a severe (and growing) misunderstanding regarding what something like MSSQL/Oracle/DB2 can accomplish - even in an enterprise as large as Netflix.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#32

Post by rorylaitila »

Good luck. This is not new. Back in the Enterprise OOP era, there was a fad of developing universal data entities. Everyone eventually learned that there is no such thing as a universal entity. The semantic meaning of the data model depends on the user context, not the producer context. A "Movie" is not the same thing to the Finance team, Acquisition team, Infrastructure team, or Customer. There is not even always a common identifier, let alone common fields, let alone common meaning of the fields.

Edit: The more I read this article the more I hear this voice https://www.youtube.com/watch?v=y8OnoxKotPQ

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#33

Post by dboreham »

jawns wrote: post #13

For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…

Reminds me of my experience trying to understand what SAP actually is. For decades I wondered what sort of magic tech must be in there that allowed their software to be used by thousands of different businesses. Then someone who knew about SAP told me: "oh, no that's not how it works -- what they do is have a fixed schema and tell the customer that they must adopt it".

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#34

Post by thefourthchime »

jawns wrote: post #13

For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…

sometimes grug go too early and get abstractions wrong, so grug bias towards waiting

big brain developers often not like this at all and invent many abstractions start of project

grug tempted to reach for club and yell "big brain no maintain code! big brain move on next architecture committee leave code for grug deal with!"

but grug learn control passions, major difference between grug and animal

instead grug try to limit damage of big brain developer early in project by giving them thing like UML diagram (not hurt code, probably throw away anyway) or by demanding working demo tomorrow

working demo especially good trick: force big brain make something to actually work to talk about and code to look at that do thing, will help big brain see reality on ground more quickly

remember! big brain have big brain! need only be harness for good and not in service of spirit complexity demon on accident, many times seen

https://grugbrain.dev/#grug-on-complexity

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#37

Post by b0a04gl »

how much of upper is actually enforced at runtime vs just used for schema generation? like if a downstream system silently breaks a semantic assumption (say, infers enum incorrectly or drops a type constraint), does uda catch that anywhere or is this trust-based across projections?

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#38

Post by wjnc »

jawns wrote: post #13

For all the benefits, there is a large problem with this approach that often goes unacknowledged. It is fundamentally a business problem, rather than a technical problem, but it has impact on development speed, so it's secondarily a technical problem. The business contract with a consolidated data definition is that everyone in the business, no matter which domain, can rely on it. But think about the red tape that in…

Data drift is real! I’ve recently restored sanity in a medium sized enterprise where there were three concurrent financial data flows. Including people not understanding each other, projects to find out ground truth and triple the workload in maintaining the dataflows. I’ve quipped to the team that endless summer is near. What if we only work on business relevant development. I would dream that the bigcorp we are part of would do the same. They are more of a tack on another Excel based solution kind of firm.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#39

Post by regularfry »

bertylicious wrote: post #22

How does this relate to domain-driven design? It seems to be at odds with it, because in DDD it's kind of expected that the same concept will be represented in a different way by each system? But to be honest, I didn't read the whole blog post because of the UML vibes.

It doesn't. It's a blessing that they avoided the term "ubiquitous language" because that's almost exactly the dual of this concept, although people who have only ever heard the words and not dug any deeper won't know what the difference is.

Re: Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

#40

Post by regularfry »

praveen9920 wrote: post #15

Main challenge with this approach is change management of models scheme. Apart from Consensus for updating schema, maintaining versioned models across services becomes a challenge. Let’s say someone deprecates a field in schema, all services needs to update the business logic based on that which is challenging and against the ethos of distributed services.

It's a challenge but in principle it's doable with contract testing, in the style of Pact, where there's a contract broker that disparate services all coordinate through. If you've got that, you can publish your new model version as a new contract version, and everyone can see immediately where their APIs need to change. Contracts do get a passing mention in the article, but it's not a focus.
Post reply on HN