Live data from Hacker News

Principles of Data Oriented Programming

blog.klipse.tech

91–100 of 139 posts

Re: Principles of Data Oriented Programming

#91
post #66

Earlier quoted context omitted.

Even with this type of design, it's possible to implement in a typesafe way. I have seen clever ECS systems accomplish this

Maybe but ECS does in some sense circumvent types.

In some sense. You could build a type system around "has a " relationships instead of "is a " and get something pretty cohesive out that closely aligns with ECS.

Re: Principles of Data Oriented Programming

#92
post #72

I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space. The big one: "Data is immutable". The problem here is that data isn't actually immutable (generally) and mutability isn't actually the problem. The problem is unmanaged references or other dependencies on the mutable data. The "source of truth" becomes muddled whi…

Honestly, I think this series of blog posts could have had a more complete title: "Principals of Data Oriented Programming (aka, Idiomatic Clojure)"

All of these ideas (and I think they are good ones) are inspired heavily by Rich Hickey's talks and rational behind developing the Clojure language (the author of the post states as much). And while you can use these techniques in other languages/paradigms/problem domains, they are really intended to work well inside the constructs of Clojure, and when applied to "information-driven situated programs" [0] (read business applications with dynamic requirements).

As for some of the short-comings you mentioned:

"But you still need a mechanism to manage mutating data"

Clojure supports this through the use of locking constructs like atoms. [1]

"I think what it's getting at is that you don't really know the precise type of your data, over time, in a distributed system, so it's good to include the flexibility to handle that. That makes sense to me. but generic data structures aren't necessarily always the right way to handle that."

Clojure attempts to bridge the gap between generic data-structures and strongly-typed constructs using run-time specifications. [2]

I mean, the ideas presented here can be generally useful, but your mileage may vary if the principals take you too far out of the idiomatic for your particular language/paradigm/problem domain. If that's the case, you could find yourself wasting energy swimming up stream.

[0] - https://www.youtube.com/watch?v=2V1FtfBDsLU [1] - https://clojure.org/reference/atoms [2] - https://clojure.org/about/spec

Re: Principles of Data Oriented Programming

#93
post #85
post #72

I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space. The big one: "Data is immutable". The problem here is that data isn't actually immutable (generally) and mutability isn't actually the problem. The problem is unmanaged references or other dependencies on the mutable data. The "source of truth" becomes muddled whi…

Immutability is not the fact that something can not change in this case. It has more to do with the identity of a value. Every time you change anything inside the data structure you get a new reference, that is it!

Think of it as versioning for your data. So instead of referring to some data structure by its general concept with immutable data structures we are a lot more specific with respect to the identity represented by the name, here the name represents a version.

This is useful because, for example, you stop having the "unmanaged references" you were talking about because now since you are pointing to a version of the data and not the data itself you can be sure of what you are talking about.

Re: Principles of Data Oriented Programming

#94
post #56

Earlier quoted context omitted.

Functions are a mapping between a domain and a codomain, the mapping absolutely isn’t mutable, the definition of the function is the relationship between the domains. If I have a function: int Add1(int x) => x + 1 I would expect the domain and codomain to be immutable; I would also expect that x+1 to not turn in x/2 randomly also

> the mapping absolutely isn’t mutable Assume f: X -> Y. We can now map x_1 to y_1 f(x_1)=y_1. And then change this same function by mapping x_1 to y_2: f(x_1)=y_2. Thus we can easily modify functions. Moreover, we do it constantly when we modify object fields in OOP. It is probably easier to comprehend if a function is represented as a table which we modify. In contrast, we cannot modify data values (mathematical tu…

Can you expand upon this? Perhaps the difference between "re-mapping" the function:

    f(x_1)=y_2
and "re-mapping" the value:

    x=42+2
How is the former different than the latter? And by what mechanism is the former achieved? I understand what you are saying, but how does one simply "change this same function"? Redefine it?

To be clear, I'm not suggesting you are incorrect. I just don't fully understand what you are getting at.

Re: Principles of Data Oriented Programming

#95
post #77
post #27

Cheekily resubmitted, I see! Not that I mind. I think it's a great idea that deserves sharing. https://news.ycombinator.com/item?id=24682380#24685657

How did it open a second HN thread?

The URLs differ. This one includes "?essence".

Re: Principles of Data Oriented Programming

#96
post #72

I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space. The big one: "Data is immutable". The problem here is that data isn't actually immutable (generally) and mutability isn't actually the problem. The problem is unmanaged references or other dependencies on the mutable data. The "source of truth" becomes muddled whi…

I think the principles do basically summarize data oriented programming. Particularly, principle 1 is important since it is the exact opposite of OOP's insistence of encapsulation, and is possibly the biggest reason DOP works so much better than OOP.

Re: Principles of Data Oriented Programming

#97
post #72

I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space. The big one: "Data is immutable". The problem here is that data isn't actually immutable (generally) and mutability isn't actually the problem. The problem is unmanaged references or other dependencies on the mutable data. The "source of truth" becomes muddled whi…

> I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space.

These are are part of a book that is being written at the moment as far as I can tell:

"This article is an excerpt from my upcoming book about Data Oriented Programming. The book will be published by Manning, once it is completed (hopefully in 2021). "

Re: Principles of Data Oriented Programming

#98

Earlier quoted context omitted.

Be careful because there are perf issues if you are using parametric polymorphism. Monomorphic functions are preferable but the real problems occur once you get past the inline cache's maximum number of 'shapes'. This obviously applies to regular JS as well.

Perhaps I misunderstand, but wouldn't that only apply if you attach functions to objects? I suppose you wouldn't do that if you follow data oriented programming principles.

Nope. It's about the different types passed into a function parameter. For optimization JS engines look at the differing shapes of JS objects. Which is determined by the order and type of each member.

So:

  { a: 6, b: 7 } is different to { b: 7, a: 6 } and { a: "six", b: "seven" } is different to both.
This is done so the member can be looked up quickly by offset. Functions then have an inline cache that stores the shapes the function has seen. If the function is called monomorphically it will only ever see one shape and hit the fastest path. If it is called with up to three shapes (in V8) it will be pretty quick. Once past three the cache falls through to a global table and is dog slow.

This matters if you are using structural typing as you are still creating different shapes to be passed into the same function(s).

Re: Principles of Data Oriented Programming

#99

Earlier quoted context omitted.

Perhaps I misunderstand, but wouldn't that only apply if you attach functions to objects? I suppose you wouldn't do that if you follow data oriented programming principles.

Nope. It's about the different types passed into a function parameter. For optimization JS engines look at the differing shapes of JS objects. Which is determined by the order and type of each member. So: { a: 6, b: 7 } is different to { b: 7, a: 6 } and { a: "six", b: "seven" } is different to both. This is done so the member can be looked up quickly by offset. Functions then have an inline cache that stores the sha…

I see. Thanks for explaining. That's very good to know!

Re: Principles of Data Oriented Programming

#100
I've always liked the idea of "table oriented programming" where more detailed schema info is used to do most of the CRUD and UI work. In my experiments, the tricky part is exceptions to the patterns. You always need to be able to tweak things imperatively (via code). But the attributes can still do roughly 90% of the job.

My latest approach to get enough tweakability is what I tentatively call "fractal rendering events" or "staged rendering". When rendering HTML or SQL, you need event "hooks" for the different stages. Level 1 events may override/alter field attributes. Level 2 events may override/alter the HTML (or sql) generated for the field based on the Level 1 values. Level 3 events may override/alter the HTML of page sections (or entire SQL clauses). Level 4 is overriding/altering the entire page (or final sql statement).

In other words, the schema provides drafts, which can then be adjusted along the way through event hooks. The granularity of what's tweaked goes up with each stage.

But managing that many potential events needs something more powerful than a file-based system. It may be better to manage such source-code in an RDBMS so you can search, sort, and group by different factors at different times rather than hard-wire in one viewpoint as file systems do.

But current IDE's are not ready for this. I do believe it's the future, though. File trees are too limiting.

Consider this: it's common for a non-coding analyst to want to change a field label, page title, max field length, or "required" status. If they could do it in the schema info (data dictionary), then they don't have to involve the coders. Whether the data dictionary is referenced directly or generates scaffolded code is a stack-specific or shop-specific choice. Minor things like this shouldn't involve a lot of effort.

Post reply on HN