Live data from Hacker News

Principles of Data Oriented Programming

blog.klipse.tech

121–130 of 139 posts

Re: Principles of Data Oriented Programming

#121
post #72

I don't think these principles add up to something useful. It's not complete and I think some of the principles don't align that well to the problem space. The big one: "Data is immutable". The problem here is that data isn't actually immutable (generally) and mutability isn't actually the problem. The problem is unmanaged references or other dependencies on the mutable data. The "source of truth" becomes muddled whi…

I was a self-taught programmer and when I first learned to program, I found the concept of 'reference' is very counter-intuitive to me. I assume people have no CS background may find the same - why is this 5 and another 5 is the same, but not people with the same name and social security numbers?

Traditionally people use mutable struct or objects for non-primitive data structures, which implicitly allocate addresses in computer memories - that is a very implementation-wised approach, more specifically, a very Von Neumann approach.

For example, there are many systems having entities in the databases, like Person { id, name }. But if there's no database, in many programming languages people would write Person { name } only. If people write the in-memory version first, they have to re-write their models. Systems with mutable references inconsistently force the referencing system on people by default, instead of just giving people only they ask for.

In modern days, a lot of computing goes beyond a single computer, there are many distributed systems, a lot of serialization is going on, and the reference in one computer's memory doesn't mean anything to another.

If we try to loosen the definition of the computer, we can think of many systems as bizarre computers. If we take a look at data processing systems like Spark or Flink, conceptually they also look like computers - bunches of hardware and 'operating systems' sit on top of them, but in the form of clusters instead of single computers. In such cases, there are no shared memories, and the memories are implementation details - users won't aware of them like they have to realize there are underlying memory systems in Von Neumann computers. In such kind of systems, people only care about computing which is more substitution based instead of mutable references. While at most, they only need to aware of nodes as a whole in the cluster.

It's close to the 'substitution model' vs the 'environment model', or the 'functional' vs the 'object-oriented' described in the SICP. Where I found conceptually the 'substitution model' is more fundamental, and simpler - it's basically elementary or middle school maths.

> The problem here is that data isn't actually immutable (generally) and mutability isn't actually the problem. The problem is unmanaged references or other dependencies on the mutable data.

So another novel way to look at it, is if you don't have mutable references, you don't have to manage it, then the problem is eliminated as a whole. Like Haskell, Erlang, or Spark as we just mentioned. At the end of the day, you can have references in outer worlds, be it ST monads, processes, database, or just our real world, sometimes the processing part of the program don't have to care about them.

Re: Principles of Data Oriented Programming

#122
post #50

Earlier quoted context omitted.

I know that there some common topics with Eric's books. Do you think Eric focuses as much as I do about data?

No, he's much more focused on what he calls actions and calculations. Perhaps overly so. I'd have liked a bit more focus on data. That could be a by-product of his Clojurist roots. Data-oriented programming is so integral to Clojure's culture that I'm not sure Clojurists even realize they're doing it half the time.

> Data-oriented programming is so integral to Clojure's culture that I'm not sure Clojurists even realize they're doing it half the time.

I totally agree with you. I am a Clojurist and my hope is that my DOP book will spread the "light" to non Clojurist.

Re: Principles of Data Oriented Programming

#123

We certainly use some of these principles in OrgPad.com and some of those inspired even the User Experience in a fundamental way. E.g. a bullet point list in a linear medium such as a text or a slide in a presentation is like a star in a graph, where all children have the same weight. The thing is, when people see it like that graphically, they sometimes get ideas they wouldn't have, if they stared at a long text. So…

My original book with Manning about Clojure has been abandoned. Now I am writing a book about Data oriented programming

Re: Principles of Data Oriented Programming

#125

Earlier quoted context omitted.

What do you mean by "functions are supposed to be mutable"? Perhaps you are just pointing out that the output of the function (and therefore the value of the field...?) will change as the input changes? If mathematical tuples are immutable, then surely mathematical functions are immutable as well ;)

Function is a mapping between two sets (of values). This mapping between values is mutable although the values are not.

Functions might be isomorphic to one-another, that doesn't make the function itself mutable.

Re: Principles of Data Oriented Programming

#127
post #89

Earlier quoted context omitted.

You realize data is immutable when you first try to implement history. Mutability is just a hack to save some memory.

> just a hack to save some memory .. and time. If we needed to compute an account balance by summing all the debits and credits since the account was opened...

Persistent data structures already solved this issue.

Re: Principles of Data Oriented Programming

#128

Earlier quoted context omitted.

Evidence that "DOP works so much better than OOP" is scant imo. The rise and fall of paradigms that present themselves as panaceas is instructive. You have "structured programming", "object oriented programming", "functional programming" and now "data oriented programming". What I'd like to see is paradigms paired with "where this works well" rather than paradigms sold based on "this will solve the software crisis",…

That's because data oriented programming has already succeed on a large scale, you just might not recognize it. SQL is about as data-oriented as it gets, you have a programming model that's constrained and focused on data layout, structure and performance over being a general purpose language. There's a good article a while back about ECS that I can't find which talks about how most performant ECS start to mirror SQL…

It has succeeded but it has also failed. Using a shared database for many different functions/applications can be a disaster that eventually seizes up. Everyone is afraid to change the data structures because they don't know what it will break. It's a huge piece of global state with no isolation. Enter encapsulation and service orientation and bounded contexts.

So the parent is correct. It's not enough to say data orientation is a good thing. It needs to be compared with previous approaches (encapsulation) and then explain what the tradeoffs are.

Re: Principles of Data Oriented Programming

#129
post #112
post #32

Earlier quoted context omitted.

Usually a map can be fine, but isnt it a maintenance nightmare. At least if you change an object, the compiler will complain if an attribute is not found, but this leads to runtime error/ or bad behaviour

Yes, for sure it is. But what is the alternative? A compiler can only complain on a monolith, it can't scratch boundaries out of the system it compiles (and still within that monolith, encapsulated system the data might be inconsistent). For distributed systems this has been solved in the last two decades (if not even longer), but still systems are pressing against formality, how come? Continue to write the code to d…

How has this been solved for distributed systems?

Re: Principles of Data Oriented Programming

#130
The articles sounds like an advertisment to use Elm. Immutability and value semantic is enforcd by the language and the typesystem is rich enough to allow to use generic data sructures while been statically typed. As a functional language Elm has closures allowing to violate separation of data and code and get non-literal data types, but this is strongly discouraged by documentation, runtime (aka Elm architecture) and tools.
Post reply on HN