An Introduction to Data Oriented Design with Rust • Hacker News Board

Re: An Introduction to Data Oriented Design with Rust

#81

Post by swsieber » Thu, Sep 17, 2020, 8:59 PM UTC

bfrydl wrote: post #64

meheleventyone wrote: post #63

Earlier quoted context omitted.

Likewise the focus on ECS architecture as being the one true DoD pattern when it’s not necessarily data oriented at all and the unfounded assumption it’s used everywhere in game development when it isn’t . And somehow the idea that DoD is a replacement for OOP when it’s really an orthogonal concern. DoD is about recognising data use patterns in your software and organising your software architecture around them in a…

I think this is because of Rust. In my opinion the Rust game dev community is overly fixated on ECS. On the bright side, Rust has some damn good ECS libraries.

Rust lends itself to ECS a lot more than it lends itself to behavior hierarchies using inheritance. That's an over simplification, but a useful one.

Re: An Introduction to Data Oriented Design with Rust

#82

Post by JesseMeyer » Thu, Sep 17, 2020, 9:04 PM UTC

SquishyPanda23 wrote: post #70

jessermeyer wrote: post #14

Earlier quoted context omitted.

'The hardware is the platform' is the DOD mantra. It's not a weakness to integrate more information about the problem space into the solution. It's engineering. To _ignore_ information about the problem space is certainly a weakness, and is partly responsible why the Windows boot time appears to be a cosmological constant.

> It's not a weakness to integrate more information about the problem space into the solution. It's engineering. You want your solution to depend on the right abstract model of the problem, not on the particulars of the problem. Otherwise your solution is difficult to extend, and is generally expensive and error prone to change when your problem changes, which happens with 100% probability.

If a well established abstraction solves the problem, then that's just a particular known about the solution space.

If your data changes, your problem changes. We only ever solve particular problems, given the distribution, shape and density characteristics of input data.

Re: An Introduction to Data Oriented Design with Rust

#83

Post by jayd16 » Thu, Sep 17, 2020, 9:11 PM UTC

Arelius wrote: post #31

cogman10 wrote: post #10

Earlier quoted context omitted.

DOD generally implies a high level of coupling on the system. That's it's biggest weakness. It also pushes even more data management problems onto programmers which makes it easy to get things wrong. You take those downsides and you trade them for higher performance. Now, that doesn't mean that you can't have hybrid systems and get most of the benefits of both worlds. It does, however, mean that you will end up with…

> DOD generally implies a high level of coupling on the system. That's it's biggest weakness. I'm not sure I understand what you mean by DOD generally implies a high level of coupling? Could you perhaps elaborate a bit?

I would phrase it in the inverse. OOP implies some encapsulation/abstraction that you give up when you use sets of arrays instead.

Re: An Introduction to Data Oriented Design with Rust

#84

Post by Hammershaft » Thu, Sep 17, 2020, 9:13 PM UTC

oehtXRwMkIs wrote: post #22

ekidd wrote: post #18

Earlier quoted context omitted.

As always, it depends on the problem you're solving. A 2x speedup in the inner loops of a game can be a very big deal. And for business workloads, I occasionally spend time babysitting clusters running batch jobs. A 2x performance increase in the right inner loop might save a couple hundred dollars per run. So certainly, profile before you optimize, as the article demonstrates. But 2x speedups can be worth applying a…

I think the data-oriented version is very appropriate for a game, where performance is king and there's not that much maintenance afterwards once its done. Perhaps I'm wrong but games seem to be very much a finish and toss it over the wall sort of project. Much less iteration than other contexts.

That is changing over time as post launch support grows and some games shift to games as a service

Re: An Introduction to Data Oriented Design with Rust

#85

Post by nikki93 » Thu, Sep 17, 2020, 9:13 PM UTC

cogman10 wrote: post #78

Arelius wrote: post #31

Earlier quoted context omitted.

> DOD generally implies a high level of coupling on the system. That's it's biggest weakness. I'm not sure I understand what you mean by DOD generally implies a high level of coupling? Could you perhaps elaborate a bit?

From what I've seen in DOD systems, there are generally common structures that hold data for the entire application. The entire application has to know about these structures in order to function. Take ECS [1] as an example. In order to create a new entity on the system you might need to talk to a location component, a health component, a velocity component all to register a brand new entity on the system. Now, you m…

https://youtu.be/W3aieHjyNvw I think this talk on how an ECS actually reduced coupling in the case of Overwatch would be a good concrete real world example to look at. It enabled some architectural benefits for implementing replay cameras or for networking code in their game through actual reduced coupling between input ->commands and commands->simulation.

Re: An Introduction to Data Oriented Design with Rust

#86

Post by Jtsummers » Thu, Sep 17, 2020, 9:28 PM UTC

narwally wrote: post #77

I find it interesting, and somewhat amusing, that we're seeing Data-Oriented-Design becoming popular in C++ and Rust while Data-Oriented-Programming/Data-Driven-Design[1] is being promoted in languages like Clojure and Elm. Both have similar names, and seem to have arisen out of frustrations with OOP. However, their solutions went in completely opposite, though not unrelated, directions. Seems like DOD is about impro…

"Data-driven programming" tends to give more results than "data-driven design" which mostly (on my Google search) turned up content about usage data driving interface design.

https://en.wikipedia.org/wiki/Data-driven_programming

Re: An Introduction to Data Oriented Design with Rust

#87

Post by runevault » Thu, Sep 17, 2020, 9:32 PM UTC

meheleventyone wrote: post #68

bfrydl wrote: post #64

Earlier quoted context omitted.

I think this is because of Rust. In my opinion the Rust game dev community is overly fixated on ECS. On the bright side, Rust has some damn good ECS libraries.

I’d agree to a point but it really crosses the whole gamut of hobby game engine development. It’s weirdly self reinforcing even in the face of more interesting architectural choices like DOOM Eternal’s.

The way Eternal's engine is described it could simply be an ECS with a more advanced job system for farming out work because it can parse how the various objects are updated and only do reads after all writes to the objects happen.

Re: An Introduction to Data Oriented Design with Rust

#88

Post by munificent » Thu, Sep 17, 2020, 9:33 PM UTC

bfrydl wrote: post #64

meheleventyone wrote: post #63

Earlier quoted context omitted.

Likewise the focus on ECS architecture as being the one true DoD pattern when it’s not necessarily data oriented at all and the unfounded assumption it’s used everywhere in game development when it isn’t . And somehow the idea that DoD is a replacement for OOP when it’s really an orthogonal concern. DoD is about recognising data use patterns in your software and organising your software architecture around them in a…

I think this is because of Rust. In my opinion the Rust game dev community is overly fixated on ECS. On the bright side, Rust has some damn good ECS libraries.

> the Rust game dev community is overly fixated on ECS.

Not just Rust. The game dev community everywhere is infatuated with ECS.

It's basically cargo culting. There is a large base of amateur or indie game developers who want to feel like they are doing game development the "right" way. One big aspect of that is performance. ECS has a reputation for efficiency (which is true, when used well in a context where your performance problems are related to caching), so you see a lot of game devs slavishly applying it to their code in hopes that the "go as fast as a AAA game" Gods will land on their runway and deliver the goods.

Every time I see a new ECS framework in JavaScript, I die a little on the inside.

Re: An Introduction to Data Oriented Design with Rust

#89

Post by yumaikas » Thu, Sep 17, 2020, 9:42 PM UTC

renewiltord wrote: post #73

algorithmsRcool wrote: post #48

Earlier quoted context omitted.

Part of the Trill[0] processing engine in C# does this. It uses reflection to vectorize normal objects into a columnar format internally. [0] : https://github.com/microsoft/Trill

Fascinating. Isn't reflection in these VM languages pretty slow usually? Won't you pay that cost on access each time? Still, C# seems to be capable of a lot if you can do this ergonomically. Very cool.

That really depends which reflection APIs are being used.

For C#, you can use type reflection to get the data or shape of a piece of code or a type (GetType and friends). That isn't super fast, and isn't intended to be.

The other major C# reflection API is Reflection.Emit, which allows you emit IL at runtime. The Emit->IL->JIT process is slower than executing the code outright the first time, but once you've done it, you can get code that's as fast as as if it were compiled from source. And, in the case, you might end up with faster code, since it allows you to do transforms on how the data is laid out and compiled.

Notably, the .NET regex engine exposes options for compiling the regex down to IL if it's going to be used a lot, so a library like Trill using reflection to accelerate code by vectorizing it, while keeping better ergonomics makes a lot of sense. You still pay the cost of compiling at runtime, since it's not a compile-time transform. Those do exist for .NET, in the form of Rosalyn plugins, but that's a bit more involved still.

Re: An Introduction to Data Oriented Design with Rust

#90

Post by throwaway894345 » Thu, Sep 17, 2020, 9:47 PM UTC

meheleventyone wrote: post #63

iherbig wrote: post #21

It seems like a lot of the discussion surrounding DOD that gets popular interest is centered on a small set of patterns that you can apply. And the implication that DOD is the application of these patterns usually follows. Taking this article as an example, it frames DOD as an optimization technique and explicitly states that these patterns are the main concepts of DOD. But while these patterns are interesting and of…

Likewise the focus on ECS architecture as being the one true DoD pattern when it’s not necessarily data oriented at all and the unfounded assumption it’s used everywhere in game development when it isn’t . And somehow the idea that DoD is a replacement for OOP when it’s really an orthogonal concern. DoD is about recognising data use patterns in your software and organising your software architecture around them in a…

> And somehow the idea that DoD is a replacement for OOP when it’s really an orthogonal concern.

I’m sure this is true for some definition of OOP, but I’ve seen too much OOP code that insists on hanging every method off of the most plain-old-data of classes. A Book class has to have a buyFromAmazon() method and consequently each instance has a private reference to an Amazon SDK client, and if you want to buy 10K books, you iterate over a list and invoke this method 10K times.

Of course, some will argue that this is bad OOP and true OOP doesn’t look like this, and anyway you can write bad code in any paradigm! Of course, OOP is uniquely plagued with bad code (much more so than other paradigms) and this pretty transparent no-True-Scotsman argument is just moving semantic goalposts (as such arguments do).