Thank you, I enjoyed this post. I am thinking that iteration is just traversal and traversal is just execution. Take iterators in C++ or standard library "algorithms" library in C++. Or iterators in Rust or in Java. You just want to traverse and collect values (and calls on functions in some interleaving), which is like joining tables as this article says. I'm thinking that the definition of the traversal (such as Ka…
Databases are the endgame for data-oriented design
51–60 of 157 posts
Re: Databases are the endgame for data-oriented design
#52Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…
How does writing an application as a single language/binary prevent read-replicas or redundancy?
The application could do that.
Or you could do it at disk level (RAID).
Re: Databases are the endgame for data-oriented design
#53Earlier quoted context omitted.
The types of issues you're enumerating are really part of a learning curve that every language and environment will have. > not having basic functions for calculating percentiles or even a basic median I don't think this is really true anymore; windowing functions are pretty prolific, and I think every major database will have some percentile functions
> The types of issues you're enumerating are really part of a learning curve that every language and environment will have. They're not though. That's why excellent coders often despise having to learn and use SQL, and basically just refuse to. SQL is uniquely terrible -- and I say this as someone with a career spanning from Win32 C++ to every major web technology. > every major database will have some percentile fun…
Re: Databases are the endgame for data-oriented design
#54Earlier quoted context omitted.
Are you using Postgres? It seems like you haven’t discovered CTEs or lateral joins.
Oh, I've worked with all the major engines. To which we can add: trying to remember which seemingly basic features are supported by which engines, and which aren't (e.g. percentiles are great in Postgres, but you're SOL in MySQL). That there's virtually nothing "standard" about SQL syntax or functionality at all, except for the simplest of queries and the general basic concepts.
Some similarities, but a lot of differences too.
Re: Databases are the endgame for data-oriented design
#55My colleagues hate me, but I also found that SQL is The way to write business logic. Lots of caveats about difficulty to test and weird syntax. But it is just that SQL is the most terse and standard way so express logic. And that in itself is the most important factor to avoid bugs. Not what testing strategy you choose.
Re: Databases are the endgame for data-oriented design
#56As an ex-game developer and software architecture nerd, I'm very excited about data-oriented design and ECS. It really is a cool pattern, and it's a very common one in shipping games today. It's not just architecture astronaut stuff. At the same time, the level of hype about ECS today reminds me an awful lot of the amount of hype surrounding OOP in the 90s. Can ECS be a better way to structure your game entities and…
While JS does not provide great support for bit packing complex structs, typed arrays give you quite a bit of control over memory layout for simple numeric types, which is what you usually want for optimal data-oriented code anyway. This is a common technique used in fast JS libs for data visualization, ie:
https://github.com/mourner/flatbush
There are also basic operators required for bitarrays, which are useful for ECS and memory-efficient code generally.
Re: Databases are the endgame for data-oriented design
#57Earlier quoted context omitted.
Most other programming languages are more composable than a SQL query. You can make functions, split logic into different classes, extract common logic, etc. Much harder with big sql queries
Use views and stored procedures if you need composability.
Some co-workers and I inherited a code-base where the authors went down the views and stored procedures route. It was basicallly impossible to untangle; there was no knowing what relied on a view or a proc, so you couldn’t touch them at all, there were no docs, there were duplicates of everything (and again, no way to know what’s used). A good number of them had all kinds of side effects and weird performance characteristics. They weren’t version controlled.
The issues just kept going.
Re: Databases are the endgame for data-oriented design
#58Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…
> Databases should never, ever, ever, be used to perform logic, they are datastores, that is it. I wouldn't go that far. Relational algebra is performing logic. Constraints and foreign keys are logic, as well. I'm not going to argue that you should go back in time 15-20 years and start shredding XML strings in stored procedures again. But thinking of the database as one step above a flat file is similarly backward th…
But at a practical level they very different. If you have a middleware layer as you are describing, it's written in a real programming language with all the adjacent tools (source control, debugging, etc).
I'm not hard-core against stored procedures used lightly but they have a lot of downsides and they simply aren't needed. There's no performance advantage. There are complexity advantages and disadvantages that might be a wash.
Re: Databases are the endgame for data-oriented design
#59Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…
> Databases should never, ever, ever, be used to perform logic You're talking about a best practice like it's a fundamental law. It's not, it's just how we've mostly been doing things. A lot of interesting innovations in distributed systems / architecture (serverless, graphql, thin clients, thick clients, ORMs, RSC, WSGI, nodejs) have been made because the designers tried relaxing a constraint or taking a counterintu…
Re: Databases are the endgame for data-oriented design
#60Earlier quoted context omitted.
Glad that doesn't happen any where else!
Most other programming languages are more composable than a SQL query. You can make functions, split logic into different classes, extract common logic, etc. Much harder with big sql queries