Live data from Hacker News

Databases are the endgame for data-oriented design

spacetimedb.com

51–60 of 157 posts

Re: Databases are the endgame for data-oriented design

#51

Thank you, I enjoyed this post. I am thinking that iteration is just traversal and traversal is just execution. Take iterators in C++ or standard library "algorithms" library in C++. Or iterators in Rust or in Java. You just want to traverse and collect values (and calls on functions in some interleaving), which is like joining tables as this article says. I'm thinking that the definition of the traversal (such as Ka…

[deleted]

Re: Databases are the endgame for data-oriented design

#52

Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…

> It also means you have a single point of failure, no read-replicas or redundancy. Hate everything about this.

How does writing an application as a single language/binary prevent read-replicas or redundancy?

The application could do that.

Or you could do it at disk level (RAID).

Re: Databases are the endgame for data-oriented design

#53
post #40

Earlier quoted context omitted.

The types of issues you're enumerating are really part of a learning curve that every language and environment will have. > not having basic functions for calculating percentiles or even a basic median I don't think this is really true anymore; windowing functions are pretty prolific, and I think every major database will have some percentile functions

> The types of issues you're enumerating are really part of a learning curve that every language and environment will have. They're not though. That's why excellent coders often despise having to learn and use SQL, and basically just refuse to. SQL is uniquely terrible -- and I say this as someone with a career spanning from Win32 C++ to every major web technology. > every major database will have some percentile fun…

How many application languages have percentile functions?

Re: Databases are the endgame for data-oriented design

#54

Earlier quoted context omitted.

Are you using Postgres? It seems like you haven’t discovered CTEs or lateral joins.

Oh, I've worked with all the major engines. To which we can add: trying to remember which seemingly basic features are supported by which engines, and which aren't (e.g. percentiles are great in Postgres, but you're SOL in MySQL). That there's virtually nothing "standard" about SQL syntax or functionality at all, except for the simplest of queries and the general basic concepts.

Well Java and JavaScript differ a lot too.

Some similarities, but a lot of differences too.

Re: Databases are the endgame for data-oriented design

#55
post #4

My colleagues hate me, but I also found that SQL is The way to write business logic. Lots of caveats about difficulty to test and weird syntax. But it is just that SQL is the most terse and standard way so express logic. And that in itself is the most important factor to avoid bugs. Not what testing strategy you choose.

I’ve heard that datalog (and similar languages in that vein) is far easier and terser to write. What do you think about that?

Re: Databases are the endgame for data-oriented design

#56

As an ex-game developer and software architecture nerd, I'm very excited about data-oriented design and ECS. It really is a cool pattern, and it's a very common one in shipping games today. It's not just architecture astronaut stuff. At the same time, the level of hype about ECS today reminds me an awful lot of the amount of hype surrounding OOP in the 90s. Can ECS be a better way to structure your game entities and…

> (There are ECS frameworks in JavaScript, which gives you absolutely no control over memory layout and thus completely defeats one of the primary purposes of the pattern.)

While JS does not provide great support for bit packing complex structs, typed arrays give you quite a bit of control over memory layout for simple numeric types, which is what you usually want for optimal data-oriented code anyway. This is a common technique used in fast JS libs for data visualization, ie:

https://github.com/mourner/flatbush

There are also basic operators required for bitarrays, which are useful for ECS and memory-efficient code generally.

Re: Databases are the endgame for data-oriented design

#57
post #47

Earlier quoted context omitted.

Most other programming languages are more composable than a SQL query. You can make functions, split logic into different classes, extract common logic, etc. Much harder with big sql queries

Use views and stored procedures if you need composability.

Counterpoint: don’t do this. The path to hell is paved with good intentions.

Some co-workers and I inherited a code-base where the authors went down the views and stored procedures route. It was basicallly impossible to untangle; there was no knowing what relied on a view or a proc, so you couldn’t touch them at all, there were no docs, there were duplicates of everything (and again, no way to know what’s used). A good number of them had all kinds of side effects and weird performance characteristics. They weren’t version controlled.

The issues just kept going.

Re: Databases are the endgame for data-oriented design

#58

Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…

> Databases should never, ever, ever, be used to perform logic, they are datastores, that is it. I wouldn't go that far. Relational algebra is performing logic. Constraints and foreign keys are logic, as well. I'm not going to argue that you should go back in time 15-20 years and start shredding XML strings in stored procedures again. But thinking of the database as one step above a flat file is similarly backward th…

> At a concept level, that's exactly the same thing.

But at a practical level they very different. If you have a middleware layer as you are describing, it's written in a real programming language with all the adjacent tools (source control, debugging, etc).

I'm not hard-core against stored procedures used lightly but they have a lot of downsides and they simply aren't needed. There's no performance advantage. There are complexity advantages and disadvantages that might be a wash.

Re: Databases are the endgame for data-oriented design

#59
post #18

Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…

> Databases should never, ever, ever, be used to perform logic You're talking about a best practice like it's a fundamental law. It's not, it's just how we've mostly been doing things. A lot of interesting innovations in distributed systems / architecture (serverless, graphql, thin clients, thick clients, ORMs, RSC, WSGI, nodejs) have been made because the designers tried relaxing a constraint or taking a counterintu…

[dead]

Re: Databases are the endgame for data-oriented design

#60
post #14

Earlier quoted context omitted.

Glad that doesn't happen any where else!

Most other programming languages are more composable than a SQL query. You can make functions, split logic into different classes, extract common logic, etc. Much harder with big sql queries

You can define functions in all major databases
Post reply on HN