Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databases are the endgame for data-oriented design

spacetimedb.com

Post reply on HN
Print view View original

61–70 of 157 posts

Re: Databases are the endgame for data-oriented design

#61

Post by magicalhippo »

crazygringo wrote: post #21
deadbabe wrote: post #19

Earlier quoted context omitted.

Are you using Postgres? It seems like you haven’t discovered CTEs or lateral joins.

Oh, I've worked with all the major engines. To which we can add: trying to remember which seemingly basic features are supported by which engines, and which aren't (e.g. percentiles are great in Postgres, but you're SOL in MySQL). That there's virtually nothing "standard" about SQL syntax or functionality at all, except for the simplest of queries and the general basic concepts.

We're in the process of migrating our on-premise business application from SQLAnywhere to MSSQL.

Imagine the fun we're having...

Re: Databases are the endgame for data-oriented design

#62

Post by p1necone »

munificent wrote: post #23

As an ex-game developer and software architecture nerd, I'm very excited about data-oriented design and ECS. It really is a cool pattern, and it's a very common one in shipping games today. It's not just architecture astronaut stuff. At the same time, the level of hype about ECS today reminds me an awful lot of the amount of hype surrounding OOP in the 90s. Can ECS be a better way to structure your game entities and…

> There are ECS frameworks in JavaScript, which gives you absolutely no control over memory layout and thus completely defeats one of the primary purposes of the pattern.

It doesn't give you direct control over the memory layout, but it's still fairly safe to assume that arrays are going to end up in relatively contiguous memory, which the relevant part for the performance difference between structs of arrays and arrays of structs.

I don't recall explicitly telling many compiled languages to stick all the items in an array together in memory either - it just happens by default, same as in JS.

Re: Databases are the endgame for data-oriented design

#63

Post by whstl »

FridgeSeal wrote: post #57
ako wrote: post #47

Earlier quoted context omitted.

Use views and stored procedures if you need composability.

Counterpoint: don’t do this. The path to hell is paved with good intentions. Some co-workers and I inherited a code-base where the authors went down the views and stored procedures route. It was basicallly impossible to untangle; there was no knowing what relied on a view or a proc, so you couldn’t touch them at all, there were no docs, there were duplicates of everything (and again, no way to know what’s used). A go…

If your colleagues are not applying basic engineering rules to views and stored procedures, it's not fair to say that this is a problem with views and sprocs.

They can and should most definitely be testable, documentable, trackable and version controlled.

Should one also judge all backend programming by PHP standards circa 1998?

Re: Databases are the endgame for data-oriented design

#64

Post by eddd-ddde »

magicalhippo wrote: post #61
crazygringo wrote: post #21

Earlier quoted context omitted.

Oh, I've worked with all the major engines. To which we can add: trying to remember which seemingly basic features are supported by which engines, and which aren't (e.g. percentiles are great in Postgres, but you're SOL in MySQL). That there's virtually nothing "standard" about SQL syntax or functionality at all, except for the simplest of queries and the general basic concepts.

We're in the process of migrating our on-premise business application from SQLAnywhere to MSSQL. Imagine the fun we're having...

I'm really curious, in the practice, what causes a product to decide "we are moving from abcSQL TO xyzSQL" ?

Re: Databases are the endgame for data-oriented design

#65

Post by coldtea »

hitpointdrew wrote: post #12

Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…

Stored procedures ensure all your clients get the same logic. They're only "the worst feature of any database" if the language you're writing them in is not suitable (which is the case with PL/SQL and co, which were tacked on to RDBs and reek of bad 80s syntax and facilities).

If the language is nice and has well designed access to db facilities like records and such, it can be better than writing your code outside the database, especially coupled with a data-oriented design model/ECS (which can be extremely debuggable and offer great visibility).

>and increasing the chance that you will fuck up the most valuable part of your system, your data.

Since you can write anything outside you can inside, no. I can send a "delete from/drop " from any client at any moment, or make any mistake in updating. That's what backups are for, and databases make them even easier (not to mention transactions being very good and neglected part of business logic).

Re: Databases are the endgame for data-oriented design

#66

Post by DrDroop »

munificent wrote: post #23

As an ex-game developer and software architecture nerd, I'm very excited about data-oriented design and ECS. It really is a cool pattern, and it's a very common one in shipping games today. It's not just architecture astronaut stuff. At the same time, the level of hype about ECS today reminds me an awful lot of the amount of hype surrounding OOP in the 90s. Can ECS be a better way to structure your game entities and…

What about TypedArrays though, they could be useful to implement a ECS in JS. I do agree with your point, but ECS could be a useful pattern in JS not just for graphics or games but also OLAP workloads.

Re: Databases are the endgame for data-oriented design

#67

Post by manicennui »

ako wrote: post #48
buryat wrote: post #10

Earlier quoted context omitted.

that's how you end up with multi-thousand LOC sql files that contain all the historical business logic and edge cases of a company and it just keeps being piled up on and its complexity keeps growing because not everything is expressible in SQL. So you end up having a big monster that someone needs to support and one that's not very testable easily

And in any other language this wouldn’t result in multi-thousand lines of code with historical business logic and edge cases?

The difference is that most people don't write their business logic in a single flat function in most programming languages, and there is a good chance that individual pieces would have tests.

Re: Databases are the endgame for data-oriented design

#68

Post by manicennui »

camgunz wrote: post #60
nightpool wrote: post #42

Earlier quoted context omitted.

Most other programming languages are more composable than a SQL query. You can make functions, split logic into different classes, extract common logic, etc. Much harder with big sql queries

You can define functions in all major databases

And how many non-technical users do you know who do this?

Re: Databases are the endgame for data-oriented design

#69

Post by coldtea »

wvenable wrote: post #58
da_chicken wrote: post #24

Earlier quoted context omitted.

> Databases should never, ever, ever, be used to perform logic, they are datastores, that is it. I wouldn't go that far. Relational algebra is performing logic. Constraints and foreign keys are logic, as well. I'm not going to argue that you should go back in time 15-20 years and start shredding XML strings in stored procedures again. But thinking of the database as one step above a flat file is similarly backward th…

> At a concept level, that's exactly the same thing. But at a practical level they very different. If you have a middleware layer as you are describing, it's written in a real programming language with all the adjacent tools (source control, debugging, etc). I'm not hard-core against stored procedures used lightly but they have a lot of downsides and they simply aren't needed. There's no performance advantage. There…

>If you have a middleware layer as you are describing, it's written in a real programming language with all the adjacent tools (source control, debugging, etc).

And nothing stops a DB internal language used to write store procedures and such be a "real language" -- with source control, debugging and everything. RDBs can do a whole lot more offering an even better environment for those than common RDBs do - Smalltalk or Symbolics level good.

In fact in the case of this project, that language is Rust.

Re: Databases are the endgame for data-oriented design

#70

Post by robertlagrant »

ako wrote: post #48
buryat wrote: post #10

Earlier quoted context omitted.

that's how you end up with multi-thousand LOC sql files that contain all the historical business logic and edge cases of a company and it just keeps being piled up on and its complexity keeps growing because not everything is expressible in SQL. So you end up having a big monster that someone needs to support and one that's not very testable easily

And in any other language this wouldn’t result in multi-thousand lines of code with historical business logic and edge cases?

Testing code is pretty popular in most non-SQL environments.
Post reply on HN