Live data from Hacker News

Databases are the endgame for data-oriented design

spacetimedb.com

81–90 of 157 posts

Re: Databases are the endgame for data-oriented design

#81

Author feedback: the large print, and the static elements (screen-thieving banner + “AI assistant” button), made attempting to read this blog as presented a vile experience on my phone. The AI button can’t be dismissed, if you ask it how to dismiss itself it doesn't know, and it won’t even accept negative feedback due to obscured submit buttons. Fortunately there’s Reader mode.

Thanks for the feedback, we've definitely got to fix the mobile scaling stuff.

Re: Databases are the endgame for data-oriented design

#82
post #60

Earlier quoted context omitted.

You can define functions in all major databases

And how many non-technical users do you know who do this?

And how many non-technical users do you know that write good javascript/java/.net/etc code?

Re: Databases are the endgame for data-oriented design

#83
post #20

Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…

> Why? Databases should never, ever, ever, be used to perform logic, they are datastores, that is it. Your logic goes elsewhere. Stored procedures are the worst "feature" of any database, you are just asking for a hard time debugging, troubleshooting, and increasing the chance that you will fuck up the most valuable part of your system, your data. You cannot sanely use a database with multiple heterogenous clients wi…

This is of course what SpacetimeDB does. The stored procedures fail any transactions that are not authorized to be carried out. e.g. if a player is not high enough level or something.

Re: Databases are the endgame for data-oriented design

#84
post #4

My colleagues hate me, but I also found that SQL is The way to write business logic. Lots of caveats about difficulty to test and weird syntax. But it is just that SQL is the most terse and standard way so express logic. And that in itself is the most important factor to avoid bugs. Not what testing strategy you choose.

We've been doing this for a while and it has some pretty severe limitations at scale. The biggest issue is that humans are human and you cannot make everyone else on the team use your elegant SQL DSL business engine in exactly the same way you had envisioned. Eventually, it will grow into a bit of a monster.

We are headed back in the other direction, but with some hedge. Code with a very simple & transparent CI/CD experience seems equivalent to some SQL configurable thing in our minds now, but with way more potential upside.

In our latest code-based abstractions, nothing is stopping you from breaking out a SQL connection and running any arbitrary query. In fact, all of the data is still in SQL and it's still authoritative, you just now have access to way more powerful tooling to work with that data. Code+SQL together is the answer.

For non-expert team members, modern C# is turning out to be way more intuitive than SQL the moment you encounter a 3+ table join scenario.

Re: Databases are the endgame for data-oriented design

#85

As an ex-game developer and software architecture nerd, I'm very excited about data-oriented design and ECS. It really is a cool pattern, and it's a very common one in shipping games today. It's not just architecture astronaut stuff. At the same time, the level of hype about ECS today reminds me an awful lot of the amount of hype surrounding OOP in the 90s. Can ECS be a better way to structure your game entities and…

> There are ECS frameworks in JavaScript, which gives you absolutely no control over memory layout and thus completely defeats one of the primary purposes of the pattern. It doesn't give you direct control over the memory layout, but it's still fairly safe to assume that arrays are going to end up in relatively contiguous memory, which the relevant part for the performance difference between structs of arrays and arr…

To get this in JS you should do it explicitly with TypedArrays otherwise you’re at risk of what the underlying VM does and in current modern implementations it most assuredly doesn’t make sure most kinds of arrays end up relatively contiguous. Notably for this conversation arrays of objects are unlikely to.

Re: Databases are the endgame for data-oriented design

#86

As an ex-game developer and software architecture nerd, I'm very excited about data-oriented design and ECS. It really is a cool pattern, and it's a very common one in shipping games today. It's not just architecture astronaut stuff. At the same time, the level of hype about ECS today reminds me an awful lot of the amount of hype surrounding OOP in the 90s. Can ECS be a better way to structure your game entities and…

I don't think memory layout is the sole draw, it's also about incrementally building up programs live.

Re: Databases are the endgame for data-oriented design

#87

Earlier quoted context omitted.

We're in the process of migrating our on-premise business application from SQLAnywhere to MSSQL. Imagine the fun we're having...

I'm really curious, in the practice, what causes a product to decide "we are moving from abcSQL TO xyzSQL" ?

For us there's been two major motivations, and another few lesser.

First and most prominent is that Sybase was bought by SAP and SAP has pivoted SQLAnywhere from a general RMDBS to focus on mobile applications (ie more embedded). Apart from product development changing focus we've struggled to buy licenses for our customers, so the writing on the wall was clear.

The seconds is that we've got some big customers who run their own MSSQL servers, and they've been pushing harder and harder for us to use their servers so they can handle administration and access control directly.

A third motivation is that while we've been primarily on-prem, a lot of customers these days don't want that, so we've started offering cloud hosted options. Since we're a Windows shop Azure made sense, and as such the hosted MSSQL also makes sense and is quite convenient.

Fourth is that, while MSSQL is quite anemic in terms of SQL language functionality, it does support transactional schema updates, which makes schema upgrades much less error prone.

Given that MSSQL syntax support is quite limited and we now have developed tooling and library support for multiple databases, we'll probably also add support for PostgreSQL later.

Re: Databases are the endgame for data-oriented design

#88

Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…

Nothing you mentioned is an inherent problem with the databases. In fact, databases have arguably more potential to solve these problems then any currently used system.

Like debugging - if every memory access is a database access, then you have a builtin logging for all your memory access that is both more performant and optimal then any normal normal debugger. You can take snapshots of your 'memory' pretty much at any time, the data layout is clearly defined. You can manually edit your memory any time you want, and serialization is already solved for you.

The more I think about it, the more possibilities I come up with. Damn, this is actually genius!

Re: Databases are the endgame for data-oriented design

#89
post #46

Earlier quoted context omitted.

> They're not though. That's why excellent coders often despise having to learn and use SQL, and basically just refuse to. SQL is uniquely terrible -- and I say this as someone with a career spanning from Win32 C++ to every major web technology. This is basically a strawman + appeal to authority argument. I've no desire to bicker. "Uniquely terrible" is quite subjective, we'll have to agree to disagree on if it appli…

Excuse me, but I'm certainly not "bickering" and I resent the insinuation. And you don't seem to understand what a strawman argument is. I'm not going to write an article on this for you, but you can certainly see the sentiment frequently expressed on HN that the rise of NoSQL and of ORM's is a direct result of otherwise capable programmers not wanting to deal with SQL and its conceptual building blocks such as joins…

I take it as the same reason they don’t want to use boring technology like HAProxy, or why they think infrastructure in general is beneath them: because they are easily influenced by breathless grifters on Medium promoting the New Thing.

SQL isn’t going away any time soon. At its heart, it’s relational algebra. If you can learn DS&A, you can learn relational algebra.

Re: Databases are the endgame for data-oriented design

#90
post #26

Earlier quoted context omitted.

"At the same time, the level of hype about ECS today reminds me an awful lot of the amount of hype surrounding OOP in the 90s." I appreciate that most of the ECS hype has been around specific use cases, though. OOP was claimed as not a specific useful tool, but the answer to all programming, a billing it has not lived up to. It achieved "useful tool", no question, especially as some of the very rough bits were sanded…

OOP even got mixed with databases to form nasty hype around ORMs and NoSQL.

[deleted]
Post reply on HN