Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databases are the endgame for data-oriented design

spacetimedb.com

Post reply on HN
Print view View original

111–120 of 157 posts

Re: Databases are the endgame for data-oriented design

#111

Post by gefjon »

wvenable wrote: post #58
da_chicken wrote: post #24

Earlier quoted context omitted.

> Databases should never, ever, ever, be used to perform logic, they are datastores, that is it. I wouldn't go that far. Relational algebra is performing logic. Constraints and foreign keys are logic, as well. I'm not going to argue that you should go back in time 15-20 years and start shredding XML strings in stored procedures again. But thinking of the database as one step above a flat file is similarly backward th…

> At a concept level, that's exactly the same thing. But at a practical level they very different. If you have a middleware layer as you are describing, it's written in a real programming language with all the adjacent tools (source control, debugging, etc). I'm not hard-core against stored procedures used lightly but they have a lot of downsides and they simply aren't needed. There's no performance advantage. There…

> If you have a middleware layer as you are describing, it's written in a real programming language with all the adjacent tools (source control, debugging, etc).

This is very much the point of SpacetimeDB. We write stored procedures in Rust or C# (or theoretically, any language that compiles to WASM), and we get all the same developer tools we're used to.

> There's no performance advantage.

There very much is a performance advantage to executing WASM in-process compared to communicating over an IPC barrier or, god forbid, a network.

Re: Databases are the endgame for data-oriented design

#112

Post by gorgoiler »

Has anyone here seen a database where your users are users in the db itself? Not just a user(id, name, email, password) table but actual db users with GRANTed permissions and ACLs etc set appropriately, and open access to the DB for these users.

It seems like it would solve a lot of problems by eliminating the need for data broker apps / endpoints that simply put POST or GET parameters into different SQL queries based on which endpoint you hit. I have no idea if it would scale but I don’t see why it couldn’t.

This sort of idea came up earlier this month in another post as well:

https://news.ycombinator.com/item?id=38489307

Re: Databases are the endgame for data-oriented design

#113

Post by zengid »

munificent wrote: post #23

As an ex-game developer and software architecture nerd, I'm very excited about data-oriented design and ECS. It really is a cool pattern, and it's a very common one in shipping games today. It's not just architecture astronaut stuff. At the same time, the level of hype about ECS today reminds me an awful lot of the amount of hype surrounding OOP in the 90s. Can ECS be a better way to structure your game entities and…

Your chapter on ECS in _Game_Programming_Patterns_ fueled my own obsession with the power of such a design, so you could be partly responsible ;).

I’ve spent a lot of time thinking about how to incorporate Data Oriented Design, SoA and ECS into the normal boring business logic at work, and I think it’s interesting to think about keeping data in the same form as it is in the relational database and skip the impedance mismatch in Object-Relational Mapping.

ECS only makes sense if you have real structs like in .NET or C/C++/Rust/Swift/etc, and tight latency requirements, but I think when someone learns a powerful concept like ECS, they want to invent a reason to use it so they can actually use it in action. I know that is what I’ve experienced at least.

Re: Databases are the endgame for data-oriented design

#114

Post by paulddraper »

setr wrote: post #98
paulddraper wrote: post #53

Earlier quoted context omitted.

How many application languages have percentile functions?

Nearly all of them, because any language designed in the last 30 years permits user defined libraries in trivial fashion, even amazingly letting you write those libraries in the same language you’ll use it. And even more incredibly, the libraries themselves can build on other libraries! (There’s even often a standard allowing libraries to be shared by different implementations … or rather, a standard that actually st…

Every database I've ever used is extensible

Re: Databases are the endgame for data-oriented design

#115

Post by ak39 »

crazygringo wrote: post #50
chrsig wrote: post #46

Earlier quoted context omitted.

> They're not though. That's why excellent coders often despise having to learn and use SQL, and basically just refuse to. SQL is uniquely terrible -- and I say this as someone with a career spanning from Win32 C++ to every major web technology. This is basically a strawman + appeal to authority argument. I've no desire to bicker. "Uniquely terrible" is quite subjective, we'll have to agree to disagree on if it appli…

Excuse me, but I'm certainly not "bickering" and I resent the insinuation. And you don't seem to understand what a strawman argument is. I'm not going to write an article on this for you, but you can certainly see the sentiment frequently expressed on HN that the rise of NoSQL and of ORM's is a direct result of otherwise capable programmers not wanting to deal with SQL and its conceptual building blocks such as joins…

ORMs and NoSQL were a direct result of the desire to have an object-oriented data worldview in a programming landscape dominated by OO languages. They were not introduced because programmers didn’t want to learn SQL (implying incompetence etc).

Re: Databases are the endgame for data-oriented design

#117

Post by devjab »

bob1029 wrote: post #84
fifilura wrote: post #4

My colleagues hate me, but I also found that SQL is The way to write business logic. Lots of caveats about difficulty to test and weird syntax. But it is just that SQL is the most terse and standard way so express logic. And that in itself is the most important factor to avoid bugs. Not what testing strategy you choose.

We've been doing this for a while and it has some pretty severe limitations at scale. The biggest issue is that humans are human and you cannot make everyone else on the team use your elegant SQL DSL business engine in exactly the same way you had envisioned. Eventually, it will grow into a bit of a monster. We are headed back in the other direction, but with some hedge. Code with a very simple & transparent CI/CD ex…

I think you make some decent points, but at the same time I really don’t understand this part:

> For non-expert team members, modern C# is turning out to be way more intuitive than SQL the moment you encounter a 3+ table join scenario.

Partly because this is so inefficient that it’s never going to work at scale, but also because of how you’re moving the complexity into the realm of “magic”.

I know it’s very easy and intuitive to use one of C#s ORMs, but with that comes an reliance on things like linq and a model builder, both of which may not work the way you think they do. If your developers think about it at all, and having seen so many C# developers use IEnumerable where they really should have been using IQuerable… well if your developers can’t do relatively basic SQL then I’m not sure I’d trust them with the abstraction either.

I don’t think you’re really, wrong, either, but I think it’s much more a question of building developer tools to help your developers handle good data access than it is about picking a particular tech on the consuming side.

As far as data storage goes, however, I think we’re just beginning to see the move away from the classics because the classics just weren’t build to support how we use data in 2023.

I can’t begin to tell you the nightmare it is to update old school SQL “data wells” to be capable of temporality, good BI access as well as being compliant with the various EU legislative rules, and almost none of that can be done above the database. Well I guess you could, but you’d probably go insane. And that’s just path of it, the other part is just how much data we transact now. It used to be that banks were basically the only organisations to move massive amounts of tiny bits of data, and now we all do it. Like, a single solar plant moves a gazillion points of data into your various systems a minute, where 30 years ago the entire data for that place might’ve been a couple of kb a year, it’s now mbs a day.

Re: Databases are the endgame for data-oriented design

#118

Post by hahn-kev »

gorgoiler wrote: post #112

Has anyone here seen a database where your users are users in the db itself? Not just a user(id, name, email, password) table but actual db users with GRANTed permissions and ACLs etc set appropriately, and open access to the DB for these users. It seems like it would solve a lot of problems by eliminating the need for data broker apps / endpoints that simply put POST or GET parameters into different SQL queries base…

Yes I have, it was a nightmare. The biggest scaling issue was that on a server you must open a new db connection for each request.

There's also little benefit because you still need to track application user data, so you still need a user table.

Re: Databases are the endgame for data-oriented design

#119

Post by lgas »

gorgoiler wrote: post #112

Has anyone here seen a database where your users are users in the db itself? Not just a user(id, name, email, password) table but actual db users with GRANTed permissions and ACLs etc set appropriately, and open access to the DB for these users. It seems like it would solve a lot of problems by eliminating the need for data broker apps / endpoints that simply put POST or GET parameters into different SQL queries base…

> Has anyone here seen a database where your users are users in the db itself?

PostgREST does this and it was a pretty nice experience when I used it.

https://postgrest.org/en/stable/references/auth.html

Re: Databases are the endgame for data-oriented design

#120

Post by rglover »

sgarland wrote: post #89
crazygringo wrote: post #50

Earlier quoted context omitted.

Excuse me, but I'm certainly not "bickering" and I resent the insinuation. And you don't seem to understand what a strawman argument is. I'm not going to write an article on this for you, but you can certainly see the sentiment frequently expressed on HN that the rise of NoSQL and of ORM's is a direct result of otherwise capable programmers not wanting to deal with SQL and its conceptual building blocks such as joins…

I take it as the same reason they don’t want to use boring technology like HAProxy, or why they think infrastructure in general is beneath them: because they are easily influenced by breathless grifters on Medium promoting the New Thing. SQL isn’t going away any time soon. At its heart, it’s relational algebra. If you can learn DS&A, you can learn relational algebra.

FWIW, I love HAProxy but hate SQL (for the exact reasons the OC suggests). It turns the simple into soup and makes bugs more likely. Every single non-trivial usage of it I've seen in the wild has been a ball of spaghetti. I bought the learning curve argument for a long time until I finally realized: no, SQL is a victim of its own success. It's popular because it's popular, not because it's a superior language. That popularity has encouraged a gambling addict level of doubling down that continues to gaslight less-experienced developers into believing it's "the way."

You're right that it isn't going away, but when it does, I'll be the first to dance on its grave.

Post reply on HN