Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Databases are the endgame for data-oriented design

spacetimedb.com

Post reply on HN
Print view View original

91–100 of 157 posts

Re: Databases are the endgame for data-oriented design

#91

Post by hot_gril »

whstl wrote: post #74
hot_gril wrote: post #32

Earlier quoted context omitted.

Same, every backend service I write now has the majority of the business logic in SQL and a little pre/post-processing in regular code. A well-designed schema will mean that your queries don't get messy. If some more complex thing starts feeling forced, I add a bit more non-SQL code to make it reasonable. When teammates look at my code, and logic is all right there instead of scattered around, and there's a schema fi…

The fear of committing SQL to the codebase is one of the most baffling things in modern backends. To me the most infuriating thing is the "SQL query scattered around multiple files" pattern, where a backend engineer will decompose a perfectly fine SQL query into 3 or 4 files, with multiple functions, often with very artificial separations (for example: a function just for the "select ..." part, another for the joins)…

We're using some tool at work that recommends putting SQL into dedicated separate files, one file per query. Or you can use the query builder which is just SQL but with extra steps, limitations, and caveats. It's so annoying.

Re: Databases are the endgame for data-oriented design

#92

Post by whizzter »

ndriscoll wrote: post #80
buryat wrote: post #10

Earlier quoted context omitted.

that's how you end up with multi-thousand LOC sql files that contain all the historical business logic and edge cases of a company and it just keeps being piled up on and its complexity keeps growing because not everything is expressible in SQL. So you end up having a big monster that someone needs to support and one that's not very testable easily

Testing SQL is easy. E.g. the JVM has H2, which works for simple stuff, or you can use testcontainers, or just spin up a container and run your tests against that. You just run your migrations, insert mock data, and run your test. In fact testability is one of the best parts. You can safely test read-only queries against a prod secondary database to see that it gives reasonable results on real data, and use the repl…

SQL queries yes, what Bury above probably is talking about is obscure triggers/stored-procs,etc that encodes the business logic inside the database far away from any version control or sane ways to track it.

Re: Databases are the endgame for data-oriented design

#93

Post by SahAssar »

crazygringo wrote: post #50
chrsig wrote: post #46

Earlier quoted context omitted.

> They're not though. That's why excellent coders often despise having to learn and use SQL, and basically just refuse to. SQL is uniquely terrible -- and I say this as someone with a career spanning from Win32 C++ to every major web technology. This is basically a strawman + appeal to authority argument. I've no desire to bicker. "Uniquely terrible" is quite subjective, we'll have to agree to disagree on if it appli…

Excuse me, but I'm certainly not "bickering" and I resent the insinuation. And you don't seem to understand what a strawman argument is. I'm not going to write an article on this for you, but you can certainly see the sentiment frequently expressed on HN that the rise of NoSQL and of ORM's is a direct result of otherwise capable programmers not wanting to deal with SQL and its conceptual building blocks such as joins…

> you can certainly see the sentiment frequently expressed on HN that the rise of NoSQL and of ORM's is a direct result of otherwise capable programmers not wanting to deal with SQL and its conceptual building blocks such as joins

I think I saw that sentiment a lot 5+ years ago. The last 2-3 years have been a lot more for postgresql and writing SQL directly.

Either way I don't think that "excellent coders" "despise" learning SQL, almost every senior coder that has interacted with a medium-to-large database has understood and valued SQL in my experience.

Re: Databases are the endgame for data-oriented design

#95

Post by whizzter »

hitpointdrew wrote: post #12

Nope, nope and nope. Went to the github page for spacetimedb, it does everything that is terrible. >Instead of deploying a web or game server that sits in between your clients and your database, your clients connect directly to the database and execute your application logic inside the database itself. You can write all of your permission and authorization logic right inside your module just as you would in a normal…

I'm writing a quite similar system to this so I can give the differences why this is useful (esp for this usecase).

1: Separating stores was crucial during the 90s and earlier when people were still writing in memory-unsafe languages (C/C++ cough) since it could cause wild corruptions with simple stray pointers. Process-separation was just a sanity thing. As you notice these are other languages in play here so random corruptions shouldn't happen (memory exhaustion can still be a thing though with their model)

2: Yes, debugging stored procedures/triggers on SQL-Server,etc is a PITA because they're database first centric objects, however the idea here is to make the database fill the job of app-servers of the 90s/00s with "regular" debugging workflows for developers. (Don't confuse implementations with the concept)

3: And MOST importantly, this is a game-focused thing, gamedevs will often end up replicating most cache/database functionality (badly) to squeeze things into main memory with the goal to achieve realtime performance targets anyhow, why not forgo that duplicated crazy work with a solid framework?

4: As a corollary to the above, the benefits of separating storage from applications (to run multiple applications against the dataset as it often happens in enterprise scenarios) isn't really the focus, application to database mappings are intended to be more or less 1:1

Re: Databases are the endgame for data-oriented design

#96

Post by rendall »

Hi spacetimedb.com. You apparently want people to read about your product, but have a non-GDPR-compliant popup that requires people to uncheck multiple "vendors" if your readers do not want to be tracked.

Edit: it actually does not allow any unchecking. Just "you agree to these marketing and tracking cookies by using this site" Nope.

Baffling. It does not lend confidence in your core product. If you're not respectful of your casual reader's data, how can we expect you to be careful with your user's data?

I recommend zero tracking, or if you must, have it be genuinely opt-in.

Re: Databases are the endgame for data-oriented design

#97

Post by hot_gril »

hot_gril wrote: post #36
jerf wrote: post #26

Earlier quoted context omitted.

"At the same time, the level of hype about ECS today reminds me an awful lot of the amount of hype surrounding OOP in the 90s." I appreciate that most of the ECS hype has been around specific use cases, though. OOP was claimed as not a specific useful tool, but the answer to all programming, a billing it has not lived up to. It achieved "useful tool", no question, especially as some of the very rough bits were sanded…

OOP even got mixed with databases to form nasty hype around ORMs and NoSQL.

But to their credit, there was a feature gap in SQL before jsonb was added.

Re: Databases are the endgame for data-oriented design

#98

Post by setr »

paulddraper wrote: post #53
crazygringo wrote: post #44

Earlier quoted context omitted.

> The types of issues you're enumerating are really part of a learning curve that every language and environment will have. They're not though. That's why excellent coders often despise having to learn and use SQL, and basically just refuse to. SQL is uniquely terrible -- and I say this as someone with a career spanning from Win32 C++ to every major web technology. > every major database will have some percentile fun…

How many application languages have percentile functions?

Nearly all of them, because any language designed in the last 30 years permits user defined libraries in trivial fashion, even amazingly letting you write those libraries in the same language you’ll use it. And even more incredibly, the libraries themselves can build on other libraries! (There’s even often a standard allowing libraries to be shared by different implementations … or rather, a standard that actually standardizes things beyond the language aesthetic)

Databases follow the COBOL model of extensibility — tis for me and not for thee. It is only to be done by the vendor, and maybe some third party specialists

Re: Databases are the endgame for data-oriented design

#99

Post by icedchai »

whstl wrote: post #74
hot_gril wrote: post #32

Earlier quoted context omitted.

Same, every backend service I write now has the majority of the business logic in SQL and a little pre/post-processing in regular code. A well-designed schema will mean that your queries don't get messy. If some more complex thing starts feeling forced, I add a bit more non-SQL code to make it reasonable. When teammates look at my code, and logic is all right there instead of scattered around, and there's a schema fi…

The fear of committing SQL to the codebase is one of the most baffling things in modern backends. To me the most infuriating thing is the "SQL query scattered around multiple files" pattern, where a backend engineer will decompose a perfectly fine SQL query into 3 or 4 files, with multiple functions, often with very artificial separations (for example: a function just for the "select ..." part, another for the joins)…

I hate this, also. This was a "design pattern" (antipattern) at a previous company. It was incredibly difficult to navigate SQL built like this. We were also asked to write "unit tests" for each micro-function. This was essentially a pointless exercise that compared each function call to a string, copy-and-pasted from the function itself, with no regard to the correctness of the overall query. At least it was easy to make the tests pass.

Re: Databases are the endgame for data-oriented design

#100

Post by stuhood »

The fundamental difference between an ECS and a struct/object layout is that an ECS is column-oriented (aka columnar), while a struct/object layout is row-oriented.

Everything else about how you might query these layouts is more superficial... you can provide the same API with either layout, the same way you can in relational database systems (both layouts can be queried with SQL, but with different performance characteristics.)

Post reply on HN