Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Database internals are becoming less important than developer experience

planetscale.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Database internals are becoming less important than developer experience

#51

Post by dragontamer »

TekMol wrote: post #50
dragontamer wrote: post #44

Earlier quoted context omitted.

You don't even need to reach two nodes before SQLite becomes grossly inadequate. Even on a single node: SQLite's paradigm of global locks leads to poor performance when multiple threads write to the same table. You could be a single-node 4-core $5/month VPS instance and run into this issue. SQLite requires "exclusive" access to a table to handle writes (meaning when writing, no other thread can be reading the table).…

The question is if simultaneous operations really speed up your application. It is not as if a 4-core machine can do 4 times the DB work if you only allow it. Memory access, disk access .. they all have their specific behaviour when you try to do things simultaneously. In the worst case, things will just get serialized on a lower level, even if multiple CPU cores send and/or request data simultaneously.

> It is not as if a 4-core machine can do 4 times the DB work if you only allow it.

If Thread#1 goes "lock. write. unlock", then that means Thread#2, #3, and #4 all have to wait until Thread#1 is done with the write.

Even if Thread#2/#3/#4 have the data in their CPU-cache, they have to wait for Thread#1 to be complete.

--------

If Thread#1 is writing to a __hard drive__, that means Thread#2, #3, and #4 are waiting on a hard drive read, when they could have instead been reading from L3 cache.

SQLite's model scales extremely poorly in practice. You run into all sorts of problems like this.

Re: Database internals are becoming less important than developer experience

#52

Post by jasonwatkinspdx »

gagejustins wrote: post #40

~hello everyone, author here~ I know posts with ThOuGhT LeaDeRshIp titles like this are usually annoying, but I thought it would be interesting to write down some of the lessons I've been gathering as I've spent more time covering and using specific databases. My background is in data science / analytics with a couple of years of more traditional full stack here and there. Broadly we've seen this pattern with infrast…

I think the article's thesis is a false and misleading dichotomy.

It's absolutely true that a low friction developer experience is necessary for a database product to be successful. But this in no way implies that database internals are being commoditized or relegated to minor importance.

Snowflake is a particularly bad example as taking a clean sheet and novel approach to internals is the very fulcrum that creates the easy developer experience.

Admittedly it's been a while since I looked at vitess, but my recollection is that it's cross shard functionality is so limited as to make claiming internals no longer matter a bit dubious.

The reason there's only a handful of spanner style systems is exactly because the internals both matter and are quite daunting to get right.

Re: Database internals are becoming less important than developer experience

#53

Post by TekMol »

dragontamer wrote: post #51
TekMol wrote: post #50

Earlier quoted context omitted.

The question is if simultaneous operations really speed up your application. It is not as if a 4-core machine can do 4 times the DB work if you only allow it. Memory access, disk access .. they all have their specific behaviour when you try to do things simultaneously. In the worst case, things will just get serialized on a lower level, even if multiple CPU cores send and/or request data simultaneously.

> It is not as if a 4-core machine can do 4 times the DB work if you only allow it. If Thread#1 goes "lock. write. unlock", then that means Thread#2, #3, and #4 all have to wait until Thread#1 is done with the write. Even if Thread#2/#3/#4 have the data in their CPU-cache, they have to wait for Thread#1 to be complete. -------- If Thread#1 is writing to a __hard drive__, that means Thread#2, #3, and #4 are waiting on…

That is why I mentioned the "worst case".

You are describing the best case. Where everything is in a cache close enough to the CPU that it is not impacted by the other CPUs data access.

Re: Database internals are becoming less important than developer experience

#54

Post by jeffbee »

015a wrote: post #22
jeffbee wrote: post #11

Earlier quoted context omitted.

I can't imagine any scenario under which a reasonable person of at least median intelligence would perceive MongoDB as "easier to use" than Spanner, unless their entire experience with Mongo was they put one trivial JSON doc into an M0 cluster and got it back out later. Every practical aspect of MongoDB is a complete shitshow, from sharding to backup to failover.

"Sharding" "Backups" and "Failovers" are NOT "practical" aspects of any database. They're theoretical. Most databases are not big enough to need sharding. Most backups go unused. Most failover happens automatically, totally managed by your hosting provider. You know what is practical? Schema design. Query language. That's what made MongoDB super popular; no schemas to worry about. A query is just '{ firstName: "John"…

> failover happens automatically, totally managed by your hosting provider.

I see you've never used Mongo Atlas.

Re: Database internals are becoming less important than developer experience

#55

Post by fmakunbound »

> Database internals will eventually just not matter

Of course you need to know the internals of your database. If you've ever come across a project where the team treated a key/value, or document database as a relational one (probably because the query syntax looks similar), then you will know just how important database internals are.

Re: Database internals are becoming less important than developer experience

#57

Post by pgwhalen »

abanayev wrote: post #16
fabian2k wrote: post #13

Understanding a limited amount of database internals has been very useful to me. There is one aspect of using databases that you simply cannot abstract away and that is performance. If you ask your database a question in a way it is not suited to perform or that isn't supported by indexes performance is not going to be good. And these performance differences are not small once your database has a decent size. And if…

At some point, DBaaS systems should be able to understand and make inferences about your use cases, to the point where indexes and other performance optimizations are automated whenever you register a new query or something. This would be the new era of database systems, and as the article points out is increasingly true about all “infrastructure” concerns.

One of the problems with this is that there will always be trade offs. It’s hard to imagine a database understanding the appropriate compromise between read and write speeds for your specific application, for example.

Re: Database internals are becoming less important than developer experience

#58

Post by dogma1138 »

fmakunbound wrote: post #48
TekMol wrote: post #26

Earlier quoted context omitted.

You can certainly host it on a network drive if the network filesystem has the right features and behaviour. The same goes for a local filesystem. Sqlite has certain features it requires the filesystem to have. That is independent of how that filesystem stores the data physically.

> if the network filesystem has the right features and behaviour Which network filesystems are those?

SQLite actually works just fine on Windows files shares (and yes with multiple clients since Windows file shares do support file locks) but I wouldn’t recommend it as a remote DB/multi client solution.

Re: Database internals are becoming less important than developer experience

#59

Post by dragontamer »

TekMol wrote: post #53
dragontamer wrote: post #51

Earlier quoted context omitted.

> It is not as if a 4-core machine can do 4 times the DB work if you only allow it. If Thread#1 goes "lock. write. unlock", then that means Thread#2, #3, and #4 all have to wait until Thread#1 is done with the write. Even if Thread#2/#3/#4 have the data in their CPU-cache, they have to wait for Thread#1 to be complete. -------- If Thread#1 is writing to a __hard drive__, that means Thread#2, #3, and #4 are waiting on…

That is why I mentioned the "worst case". You are describing the best case. Where everything is in a cache close enough to the CPU that it is not impacted by the other CPUs data access.

I'm talking about relatively simple cases like spinning up a phpbb3 (web forum) instance. The minute you have multiple users writing comments at the same time from different apache/php instances is the minute your SQLite database starts to fall over.

Every write (aka: every comment post) is an exclusive write on SQLite. A full lock that prevents other processes from reading. (Ex: User#1 writes a comment, but User#2 hits refresh. These two items will be sequential if you use SQLite... when normally they'd be concurrent in most other databases)

------

SQLite is an exceptionally good database for many purposes. But it has its weaknesses. There's a reason why MySQL and PostgreSQL exist after all.

Re: Database internals are becoming less important than developer experience

#60

Post by munk-a »

DelightOne wrote: post #43
munk-a wrote: post #32

Earlier quoted context omitted.

The biggest database unification effort tends to be focused around ODBC compatibility from what I've seen - and the operations laid out by ODBC are quite trivial and easy to comprehend I think.

I more thought of a cli wrapper to enable simple file-like management of databases with operations like copy, list and not having to setup auth, but rather just provide the local path to the database. However, ODBC looks interesting too. ODBC might enable the creation of such a cli wrapper in a database-agnostic way. An authentication library retrieving the correct credentials based on the local file path may make th…

Yea I think that'd be an interesting approach - and ODBC takes care of all the annoying bits for you (providing you with a single unified API to interact with instead of tailoring the commands to each specific driver).
Post reply on HN