Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Database internals are becoming less important than developer experience

planetscale.com

Post reply on HN
Print view View original

61–70 of 76 posts

Re: Database internals are becoming less important than developer experience

#61

Post by bob1029 »

TekMol wrote: post #15

As a developer, I have to say that sqlite gives me the best experience. Everything else pales in comparison. Create a database? sqlite3 mydata.db Where is the database? In the current directory How is it structured on disk? It's a single file How do I backup the DB? cp mydata.db /my/backups/mydata.db Do I have to install a server? No Do I have to configure anything? No During setup and deployment I usually I dabble a…

We've been using sqlite as our only datastore for the last 5+ years.

There are a lot of fun little tricks you learn along the way.

The most important one is to ignore all the naysayers who claim it won't scale. I can easily saturate any IO subsystem by way of SQLite.

At first you may think it's impossible, but then you start to go outside the lines a bit. You start by turning on WAL. That helps a lot... But then you want another 100x? One thing you can do is heavily abuse the fact that any SQLite db is "just a file" by opening as many as you need to in parallel. 9/10 times you can break down a typical business app into grains roughly approximating the current state of each user.

Might make regulatory compliance easier if each user is stored in their own database too. Makes removing someone's entire presence from your system 1 file deletion activity.

Re: Database internals are becoming less important than developer experience

#62

Post by felixhuttmann »

jasonwatkinspdx wrote: post #52
gagejustins wrote: post #40

~hello everyone, author here~ I know posts with ThOuGhT LeaDeRshIp titles like this are usually annoying, but I thought it would be interesting to write down some of the lessons I've been gathering as I've spent more time covering and using specific databases. My background is in data science / analytics with a couple of years of more traditional full stack here and there. Broadly we've seen this pattern with infrast…

I think the article's thesis is a false and misleading dichotomy. It's absolutely true that a low friction developer experience is necessary for a database product to be successful. But this in no way implies that database internals are being commoditized or relegated to minor importance. Snowflake is a particularly bad example as taking a clean sheet and novel approach to internals is the very fulcrum that creates t…

I agree. It is also amazing how different the database systems are that are competing against each other today:

Partitioning: 1) DynamoDb: Partitioning is explicit and one of the most important parts of schema design 2) Spanner, Cockroach: Database automatically partitions the key ranges. 3) Postgres: You will probably never reach the scale where you need to partition your dataset!

Transactions: 1) Spanner, firestore - no stored procedures, client-side transactions are important 2) Dynamodb: No stored procedures, no client-side transactions, only transactions where all items involved are known by primary key in advance. 3) Fauna, Supabase: Stored procedures are the way to go! You do not need application code, access your database from the client! 4) Postgres: We have everything, use what fits your particular use-case!

If database internals did not matter, why are they all doing something different and are sometimes quite opinionated about it?

Re: Database internals are becoming less important than developer experience

#63

Post by johncolanduoni »

TekMol wrote: post #50
dragontamer wrote: post #44

Earlier quoted context omitted.

You don't even need to reach two nodes before SQLite becomes grossly inadequate. Even on a single node: SQLite's paradigm of global locks leads to poor performance when multiple threads write to the same table. You could be a single-node 4-core $5/month VPS instance and run into this issue. SQLite requires "exclusive" access to a table to handle writes (meaning when writing, no other thread can be reading the table).…

The question is if simultaneous operations really speed up your application. It is not as if a 4-core machine can do 4 times the DB work if you only allow it. Memory access, disk access .. they all have their specific behaviour when you try to do things simultaneously. In the worst case, things will just get serialized on a lower level, even if multiple CPU cores send and/or request data simultaneously.

We don’t have to talk in abstract about this, you can benchmark these things or just look at the many ones run by others. For example, some benchmarks around some Postgres scaling improvements: https://wiki.postgresql.org/images/e/e8/FOSDEM2012-Multi-CPU....

If you don’t need anything Postgres offers, by all means stick with sqlite. But your pessimism about being stuck with operations “serialized on a lower level” is not empirically justified for most of the databases that support multi-threading.

Re: Database internals are becoming less important than developer experience

#64

Post by [deleted] »

bob1029 wrote: post #61
TekMol wrote: post #15

As a developer, I have to say that sqlite gives me the best experience. Everything else pales in comparison. Create a database? sqlite3 mydata.db Where is the database? In the current directory How is it structured on disk? It's a single file How do I backup the DB? cp mydata.db /my/backups/mydata.db Do I have to install a server? No Do I have to configure anything? No During setup and deployment I usually I dabble a…

We've been using sqlite as our only datastore for the last 5+ years. There are a lot of fun little tricks you learn along the way. The most important one is to ignore all the naysayers who claim it won't scale. I can easily saturate any IO subsystem by way of SQLite. At first you may think it's impossible, but then you start to go outside the lines a bit. You start by turning on WAL. That helps a lot... But then you…

[deleted]

Re: Database internals are becoming less important than developer experience

#65

Post by Tenoke »

bob1029 wrote: post #61
TekMol wrote: post #15

As a developer, I have to say that sqlite gives me the best experience. Everything else pales in comparison. Create a database? sqlite3 mydata.db Where is the database? In the current directory How is it structured on disk? It's a single file How do I backup the DB? cp mydata.db /my/backups/mydata.db Do I have to install a server? No Do I have to configure anything? No During setup and deployment I usually I dabble a…

We've been using sqlite as our only datastore for the last 5+ years. There are a lot of fun little tricks you learn along the way. The most important one is to ignore all the naysayers who claim it won't scale. I can easily saturate any IO subsystem by way of SQLite. At first you may think it's impossible, but then you start to go outside the lines a bit. You start by turning on WAL. That helps a lot... But then you…

How do you aggregate the data and query among all those databases? Doesn't it incur huge costs? Does sqlite support this natively or are you basically treating it as text files of data per user and don't need complex queries in the first place?

Re: Database internals are becoming less important than developer experience

#66

Post by ColonelPhantom »

dragontamer wrote: post #44
johncolanduoni wrote: post #33

Earlier quoted context omitted.

Unless you need to do something crazy, like run two nodes of your application. But only Google needs to do that, right?

You don't even need to reach two nodes before SQLite becomes grossly inadequate. Even on a single node: SQLite's paradigm of global locks leads to poor performance when multiple threads write to the same table. You could be a single-node 4-core $5/month VPS instance and run into this issue. SQLite requires "exclusive" access to a table to handle writes (meaning when writing, no other thread can be reading the table).…

I believe that SQLite in Write-Ahead Log (WAL) mode does not have this issue.

Re: Database internals are becoming less important than developer experience

#67

Post by bob1029 »

Tenoke wrote: post #65
bob1029 wrote: post #61

Earlier quoted context omitted.

We've been using sqlite as our only datastore for the last 5+ years. There are a lot of fun little tricks you learn along the way. The most important one is to ignore all the naysayers who claim it won't scale. I can easily saturate any IO subsystem by way of SQLite. At first you may think it's impossible, but then you start to go outside the lines a bit. You start by turning on WAL. That helps a lot... But then you…

How do you aggregate the data and query among all those databases? Doesn't it incur huge costs? Does sqlite support this natively or are you basically treating it as text files of data per user and don't need complex queries in the first place?

We don't have any use cases for querying across these datastores right now. Hypothetically, if we did need something like this it would probably be an offline ETL analysis tool. I have written a few of these and they can be incredibly performant if everything lives on the same box.

Re: Database internals are becoming less important than developer experience

#68

Post by hodgesrm »

015a wrote: post #22
jeffbee wrote: post #11

Earlier quoted context omitted.

I can't imagine any scenario under which a reasonable person of at least median intelligence would perceive MongoDB as "easier to use" than Spanner, unless their entire experience with Mongo was they put one trivial JSON doc into an M0 cluster and got it back out later. Every practical aspect of MongoDB is a complete shitshow, from sharding to backup to failover.

"Sharding" "Backups" and "Failovers" are NOT "practical" aspects of any database. They're theoretical. Most databases are not big enough to need sharding. Most backups go unused. Most failover happens automatically, totally managed by your hosting provider. You know what is practical? Schema design. Query language. That's what made MongoDB super popular; no schemas to worry about. A query is just '{ firstName: "John"…

I work for a company that supports ClickHouse. Our focus is analytic systems, which tend to run large compared to OLTP systems.

* Sharding is part of schema design for any analytic app whose data exceeds the capacity of a single host. This is very common for use cases like web analytics, observability, network management, intrusion detection, to name just a few. Automatic resharding is one of the top asks in the ClickHouse community. (We're working on it.)

* How do I backup ClickHouse is one of our top 3 support questions in order of frequency. I just taught a ClickHouse class yesterday--part of a regular series--and it was the first question out of the gate from the audience. It has come up at some point in almost every customer engagement I can think of.

In my experience, your comment is only correct for relatively small applications that are not critical to the business.

Re: Database internals are becoming less important than developer experience

#69

Post by hodgesrm »

Chiron1991 wrote: post #4

Friendly reminder that this post is published on the PlanetScale blog, a company that sells a database SaaS. Beware of the bias. I personally would argue with every single point this article makes, except scalability.

> I personally would argue with every single point this article makes, except scalability.

Maybe you could be more explicit about what you don't like about their ideas? I personally do like a lot of their ideas, such as the following:

> In the future, I’d expect to see a tighter coupling between the frameworks we’re using for reactive frontends – React, Vue, etc. – and the database, via hooks or otherwise.

This builds on the behavior that made MongoDB so phenomenally popular, as the article points out. Data management is pervasive in modern applications and anything that makes it easier for devs to implement is goodness.

Post reply on HN