Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

D1: Our SQL database

blog.cloudflare.com

Post reply on HN
Print view View original

221–230 of 241 posts

Re: D1: Our SQL database

#221

Post by sophacles »

hunterb123 wrote: post #182
ctur wrote: post #154

Quoted post unavailable.

I prefer to make the world better through actions. Not pointlessly redefining words that have no ill intent in the first place and harassing people that don't use the words you decided were proper.

Just a thought, what if we made the world a better place through clearer terminology? It's not a major improvement, but it helps reduce friction in communications.

Ignoring connotations, master/slave is pretty unclear to me and needs more explanation. I've seen it used to describe:

* a coordinator with a worker pool in which the coordinator sends jobs to the workers for actual computation

* a designation of where on the bus a device sits

* a designation of which copy is the source of truth/gets writes (master) compared to the read only copies

* a designation of which instance is currently doing the work vs which is the standby in an active-active failover

Yes those are all similar, but it's annoying to hear someone say "this is configured in a typical master/slave pattern" only to be left wondering which of the above applies.

I guess we could make some sort of purity argument about which of these is the one-true use for master/slave. Or we could differentiate the cases with more applicable and descriptive words. As a bonus some folks aren't offended.

Re: D1: Our SQL database

#222

Post by peterhunt »

manigandham wrote: post #194
peterhunt wrote: post #174

Earlier quoted context omitted.

SQLite is great but it's way overhyped and abused on HN. People are very eager to turn SQLite into a durable, distributed database and it's really not meant for that, and by going down that road instead of using something like MySQL or Postgres you're missing out on lots of important functionality and tooling. I only say this because I have made this mistake at my previous startup. We built these really cool distribu…

So you didn't use SQLite then? Because RocksDB + Kafka is not similar at all. Also databases all use the same fundamental primitives and it's up to you to choose the level of abstraction you need. For example, FoundationDB is a durable distributed database that uses SQLite underneath as the storage layer but exposes an ordered key/value API, but then allows you to build your own relational DB on top. If you just need…

It's actually quite similar. Both are embedded storage engines that are designed for a single node.

Actually, the case for RocksDB for backing a distributed data store is probably much stronger than SQLite given that it supports multiple concurrent writers.

SQLite lacks many important characteristics that one would expect a distributed data store to have. Row level locking is one obvious feature that's super important in a highly concurrent context (as mentioned, RocksDB has this). Want to backup your production DB? You're going to need to block all writes until the backup completes.

Additionally, features like profiling and replication are nonexistent or immature with SQLite. Rqlite and Litestream are super new relative to tools like Postgres and MySQL and you can't find a lot of people that know how to run them.

Also, you can't autoscale your app since your processes are now highly stateful. Sure, this is a problem with MySQL/Postgres too, but I can pay AWS or Google Cloud for a managed version that will abstract this problem away from me.

Most of these problems are solvable with enough net new software on top of SQLite. But... why? I think the only reason you'd subject yourself to such an architecture is because you want to learn (great!) or you're gunning for that next promotion and need to show off your system design skills :P

Re: D1: Our SQL database

#223

Post by manigandham »

sorenbs wrote: post #218
manigandham wrote: post #162

Earlier quoted context omitted.

Because it's not painful to others and intent always matters. These words are everywhere in the language; you're not really changing anything with these antics other than derailing the subject to appease those who assume offense on behalf of an imagined group of people that can't distinguish context.

Most of us moved on to better terminology 4+ years ago. The only ones derailing conversations are grumps like yourself who refuse to get with the program. Why is this so important to you?

Who are you and what's this "program" you deem to impose on others?

No thanks, I'll stick with the actual majority that have mastered using relevant language and rational context in discussions without being slaves to performative social constructs.

Instead of assuming what's actually important to me, perhaps some introspection of why you immediately think of slavery in a computing context would be more helpful.

Re: D1: Our SQL database

#224

Post by 1egg0myegg0 »

anyfactor wrote: post #81
chrisshroba wrote: post #65

Earlier quoted context omitted.

DuckDB comes to mind, but I can't speak to its differences from SQLite. https://duckdb.org/

I haven't tried duckdb but I have been googling about it. I think I saw a discussion where it was mentioned that duckdb isn't a replacement for SQLite. It is an OLAP database [0] which makes its ingestion time slower than SQLite, I think. So it is meant for analytics but not as fullfledge replacement for SQLite. [0]: https://en.wikipedia.org/wiki/Online_analytical_processing Duckdb on HN: https://news.ycombinator.com…

Close! DuckDB has very fast bulk insert speeds, but slower individual row insertion/update speeds. (Disclaimer: I write docs for DuckDB)

Re: D1: Our SQL database

#225

Post by jen20 »

peterhunt wrote: post #174
kurinikku wrote: post #6

wow SQLite getting a lot of love these days https://tailscale.com/blog/database-for-2022 https://fly.io/blog/all-in-on-sqlite-litestream https://blog.cloudflare.com/introducing-d1

SQLite is great but it's way overhyped and abused on HN. People are very eager to turn SQLite into a durable, distributed database and it's really not meant for that, and by going down that road instead of using something like MySQL or Postgres you're missing out on lots of important functionality and tooling. I only say this because I have made this mistake at my previous startup. We built these really cool distribu…

It’s the default storage engine for FoundationDB - not sure many would agree that isn’t a “durable, distributed database”.

Re: D1: Our SQL database

#226

Post by MuffinFlavored »

peterhunt wrote: post #174
kurinikku wrote: post #6

wow SQLite getting a lot of love these days https://tailscale.com/blog/database-for-2022 https://fly.io/blog/all-in-on-sqlite-litestream https://blog.cloudflare.com/introducing-d1

SQLite is great but it's way overhyped and abused on HN. People are very eager to turn SQLite into a durable, distributed database and it's really not meant for that, and by going down that road instead of using something like MySQL or Postgres you're missing out on lots of important functionality and tooling. I only say this because I have made this mistake at my previous startup. We built these really cool distribu…

Is this any good? https://github.com/rqlite/rqlite

I've been looking for a turn key solution that is better than me running a single node Postgres instance "bare metal" or in a container.

postgres-operator seems cool but... k8s, pretty heavy I guess.

Re: D1: Our SQL database

#227

Post by peterhunt »

jen20 wrote: post #225
peterhunt wrote: post #174

Earlier quoted context omitted.

SQLite is great but it's way overhyped and abused on HN. People are very eager to turn SQLite into a durable, distributed database and it's really not meant for that, and by going down that road instead of using something like MySQL or Postgres you're missing out on lots of important functionality and tooling. I only say this because I have made this mistake at my previous startup. We built these really cool distribu…

It’s the default storage engine for FoundationDB - not sure many would agree that isn’t a “durable, distributed database”.

For one thing, they're ripping it out because of its poor write parallelism https://youtu.be/nlus1Z7TVTI?t=271

But that's orthogonal to my point. As a user of FoundationDB, you're not programming directly against SQLite, so you aren't going to run into these issues as much since FoundationDB exposes different semantics and coordinates concurrency across many SQLite instances in parallel.

I think it's best to think of SQLite as a replacement for your filesystem, rather than a replacement for your relational DBMS.

Re: D1: Our SQL database

#228

Post by [deleted] »

MuffinFlavored wrote: post #226
peterhunt wrote: post #174

Earlier quoted context omitted.

SQLite is great but it's way overhyped and abused on HN. People are very eager to turn SQLite into a durable, distributed database and it's really not meant for that, and by going down that road instead of using something like MySQL or Postgres you're missing out on lots of important functionality and tooling. I only say this because I have made this mistake at my previous startup. We built these really cool distribu…

Is this any good? https://github.com/rqlite/rqlite I've been looking for a turn key solution that is better than me running a single node Postgres instance "bare metal" or in a container. postgres-operator seems cool but... k8s, pretty heavy I guess.

[deleted]

Re: D1: Our SQL database

#229

Post by petre »

ripley12 wrote: post #185
peterhunt wrote: post #174

Earlier quoted context omitted.

SQLite is great but it's way overhyped and abused on HN. People are very eager to turn SQLite into a durable, distributed database and it's really not meant for that, and by going down that road instead of using something like MySQL or Postgres you're missing out on lots of important functionality and tooling. I only say this because I have made this mistake at my previous startup. We built these really cool distribu…

I'm not sure why you concluded that SQLite is the problem when you built a "really cool distributed database" with Kafka. Distributed databases are complicated, Kafka's complicated. If you're saying that a replicated Postgres setup would be simpler than what you're built, I agree; but SQLite+Litestream probably would be too.

Litestream is too much work if you're not using S3: replication over sftp. Even fossil has nicer no nonsense replication done over http/s. It's way easier to set up mysql with replication than manage unix accounts and public keys.

Re: D1: Our SQL database

#230

Post by [deleted] »

1egg0myegg0 wrote: post #224
anyfactor wrote: post #81

Earlier quoted context omitted.

I haven't tried duckdb but I have been googling about it. I think I saw a discussion where it was mentioned that duckdb isn't a replacement for SQLite. It is an OLAP database [0] which makes its ingestion time slower than SQLite, I think. So it is meant for analytics but not as fullfledge replacement for SQLite. [0]: https://en.wikipedia.org/wiki/Online_analytical_processing Duckdb on HN: https://news.ycombinator.com…

Close! DuckDB has very fast bulk insert speeds, but slower individual row insertion/update speeds. (Disclaimer: I write docs for DuckDB)

[deleted]
Post reply on HN