Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

D1: Our SQL database

blog.cloudflare.com

Post reply on HN
Print view View original

201–210 of 241 posts

Re: D1: Our SQL database

#201

Post by isoprophlex »

SQLite wrote: post #184
cryptonector wrote: post #145

Earlier quoted context omitted.

It would really help if SQLite3 had a `MERGE`, or, failing that, `FULL OUTER JOIN`. In fact, I want it to have `FULL OUTER JOIN` even if it gains a `MERGE`. `FULL OUTER JOIN` is the secret to diff'ing table sources. `MERGE` is just a diff operation + insert/update/delete statements to make the target table more like the source one (or even completely like the source one). `FULL OUTER JOIN` is essential to implementin…

RIGHT and FULL JOIN are on the trunk branch of SQLite and will (very likely) appear in the next release. Please grab a copy of the latest pre-release snapshot of SQLite ( https://sqlite.org/download.html ) and try out the new RIGHT/FULL JOIN support. Report any problems on the forum, or directly to me at drh at sqlite dot org.

This is fantastic news, I'm very glad to hear that this is appearing soon! Thanks!

Re: D1: Our SQL database

#202

Post by Thaxll »

nibab wrote: post #144
Thaxll wrote: post #59

Earlier quoted context omitted.

Well I don't think it's a good fit for regular service, exactly how do you handle 2 replicas of the same service talking to the same DB? The fact that it's just a file on disk limits the usage.

Projects such as litestream and rqlite have this figured out.

Mutliple writer on the same SQLLite?

Re: D1: Our SQL database

#203

Post by Thaxll »

manigandham wrote: post #197
Thaxll wrote: post #59

Earlier quoted context omitted.

Well I don't think it's a good fit for regular service, exactly how do you handle 2 replicas of the same service talking to the same DB? The fact that it's just a file on disk limits the usage.

Transactions, locks, queues, etc. No different than multiple app instances changing the same row in other databases. Any state mutation is ultimately ordered in time and how that that ordering is accomplished depends on the abstractions you're using: in your app, network layer, database, etc.

Why would you use SQLlite once you start dealing with network, just use MySQl or PG.

It's just re-inventing the wheel badly, I need to read the details but basically you're using a tool SQLLite that was not designed to be used outside of a single app use case.

Re: D1: Our SQL database

#204

Post by ignoramous »

manigandham wrote: post #194
peterhunt wrote: post #174

Earlier quoted context omitted.

SQLite is great but it's way overhyped and abused on HN. People are very eager to turn SQLite into a durable, distributed database and it's really not meant for that, and by going down that road instead of using something like MySQL or Postgres you're missing out on lots of important functionality and tooling. I only say this because I have made this mistake at my previous startup. We built these really cool distribu…

So you didn't use SQLite then? Because RocksDB + Kafka is not similar at all. Also databases all use the same fundamental primitives and it's up to you to choose the level of abstraction you need. For example, FoundationDB is a durable distributed database that uses SQLite underneath as the storage layer but exposes an ordered key/value API, but then allows you to build your own relational DB on top. If you just need…

> So you didn't use SQLite then? Because RocksDB + Kafka is not similar at all.

To me, I could make the connection in the sense that just like sqlite, rocksdb is an embedded store, while Kafka can be used to build a replicated log (log device).

> If you just needed distributed SQL because a single instance wasn't enough then there are already plenty of choices...

Well, that was GP's point, too? In addition, they mention that existing DBMS like Postgres have way more breath and depth than a replicated sqlite can ever hope to have (which isn't really a controversial assertion at all, tbh).

Re: D1: Our SQL database

#205

Post by manigandham »

Thaxll wrote: post #203
manigandham wrote: post #197

Earlier quoted context omitted.

Transactions, locks, queues, etc. No different than multiple app instances changing the same row in other databases. Any state mutation is ultimately ordered in time and how that that ordering is accomplished depends on the abstractions you're using: in your app, network layer, database, etc.

Why would you use SQLlite once you start dealing with network, just use MySQl or PG. It's just re-inventing the wheel badly, I need to read the details but basically you're using a tool SQLLite that was not designed to be used outside of a single app use case.

What context are you talking about here?

For Cloudflare, they're offering it because it's simple and lightweight, and they already have their Durable Objects product which serves as the transaction ordering mechanism and takes care of writes.

If you're doing it yourself then sure it's probably not the best fit but that's up to you to decide.

Re: D1: Our SQL database

#206

Post by sigzero »

jgrahamc wrote: post #18
kurinikku wrote: post #6

wow SQLite getting a lot of love these days https://tailscale.com/blog/database-for-2022 https://fly.io/blog/all-in-on-sqlite-litestream https://blog.cloudflare.com/introducing-d1

SQLite has been cool forever. It was the underlying data store for my machine learning email filter POPFile 20 years ago! https://en.wikipedia.org/wiki/POPFile https://getpopfile.org/browser/trunk/engine/POPFile/Database...

I used POPFile!! It was awesome.

Re: D1: Our SQL database

#208

Post by irq-1 »

jgrahamc wrote: post #10
mwcampbell wrote: post #9

Any current or planned support for existing ORMs, such as Prisma or TypeOrm? Also, I wonder how hard it will be to migrate existing PostgreSQL databases and SQL statements. Of course, I understand if Cloudflare is focused on greenfield applications.

We are definitely interested in ORMs. Want to make it easy to use. I hope someone creates the next Rails using Workers. And having other models on top of our SQL offerings will be important. Get in contact and let us know what you'd like.

This should have a virtual file system. CF should write it so each user doesn't have to load a JS abstraction and it has better performance.

Re: D1: Our SQL database

#210

Post by eloff »

sebk wrote: post #132
slashdev wrote: post #111

For a Cloudflare article, this one is surprisingly light on technical details. And for the product where it most matters. I'm guessing this is a single master database with multiple read replicas. That means it's not consistent anymore (the C in ACID). Obviously reads after a write will see stale data until the write propogates. I'm a bit curious how that replication works. Ship the whole db? Binary diffs of the mast…

Small nitpick, but that's still consistent as in ACID. I think what you mean is it wouldn't be consistent in the CAP sense (it wouldn't be linearizable). TFA does say that read-replicas will be present at every edge location, which makes sense for a product like Workers. But it doesn't mention writes at all.

Yes, that's true.
Post reply on HN