Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

D1: Our SQL database

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 241 posts

Re: D1: Our SQL database

#111

Post by slashdev »

For a Cloudflare article, this one is surprisingly light on technical details. And for the product where it most matters.

I'm guessing this is a single master database with multiple read replicas. That means it's not consistent anymore (the C in ACID). Obviously reads after a write will see stale data until the write propogates.

I'm a bit curious how that replication works. Ship the whole db? Binary diffs of the master? Ship the SQL statements that did the write and reapply them? Lots of performance and other tradeoffs here.

What's the latency like? This likely doesn't run in every edge location. Does the database ship out on the first request. Get cached with an expiry? Does the request itself move to the database instead of running at the edge - like maybe this runs on a select subset of locations?

So many questions, but no details yet.

Re: D1: Our SQL database

#112

Post by 015a »

whitepaint wrote: post #63

Will they seriously challenge Azure, AWS and GCP eventually? Cloudflare is very innovative and what they are doing is really exciting.

The unique thing about Cloudflare's product offerings is how global-first they are; traditional cloud providers (AWS to DigitalOcean) have a very region-oriented domain model, with select christened services allowed or architected to be global (ex: AWS Cloudfront, IAM, Route53, that's about it there). That's their disaster/failure model; but all it really does is force cross-regional architecture onto the customer. Most customers don't bother.

In comparison, everything at CF is global. And its not just "global" from an AWS perspective of "we've got 14 regions and your stuff runs in all of them"; its global from 300+ points-of-presence, within 50ms of like 98% of all humans. CDN for compute, databases, etc.

CF has a way to go in DevEx on many of their products. For example; Workers, being based on V8 Isolates, is a pain to use even compared to e.g. Lambda. It's a battle of figuring out what's possible and what isn't within the runtime. But I'm sure it'll be improved!

Re: D1: Our SQL database

#113

Post by gigatexal »

chrisshroba wrote: post #65
vaughan wrote: post #48

Earlier quoted context omitted.

Has anyone tried to write a new modern SQLite?

DuckDB comes to mind, but I can't speak to its differences from SQLite. https://duckdb.org/

DuckDB is Olap SQLite. The vector engine is dope. But most of the innovation is in the OLAP stuff.

Re: D1: Our SQL database

#114

Post by geelen »

smarx007 wrote: post #74
vlovich123 wrote: post #42

Earlier quoted context omitted.

D1 does not throw away consistency. It’s built on top of Durable Objects which is globally strongly consistent.

"D1 will create read-only clones of your data, close to where your users are, and constantly keep them up-to-date with changes." Sounds like there will be no synchronous replication and instead there will be a background process to "constantly keep [read-only clones] up-to-date". This means that a stale read from an older read replica can occur even after a write transaction has successfully committed on the "primary…

Yeah, so you can always opt-in to strong consistency by transferring execution to the primary (see the "Embedded Compute" section of the blog). Then it's pretty much exactly the same as a DO.

Re: D1: Our SQL database

#115

Post by messe »

mariushn wrote: post #108
jgrahamc wrote: post #14

BTW R2 is open beta now: https://blog.cloudflare.com/r2-open-beta/

R2 is 3x more expensive than B2 (storage) https://www.backblaze.com/b2/cloud-storage-pricing.html Am I missing something? Is there no bandwidth cost at all?

Yep, you're not charged for egress.

Re: D1: Our SQL database

#116

Post by gigatexal »

Cthulhu_ wrote: post #27
jchw wrote: post #19

Earlier quoted context omitted.

I think it’s long overdue. While SQLite certainly has its limitations, it’s a winner in many categories. Even for sites with mild traffic using ordinary SQLite in PHP like a decade ago, it was always nice to use for its simplicity and the performance was totally acceptable. In comparison, the memory usage of typical relational database servers was high enough to make it hard to fit on a single lowend VPS with the sam…

The main thing for tuning SQLite will be how to open it, e.g. in write-ahead mode, to turn on foreign keys (this needs to be enabled manually), and whether it should wait to get a database lock on slower hardware before giving up. There's also some gotchas like if you mark an ID column as primary key, it'll use the rowid as key - which can be reused if a row is removed. So you need to explicitly set primary key AND a…

If you define a table with an integer primary key you get autoincrement as default at least in newer versions.

Re: D1: Our SQL database

#117

Post by teraflop »

the_duke wrote: post #97
prirun wrote: post #82

Earlier quoted context omitted.

* the big one for me: very limited migration support, requiring quite a lot of ceremony for common tasks (eg rewriting a whole table and swapping it out) I don't know where this idea of having to swap a whole table in SQLite came from, but it simply isn't true. Over the last 13 years I have upgraded production HashBackup databases at customer sites a total of 35 times without rewriting and swapping out tables by usin…

For a long time sqlite did not have DROP COLUMN and RENAME COLUMN support, which are both pretty essential. I'm embarrassed to admit that I didn't realize RENAME COLUMN was actually added in 3.25, almost four years ago. DROP COLUMN was only just added last year in 3.35. I'm surprised a database schema lasted 9/12 years without ever renaming or dropping a column. This changes things! But even now, ALTER TABLE is not t…

I'm not sure what you mean by "not transactional". SQLite implements transaction support at the "page" level, and builds all other database operations on top of it, which means anything that touches the bytes of the database file is transaction-safe. You can verify this for yourself:

    sqlite> CREATE TABLE foo(a,b,c);
    sqlite> INSERT INTO foo VALUES (1,2,3);
    sqlite> BEGIN;
    sqlite> ALTER TABLE foo DROP COLUMN b;
    sqlite> SELECT * FROM foo;
    1|3
    sqlite> ROLLBACK;
    sqlite> SELECT * FROM foo;
    1|2|3
It's of course still subject to SQLite's normal restrictions on locking, which means a long-running ALTER statement will block concurrent writers (and probably also concurrent readers if you're not running in WAL mode).

Re: D1: Our SQL database

#118

Post by sorenbs »

mwcampbell wrote: post #9

Any current or planned support for existing ORMs, such as Prisma or TypeOrm? Also, I wonder how hard it will be to migrate existing PostgreSQL databases and SQL statements. Of course, I understand if Cloudflare is focused on greenfield applications.

Prisma won't work with D1 out of the box. The primary limitations are:

- SQLite is traditionally embedded in an application, so Prisma interacts with it by mounting a file. Workers does not have a local filesystem, and D1 is exposed over the network through an API accessible from a Worker. Prisma will have to create a specific connector for D1. - Workers have a script size limit which is currently 1MB. My understanding is that Cloudflare will be increasing this in the future. We also have specific work to decrease the size of Prisma. Both of those will have to happen before Prisma could be used with D1.

Note that Prisma already support querying Postgres, MySQL, SQL Server and MongoDB from Cloudflare Workers through the Prisma Data Proxy, which will see a GA release next month.

We are also very excited about D1 as a way to bring a subset of data closer to users in order to deliver faster experiences. We hope this will be a way to bring the benefit of edge computing to larger organisations who cannot simply rearchitect everything to run on Workers.

Re: D1: Our SQL database

#119

Post by gigatexal »

Cthulhu_ wrote: post #32
mwcampbell wrote: post #9

Any current or planned support for existing ORMs, such as Prisma or TypeOrm? Also, I wonder how hard it will be to migrate existing PostgreSQL databases and SQL statements. Of course, I understand if Cloudflare is focused on greenfield applications.

Before you consider using an ORM, try using regular SQL and some tooling first; your future self will thank you. Just write the code, it's only volume and it's not so bad.

+1 to this as well.

Re: D1: Our SQL database

#120

Post by vlovich123 »

infogulch wrote: post #98
greg-m wrote: post #88

Earlier quoted context omitted.

Just clarifying - D1 without read replicas is strongly consistent. If you add read replicas, those can have replication lag and will not be strongly consistent. Disclaimer: I work at Cloudflare :)

Thanks for the clarification, that is what I would expect. Does SQLite support some kind of monotonic transaction id that can be used as a cache coherency key? Say a client writes a new record to the database which returns `{"result": "ok", "transaction_id": 123}`, then to ensure that subsequent read requests are coherent they provide a header that checks that the read replica has transaction_id >= 123 and either wai…

Yup sorry about that. I missed the entire "read replica" bit when reading that blog post.
Post reply on HN