Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

D1: Our SQL database

blog.cloudflare.com

Post reply on HN
Print view View original

211–220 of 241 posts

Re: D1: Our SQL database

#211

Post by discodave »

infogulch wrote: post #98
greg-m wrote: post #88

Earlier quoted context omitted.

Just clarifying - D1 without read replicas is strongly consistent. If you add read replicas, those can have replication lag and will not be strongly consistent. Disclaimer: I work at Cloudflare :)

Thanks for the clarification, that is what I would expect. Does SQLite support some kind of monotonic transaction id that can be used as a cache coherency key? Say a client writes a new record to the database which returns `{"result": "ok", "transaction_id": 123}`, then to ensure that subsequent read requests are coherent they provide a header that checks that the read replica has transaction_id >= 123 and either wai…

Since it's a relational DB, and supports transactions, you can have a journal table right?

I know of a very important system at AWS that did this with MySQL :D

Re: D1: Our SQL database

#212

Post by eloff »

dragonwriter wrote: post #124
slashdev wrote: post #111

For a Cloudflare article, this one is surprisingly light on technical details. And for the product where it most matters. I'm guessing this is a single master database with multiple read replicas. That means it's not consistent anymore (the C in ACID). Obviously reads after a write will see stale data until the write propogates. I'm a bit curious how that replication works. Ship the whole db? Binary diffs of the mast…

> I'm guessing this is a single master database with multiple read replicas. That means it's not consistent Single master with read replicas is fully consistent if commits don't return until propagated to and acknowledged by replicas (the expense here being commit latency.)

I would say the expense is both latency and availability because if one node doesn't ack within the timeframe then you have to drop it from the cluster. Requests that go there would need to be routed elsewhere to avoid being unavailable. If there's a network partition preventing that, then you have partial downtime. If enough nodes fail then you have full downtime across the whole cluster.

Re: D1: Our SQL database

#213

Post by discodave »

didip wrote: post #142

All these hype around SQLite recently and I am still confused. * How do you replicate it consistently? * Who has the master privilege (or masters if sharded)? What's the failover story? I am guessing a blob store is involved, but I have gaps in my understanding here.

SQLite has a write ahead log (journal) mode. If you write that log to some store that is already replicated (S3, CloudFlare Durable Objects, Kafka?) then the concept of a 'master' is less important.

Re: D1: Our SQL database

#214

Post by irq-1 »

Best Effort Writes[1] are an opportunity here. Non-transactional, write to the local replica (ensure foreign keys, constrains, valid data, etc...) and then try to write to the main write-enabled DB. Caching should work without changes since the local replica is updated. This could be cheaper (send binary diffs) and more resilient to brief network issues.

The key is to let the user decide what really needs ACID and what doesn't. If someone wants to make the next Facebook or Reddit they'll need huge write throughput and if some votes or updates are lost, that may be a good trade-off.

[1] You could add a BEW file (like WAL file) to sqlite for Best Effort Writes.

Re: D1: Our SQL database

#215

Post by prirun »

the_duke wrote: post #97
prirun wrote: post #82

Earlier quoted context omitted.

* the big one for me: very limited migration support, requiring quite a lot of ceremony for common tasks (eg rewriting a whole table and swapping it out) I don't know where this idea of having to swap a whole table in SQLite came from, but it simply isn't true. Over the last 13 years I have upgraded production HashBackup databases at customer sites a total of 35 times without rewriting and swapping out tables by usin…

For a long time sqlite did not have DROP COLUMN and RENAME COLUMN support, which are both pretty essential. I'm embarrassed to admit that I didn't realize RENAME COLUMN was actually added in 3.25, almost four years ago. DROP COLUMN was only just added last year in 3.35. I'm surprised a database schema lasted 9/12 years without ever renaming or dropping a column. This changes things! But even now, ALTER TABLE is not t…

> I'm surprised a database schema lasted 9/12 years without ever renaming or dropping a column.

I did have a couple of columns that were no longer needed and would have dropped them, but instead I just set them to null and ignored them. Nulls only take 1 byte of space in a row. I dropped them when DROP COLUMN was added.

Re: D1: Our SQL database

#216

Post by jve »

endisneigh wrote: post #40
nindalf wrote: post #36

Earlier quoted context omitted.

Which problems were you thinking of? Cloudflare and fly.io both promise hassle free read replicas and backup. They will both offer only a single node capable of writes, because that’s how SQLite rolls. This is a pretty good fit for a read heavy load that requires SQL and very low latency.

I guess I’m not understanding what the benefit is vs hosted Postgres. Also low latency and setup can be equally trivial - see supabase for example.

Hope this can give you some concrete answers: https://www.sqlite.org/whentouse.html

Re: D1: Our SQL database

#218

Post by sorenbs »

manigandham wrote: post #162
nwsm wrote: post #159

Earlier quoted context omitted.

Quoted post unavailable.

Because it's not painful to others and intent always matters. These words are everywhere in the language; you're not really changing anything with these antics other than derailing the subject to appease those who assume offense on behalf of an imagined group of people that can't distinguish context.

Most of us moved on to better terminology 4+ years ago. The only ones derailing conversations are grumps like yourself who refuse to get with the program. Why is this so important to you?

Re: D1: Our SQL database

#220

Post by rad_gruchalski »

otoolep wrote: post #158
slashdev wrote: post #111

For a Cloudflare article, this one is surprisingly light on technical details. And for the product where it most matters. I'm guessing this is a single master database with multiple read replicas. That means it's not consistent anymore (the C in ACID). Obviously reads after a write will see stale data until the write propogates. I'm a bit curious how that replication works. Ship the whole db? Binary diffs of the mast…

I agree -- this blog post is light on details. To me the value Cloudflare believes they are offering is mostly ease-of-use, particularly setup. With minimal work you can have a stateful, relational store available to your code. But in terms of actual database functionality, they are not offering anything particularly novel. Of course, I might be missing something. In fact, I don't see anything D1 is doing that is not…

I’ve been looking at rqlite for some time and it’s really great to track the product on github.

I believe that the power of what Cloudflare offers here isn’t in the actual database. It’s the packaging and how it sits in their serverless world. Even with rqlite, I still need ip addresses to run a resilient system. As someone who sometimes needs a table here snd there, I really, really don’t want a server. I want a table to store a thousand records in and that’s it. This is where I would very much enjoy using something like D1.

A combo of D1, R2 and Workers is a serious contender for over-the-top serverless distributed apps. This is great.

Post reply on HN