Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-latency SQLite storage in every Durable Object

simonwillison.net

Post reply on HN
Print view View original

101–108 of 108 posts

Re: Zero-latency SQLite storage in every Durable Object

#101

Post by simonw »

bluehatbrit wrote: post #100
simonw wrote: post #98

Earlier quoted context omitted.

A pattern where you check for and then execute any necessary migrations on initialization of a Durable Object would actually work pretty well I think - presumably you can update the code for these things without erasing the existing database?

Ah yes, that would work pretty well! You'd have to be able to guarantee any migrations can run within the timeouts, but at a per-tenant level that should be very doable for most cases. Not sure why I didn't think of that approach - great idea. I might have to try this out now.

I think those SQLite databases are capped at 1GB right now, so even complex migrations (that work by creating a new temporary table, copying old data to it and then atomically renaming it) should run in well under a second.

Re: Zero-latency SQLite storage in every Durable Object

#102

Post by paulgb »

anentropic wrote: post #84
braden-lk wrote: post #36

Durable objects seem so cool but the pricing always scares me. (Specifically, having to worry about getting hibernation right.) They’d be a great fit for our yjs document based strategy, but while everything in prod still works on plain ol redis and Postgres, it’s hard to justify an exploration.

What scares me is it is super specific to Cloudflare What is your option if you want to eject to another cloud?

For the specific example of a Yjs backend, I happen to be working on one that can be hosted either on Cloudflare or as a native process. We’ve had people running in production migrate from cloudflare to native just by swapping out the URL they connect to in their application config.

https://github.com/jamsocket/y-sweet

Re: Zero-latency SQLite storage in every Durable Object

#103

Post by kentonv »

kentonv wrote: post #91
emadda wrote: post #85

Earlier quoted context omitted.

E.g. if you have many websocket connections and they each have a snapshot at a point in time (that spans over many different await function calls/ws messages). SQLite can have many readers and a single writer with WAL, so a many read transactions can exist whilst the writers move the db state forward.

We (Cloudflare) have considered adding an API to create multiple "database connections" , especially to be able to stream a response from a long-running cursor while representing a consistent snapshot of the data. It's a bit tricky since if you hold open that old connection, the WAL could grow without bound and cannot be checkpointed back into the main database. What do we do when the WAL gets unreasonably large (e.g…

Ugh didn't notice until too late to edit, but apparently HN interpreted my asterisk as an instruction to italicize everything between it and the footnote it referred to.

Re: Zero-latency SQLite storage in every Durable Object

#104

Post by emadda »

kentonv wrote: post #91
emadda wrote: post #85

Earlier quoted context omitted.

E.g. if you have many websocket connections and they each have a snapshot at a point in time (that spans over many different await function calls/ws messages). SQLite can have many readers and a single writer with WAL, so a many read transactions can exist whilst the writers move the db state forward.

We (Cloudflare) have considered adding an API to create multiple "database connections" , especially to be able to stream a response from a long-running cursor while representing a consistent snapshot of the data. It's a bit tricky since if you hold open that old connection, the WAL could grow without bound and cannot be checkpointed back into the main database. What do we do when the WAL gets unreasonably large (e.g…

I did guess it might be harder to do than vanilla SQLite, as vanilla SQLite just has the WAL and main db on the same hard drive, so it has more space to grow the WAL and it is not an issue when the machine/instance reboots (as it just starts where it left off, even if the WAL is large and has not been check-pointed back to the main db).

To be honest this is an edge case. But I often start a read transaction on a SQLite connection just so I know multiple queries are reading from the same state (and to ensure state has not been changed between queries).

Re: Zero-latency SQLite storage in every Durable Object

#105

Post by kentonv »

tmikaeld wrote: post #57

> ..each DO constantly streams a sequence of WAL entries to object storage - batched every 16MB or every ten seconds. Which also means it may take 10 seconds before you can (reliably) read the write globally. I keep failing to see how this can replace regionally placed database clusters which can serve a continent in milliseconds. Edit: I know it uses streams, but those are only to 5 followers and CF have hundreds of…

As others have noted, you misunderstand how Durable Objects work. All traffic addressed to the same object is routed to a single machine where that object lives. That machine always has a consistent view of its SQLite database. You can have billions of objects, but each has its own separate database. There's no way to read from a database directly from a different machine than the one the DO is running on.

Re: Zero-latency SQLite storage in every Durable Object

#106

Post by CyberDildonics »

simonw wrote: post #94
CyberDildonics wrote: post #90

What is the difference between a "durable object" and a file?

You can read the full article for an answer to that: https://blog.cloudflare.com/sqlite-in-durable-objects/ Short version: it's replicated to five data centers on every transaction, and backed up as a stream to object storage as well.

So it's a file that gets backed up like dropbox

Re: Zero-latency SQLite storage in every Durable Object

#107

Post by simonw »

CyberDildonics wrote: post #106
simonw wrote: post #94

Earlier quoted context omitted.

You can read the full article for an answer to that: https://blog.cloudflare.com/sqlite-in-durable-objects/ Short version: it's replicated to five data centers on every transaction, and backed up as a stream to object storage as well.

So it's a file that gets backed up like dropbox

At a very high level, yes. But the details matter here - you commit a transaction to the SQLite database and know that the commit has been pushed out to 3/5 replicas by the time the write API request returns - and that it will be logged to object storage (supporting 30 days of rollback) within ten seconds. AND it will live in a data center physically close to the user who caused it to be created.

Re: Zero-latency SQLite storage in every Durable Object

#108

Post by parsadotsh »

skrebbel wrote: post #69

I really love the Durable Object design, particularly because it's easy to understand how it works on the inside. Unlike lots of other solutions designed for realtime data stuff, Durable Objects have a simplicity to them, much like Redis and Italian food. You can see all the ingredients. Given enough time and resources (and datacenters :) ), a competent programmer could read the DO docs and reimplement something simi…

I think for those cases you're expected to use something like this: https://developers.cloudflare.com/analytics/analytics-engine...
Post reply on HN