Zero-latency SQLite storage in every Durable Object
simonwillison.net
Zero-latency SQLite storage in every Durable Object
1–10 of 108 posts
Re: Zero-latency SQLite storage in every Durable Object
#2Does this mean SQLite for DO can lose up to 10 seconds of data in the event of a failing DO?
Re: Zero-latency SQLite storage in every Durable Object
#3This is a really interesting design, but these kinds of smart systems always inhabit an uncanny valley for me. You need them in exactly two cases:
1. You have a really high-load system that you need to figure out some clever ways to scale.
2. You're working on a toy project for fun.
If #2, fine, use whatever you want, it's great.
If this is production, or for Work(TM), you need something proven. If you don't know you need this, you don't need it, go with a boring Postgres database and a VM or something.
If you do know you need this, then you're kind of in a bind: It's not really very mature yet, as it's pretty new, and you're probably going to hit a bunch of weird edge cases, which you probably don't really want to have to debug or live with.
So, who are these systems for, in the end? They're so niche that they can't easily mature and be used by lots of serious players, and they're too complex with too many tradeoffs to be used by 99.9% of companies.
The only people I know for sure are the target market for this sort of thing is the developers who see something shiny, build a company (or, worse, build someone else's company) on it, and then regret it pretty soon and move to something else (hopefully much more boring).
Does anyone have more insight on this? I'd love to know.
Re: Zero-latency SQLite storage in every Durable Object
#4Does this mean SQLite for DO can lose up to 10 seconds of data in the event of a failing DO?
> To ensure durability beyond that ten second window, writes are also forwarded to five replicas in separate nearby data centers as soon as they commit, and the write is only acknowledged once three of them have confirmed it.
I think Simon meant "within", rather than "beyond", here.
Re: Zero-latency SQLite storage in every Durable Object
#5Does this mean SQLite for DO can lose up to 10 seconds of data in the event of a failing DO?
> To ensure durability beyond that ten second window, writes are also forwarded to five replicas in separate nearby data centers as soon as they commit, and the write is only acknowledged once three of them have confirmed it. I think Simon meant "within", rather than "beyond", here.
Thanks, I've updated that word.
Re: Zero-latency SQLite storage in every Durable Object
#6One thing I don't understand about Durable Objects yet is where they are physically located.
Are they located in the region that hosted the API call that caused them to be created in the first place?
If so, is there a mechanism by which a DO can be automatically migrated to another location if it turns out that e.g. they were created in North America but actually all of the subsequent read/write traffic to them comes from Australia?
Re: Zero-latency SQLite storage in every Durable Object
#7This is a really interesting design, but these kinds of smart systems always inhabit an uncanny valley for me. You need them in exactly two cases: 1. You have a really high-load system that you need to figure out some clever ways to scale. 2. You're working on a toy project for fun. If #2, fine, use whatever you want, it's great. If this is production, or for Work(TM), you need something proven. If you don't know you…
If you're in #1, you talk to CloudFlare. They need some great customer stories and they have some great engineers that are most likely willing to work with you on how this will work/help you with bugs in exchange for some success stories. If it gets proven out this turns into a service relationship, but early on it's a partnership.
Re: Zero-latency SQLite storage in every Durable Object
#8One thing I don't understand about Durable Objects yet is where they are physically located. Are they located in the region that hosted the API call that caused them to be created in the first place? If so, is there a mechanism by which a DO can be automatically migrated to another location if it turns out that e.g. they were created in North America but actually all of the subsequent read/write traffic to them comes…
Durable Objects have long term storage. They get hydrated from that storage, so in that sense, they can move to any Cloudflare DS. However, there is no API call to move a Durable Object. It has to have no connections and then gets recreated in the DS nearest to the next/first connection. Memory gets dropped when that happens, storage survives. (This is slightly out of date as they have some nuanced hibernation stuff that is recent).
Re: Zero-latency SQLite storage in every Durable Object
#9One thing I don't understand about Durable Objects yet is where they are physically located. Are they located in the region that hosted the API call that caused them to be created in the first place? If so, is there a mechanism by which a DO can be automatically migrated to another location if it turns out that e.g. they were created in North America but actually all of the subsequent read/write traffic to them comes…
> Durable Objects do not currently change locations after they are created
> Dynamic relocation of existing Durable Objects is planned for the future.
https://developers.cloudflare.com/durable-objects/reference/....
IIRC Orleans (https://www.microsoft.com/en-us/research/wp-content/uploads/...) allows actors to be moved between machines, which should map well to DOs being moved between locations.
Re: Zero-latency SQLite storage in every Durable Object
#10This is a really interesting design, but these kinds of smart systems always inhabit an uncanny valley for me. You need them in exactly two cases: 1. You have a really high-load system that you need to figure out some clever ways to scale. 2. You're working on a toy project for fun. If #2, fine, use whatever you want, it's great. If this is production, or for Work(TM), you need something proven. If you don't know you…
There are a lot of cases of low traffic applications that aren’t toys but instead are internal tools- this could be a great option for those.
For higher traffic they are asking you to figure out how to shard your data and it’s compute. That’s really hard to do without hitting edge cases.