Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-latency SQLite storage in every Durable Object

simonwillison.net

Post reply on HN
Print view View original

11–20 of 108 posts

Re: Zero-latency SQLite storage in every Durable Object

#11

Post by stavros »

gregwebs wrote: post #10
stavros wrote: post #3

This is a really interesting design, but these kinds of smart systems always inhabit an uncanny valley for me. You need them in exactly two cases: 1. You have a really high-load system that you need to figure out some clever ways to scale. 2. You're working on a toy project for fun. If #2, fine, use whatever you want, it's great. If this is production, or for Work(TM), you need something proven. If you don't know you…

There are a lot of cases of low traffic applications that aren’t toys but instead are internal tools- this could be a great option for those. For higher traffic they are asking you to figure out how to shard your data and it’s compute. That’s really hard to do without hitting edge cases.

Why would you use this for an internal, low-traffic tool over Postgres?

Re: Zero-latency SQLite storage in every Durable Object

#12

Post by mhart »

simonw wrote: post #6

One thing I don't understand about Durable Objects yet is where they are physically located. Are they located in the region that hosted the API call that caused them to be created in the first place? If so, is there a mechanism by which a DO can be automatically migrated to another location if it turns out that e.g. they were created in North America but actually all of the subsequent read/write traffic to them comes…

By default in the region you created them in, but you can alternatively specify a locationHint. Use "oc" for Australia. https://developers.cloudflare.com/durable-objects/reference/...

Note the "Dynamic relocation of existing Durable Objects is planned for the future"

Re: Zero-latency SQLite storage in every Durable Object

#13

Post by crabmusket »

simonw wrote: post #6

One thing I don't understand about Durable Objects yet is where they are physically located. Are they located in the region that hosted the API call that caused them to be created in the first place? If so, is there a mechanism by which a DO can be automatically migrated to another location if it turns out that e.g. they were created in North America but actually all of the subsequent read/write traffic to them comes…

Not an answer to your question, but shoutout to https://where.durableobjects.live/

Re: Zero-latency SQLite storage in every Durable Object

#14

Post by fracus »

stavros wrote: post #11
gregwebs wrote: post #10

Earlier quoted context omitted.

There are a lot of cases of low traffic applications that aren’t toys but instead are internal tools- this could be a great option for those. For higher traffic they are asking you to figure out how to shard your data and it’s compute. That’s really hard to do without hitting edge cases.

Why would you use this for an internal, low-traffic tool over Postgres?

Could this be used to get a time edge in trading? I'm not an expert, just thinking out loud. I remember hearing about firms laying wire in a certain way because getting a microsecond jump on changing rates could be everything for them.

Re: Zero-latency SQLite storage in every Durable Object

#15

Post by crabmusket »

stavros wrote: post #3

This is a really interesting design, but these kinds of smart systems always inhabit an uncanny valley for me. You need them in exactly two cases: 1. You have a really high-load system that you need to figure out some clever ways to scale. 2. You're working on a toy project for fun. If #2, fine, use whatever you want, it's great. If this is production, or for Work(TM), you need something proven. If you don't know you…

As far as I can tell, multiplayer is the killer app for Durable Objects. If you want to build another Figma, Google Docs, etc, the programming model of Durable Objects is super handy.

This article goes into it more: https://digest.browsertech.com/archive/browsertech-digest-cl...

I think this old article is quite relevant too: http://ithare.com/scaling-stateful-objects/

Anyone who read the Figma multiplayer article and thought "that's kind of what I need" would be well served by Durable Objects, I think. https://www.figma.com/blog/rust-in-production-at-figma/

There are other approaches - I've worked in the past with CRDTs over WebRTC which felt absolutely space-age. But that's a much more complicated foundation compared to a websocket and a single class instance "somewhere" in the cloud.

Re: Zero-latency SQLite storage in every Durable Object

#16

Post by MuffinFlavored »

stavros wrote: post #3

This is a really interesting design, but these kinds of smart systems always inhabit an uncanny valley for me. You need them in exactly two cases: 1. You have a really high-load system that you need to figure out some clever ways to scale. 2. You're working on a toy project for fun. If #2, fine, use whatever you want, it's great. If this is production, or for Work(TM), you need something proven. If you don't know you…

> If this is production, or for Work(TM), you need something proven.

I feel like part of Cloudflare's business model is to try to convince businesses at scale to solve problems in a non-traditional way using technology they are cooking up, no matter the cost.

Re: Zero-latency SQLite storage in every Durable Object

#17

Post by alright2565 »

stavros wrote: post #11
gregwebs wrote: post #10

Earlier quoted context omitted.

There are a lot of cases of low traffic applications that aren’t toys but instead are internal tools- this could be a great option for those. For higher traffic they are asking you to figure out how to shard your data and it’s compute. That’s really hard to do without hitting edge cases.

Why would you use this for an internal, low-traffic tool over Postgres?

It's so low traffic that you don't want to pay the minimum $35/mo for a PostgreSQL instance on AWS maybe. Or you're required by policy to have a single-tenant architecture, but a full always-on database server would be overkill.

Re: Zero-latency SQLite storage in every Durable Object

#18

Post by stavros »

crabmusket wrote: post #15
stavros wrote: post #3

This is a really interesting design, but these kinds of smart systems always inhabit an uncanny valley for me. You need them in exactly two cases: 1. You have a really high-load system that you need to figure out some clever ways to scale. 2. You're working on a toy project for fun. If #2, fine, use whatever you want, it's great. If this is production, or for Work(TM), you need something proven. If you don't know you…

As far as I can tell, multiplayer is the killer app for Durable Objects. If you want to build another Figma, Google Docs, etc, the programming model of Durable Objects is super handy. This article goes into it more: https://digest.browsertech.com/archive/browsertech-digest-cl... I think this old article is quite relevant too: http://ithare.com/scaling-stateful-objects/ Anyone who read the Figma multiplayer article an…

That's a very interesting use case. Given that your "players" aren't guaranteed to be local to the DO, doesn't using DOs only make sense in high-traffic situations again? Otherwise you might as well just serve the players from a conventional server, no?

CRDTs really do sound amazing, though.

Re: Zero-latency SQLite storage in every Durable Object

#19

Post by simonw »

mhart wrote: post #12
simonw wrote: post #6

One thing I don't understand about Durable Objects yet is where they are physically located. Are they located in the region that hosted the API call that caused them to be created in the first place? If so, is there a mechanism by which a DO can be automatically migrated to another location if it turns out that e.g. they were created in North America but actually all of the subsequent read/write traffic to them comes…

By default in the region you created them in, but you can alternatively specify a locationHint. Use "oc" for Australia. https://developers.cloudflare.com/durable-objects/reference/... Note the "Dynamic relocation of existing Durable Objects is planned for the future"

Thanks, added that to my post.

Re: Zero-latency SQLite storage in every Durable Object

#20

Post by yen223 »

stavros wrote: post #3

This is a really interesting design, but these kinds of smart systems always inhabit an uncanny valley for me. You need them in exactly two cases: 1. You have a really high-load system that you need to figure out some clever ways to scale. 2. You're working on a toy project for fun. If #2, fine, use whatever you want, it's great. If this is production, or for Work(TM), you need something proven. If you don't know you…

I almost have the opposite view:

When starting out you can get away with using a simple Postgres database. Postgres is fine for low-traffic projects with minimal latency constraints, and you probably want to spend your innovation tokens elsewhere.

But in very high-traffic Production cases with tight latency requirements, you will start to see all kinds of weird and wacky traffic patterns, that barebones Postgres won't be able to handle. It's usually in these cases where you'd need to start exploring alternatives to Postgres. It's also in these cases where you can afford to hire people to manage your special database needs.

Post reply on HN