Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-latency SQLite storage in every Durable Object

simonwillison.net

Post reply on HN
Print view View original

31–40 of 108 posts

Re: Zero-latency SQLite storage in every Durable Object

#31

Post by crabmusket »

fracus wrote: post #14
stavros wrote: post #11

Earlier quoted context omitted.

Why would you use this for an internal, low-traffic tool over Postgres?

Could this be used to get a time edge in trading? I'm not an expert, just thinking out loud. I remember hearing about firms laying wire in a certain way because getting a microsecond jump on changing rates could be everything for them.

I'm also no expert, but from reading around the subject a little (Flash Boys by Michael Lewis was pretty cool, also Jane Street's podcast has some fantastic information)... no. I doubt you'd be on a public cloud if low-latency trading is what you're doing.

Re: Zero-latency SQLite storage in every Durable Object

#32

Post by crabmusket »

emadda wrote: post #21

Some other interesting points: - The write api is sync, but it has a hidden async await: when you do your next output with a response, if the write fails the runtime will replace the response with a http failure. This allows the runtime to auto-batch writes and optimistically assume they will succeed, without the user explicitly handling the errors or awaits. - There are no read transactions, which would be useful to…

The RPC stuff is pretty interesting. More here: https://blog.cloudflare.com/javascript-native-rpc/

Re: Zero-latency SQLite storage in every Durable Object

#33

Post by paulgb »

crabmusket wrote: post #27
stavros wrote: post #18

Earlier quoted context omitted.

That's a very interesting use case. Given that your "players" aren't guaranteed to be local to the DO, doesn't using DOs only make sense in high-traffic situations again? Otherwise you might as well just serve the players from a conventional server, no? CRDTs really do sound amazing, though.

Best case, the players are co-located in a city or country, and they'll benefit from data center locality. Worst case, they're not co-located, and one participant has good latency, and the other doesn't. This is equivalent to the "deploy the backend in a single server/datacenter" approach. Aside from the data locality, I still find the programming model (a globally-unique and addressable single-threaded class instanc…

> Aside from the data locality, I still find the programming model (a globally-unique and addressable single-threaded class instance) to be quite nice, and would want to emulate it even without the Cloudflare edge magic.

You might be interested in Plane (https://plane.dev/ / https://github.com/jamsocket/plane), which we sometimes describe as a sort of Durable Object-like abstraction that can run anywhere containers can.

(I'm also one of the articles you linked, thanks for the shoutout!)

Re: Zero-latency SQLite storage in every Durable Object

#34

Post by yen223 »

simonw wrote: post #23
yen223 wrote: post #20

Earlier quoted context omitted.

I almost have the opposite view: When starting out you can get away with using a simple Postgres database. Postgres is fine for low-traffic projects with minimal latency constraints, and you probably want to spend your innovation tokens elsewhere. But in very high-traffic Production cases with tight latency requirements, you will start to see all kinds of weird and wacky traffic patterns, that barebones Postgres won'…

Have you worked on any examples of projects that started on PostgreSQL and ended up needing to migrate to something specialized?

I did, twice.

The second time, we had a reporting system that eventually stored billions of rows per day in a Postgres database. Processing times got so bad that we decided to migrate to Clickhouse, resulting in a substantial boost to query times. I maintain that we haven't exhausted all available optimisations for Postgres, but I cannot deny that the migration made sense in the long run - OLTP vs OLAP and all that.

(The first time is a funny story that I'm not quite ready to share.)

Re: Zero-latency SQLite storage in every Durable Object

#35

Post by simonw »

yen223 wrote: post #34
simonw wrote: post #23

Earlier quoted context omitted.

Have you worked on any examples of projects that started on PostgreSQL and ended up needing to migrate to something specialized?

I did, twice. The second time, we had a reporting system that eventually stored billions of rows per day in a Postgres database. Processing times got so bad that we decided to migrate to Clickhouse, resulting in a substantial boost to query times. I maintain that we haven't exhausted all available optimisations for Postgres, but I cannot deny that the migration made sense in the long run - OLTP vs OLAP and all that.…

That makes a lot of sense to me. One of my strongest hints that a non-relational data store might be a good idea is "grows by billions of rows a day".

Re: Zero-latency SQLite storage in every Durable Object

#36

Post by braden-lk »

Durable objects seem so cool but the pricing always scares me. (Specifically, having to worry about getting hibernation right.) They’d be a great fit for our yjs document based strategy, but while everything in prod still works on plain ol redis and Postgres, it’s hard to justify an exploration.

Re: Zero-latency SQLite storage in every Durable Object

#37

Post by crabmusket »

paulgb wrote: post #33
crabmusket wrote: post #27

Earlier quoted context omitted.

Best case, the players are co-located in a city or country, and they'll benefit from data center locality. Worst case, they're not co-located, and one participant has good latency, and the other doesn't. This is equivalent to the "deploy the backend in a single server/datacenter" approach. Aside from the data locality, I still find the programming model (a globally-unique and addressable single-threaded class instanc…

> Aside from the data locality, I still find the programming model (a globally-unique and addressable single-threaded class instance) to be quite nice, and would want to emulate it even without the Cloudflare edge magic. You might be interested in Plane ( https://plane.dev/ / https://github.com/jamsocket/plane ), which we sometimes describe as a sort of Durable Object-like abstraction that can run anywhere containers…

I am interested, and I really enjoy your work on Browsertech! I haven't needed Plane above/over what Cloudflare is providing, but I've got it in the back of my mind as an option.

I've long hoped other providers might jump on the Durable Objects bandwagon and provide competing functionality so we're not locked in. Plane/Jamsocket looks like one way to go about mitigating that risk to a certain extent.

Re: Zero-latency SQLite storage in every Durable Object

#38

Post by adhamsalama »

simonw wrote: post #35
yen223 wrote: post #34

Earlier quoted context omitted.

I did, twice. The second time, we had a reporting system that eventually stored billions of rows per day in a Postgres database. Processing times got so bad that we decided to migrate to Clickhouse, resulting in a substantial boost to query times. I maintain that we haven't exhausted all available optimisations for Postgres, but I cannot deny that the migration made sense in the long run - OLTP vs OLAP and all that.…

That makes a lot of sense to me. One of my strongest hints that a non-relational data store might be a good idea is "grows by billions of rows a day".

Isn't Clickhouse relational?

Re: Zero-latency SQLite storage in every Durable Object

#39

Post by ignoramous »

braden-lk wrote: post #36

Durable objects seem so cool but the pricing always scares me. (Specifically, having to worry about getting hibernation right.) They’d be a great fit for our yjs document based strategy, but while everything in prod still works on plain ol redis and Postgres, it’s hard to justify an exploration.

> Specifically, having to worry about getting hibernation right.

As long as the client doesn't exchange websocket messages with DO, it'll hibernate. From what I can tell, ping/pong frames don't count towards uptime, if you're worried about that.

Re: Zero-latency SQLite storage in every Durable Object

#40

Post by adhamsalama »

yen223 wrote: post #34
simonw wrote: post #23

Earlier quoted context omitted.

Have you worked on any examples of projects that started on PostgreSQL and ended up needing to migrate to something specialized?

I did, twice. The second time, we had a reporting system that eventually stored billions of rows per day in a Postgres database. Processing times got so bad that we decided to migrate to Clickhouse, resulting in a substantial boost to query times. I maintain that we haven't exhausted all available optimisations for Postgres, but I cannot deny that the migration made sense in the long run - OLTP vs OLAP and all that.…

Well, this isn't specific to Postgres, is it?

If you were storing billions of rows per day in MySQL, SQL Server, or Oracle, it still wouldn't be able to handle it, would it?

Post reply on HN