Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A distributed queue in a single JSON file on object storage

turbopuffer.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: A distributed queue in a single JSON file on object storage

#41

Post by UltraSane »

staticassertion wrote: post #27
UltraSane wrote: post #25

Earlier quoted context omitted.

Amazon S3 Select enables SQL queries directly on CSV, JSON, or Apache Parquet objects, allowing retrieval of filtered data subsets to reduce latency and costs

S3 Select is, very sadly, deprecated. It also supported HTTP RANGE headers! But they've killed it and I'll never forgive them :) Still, it's nbd. You can cache a billion Parquet header/footers on disk/ memory and get 90% of the performance (or better tbh).

Wow I didn't know that. To be fair now that S3 tables exists it is rather redundant.

Re: A distributed queue in a single JSON file on object storage

#43

Post by danhhz »

jrjeksjd8d wrote: post #3
soletta wrote: post #2

The usual path an engineer takes is to take a complex and slow system and reengineer it into something simple, fast, and wrong. But as far as I can tell from the description in the blog though, it actually works at scale! This feels like a free lunch and I’m wondering what the tradeoff is.

It seems like this is an approach that trades off scale and performance for operational simplicity. They say they only have 1GB of records and they can use a single committer to handle all requests. Failover happens by missing a compare-and-set so there's probably a second of latency to become leader? This is not to say it's a bad system, but it's very precisely tailored for their needs. If you look at the original K…

(author here)

> It seems like this is an approach that trades off scale and performance for operational simplicity.

Yes, this is exactly it. Given that turbopuffer itself is built on the idea of object storage + stateless cache, we're all very comfortable dealing with it operationally. This design is enough for our needs and is much easier to be oncall for than adding an entirely new dependency would have been.

Re: A distributed queue in a single JSON file on object storage

#44

Post by staticassertion »

talentedtumor wrote: post #37

Does this suffer from ABA problem, or does object storage solve that for you by e.g. refusing to accept writes where content has changed between the read and write?

> refusing to accept writes where content has changed between the read and write?

Right. You can issue a write that will only be accepted if a condition is matched, like the etag of the object matching your expectation. If it doesn't match, your object was invalidated.

Re: A distributed queue in a single JSON file on object storage

#45

Post by packetlost »

danhhz wrote: post #43
jrjeksjd8d wrote: post #3

Earlier quoted context omitted.

It seems like this is an approach that trades off scale and performance for operational simplicity. They say they only have 1GB of records and they can use a single committer to handle all requests. Failover happens by missing a compare-and-set so there's probably a second of latency to become leader? This is not to say it's a bad system, but it's very precisely tailored for their needs. If you look at the original K…

(author here) > It seems like this is an approach that trades off scale and performance for operational simplicity. Yes, this is exactly it. Given that turbopuffer itself is built on the idea of object storage + stateless cache, we're all very comfortable dealing with it operationally. This design is enough for our needs and is much easier to be oncall for than adding an entirely new dependency would have been.

IMO this is the ideal way to engineer most (not all) systems. As simple as your needs allow. Nice work!

Re: A distributed queue in a single JSON file on object storage

#46

Post by loevborg »

staticassertion wrote: post #40
allknowingfrog wrote: post #39

Earlier quoted context omitted.

This is news to me. What motivates you to reach for an S3-backed queue versus SQS?

I'm not building a queue, but a lot of things on s3 end up being queue-shaped (more like 'log shaped') because it's very easy to compose many powerful systems out of CAS + "buffer, then push". Basically, you start with "build an immutable log" with those operations and the rest of your system becomes a matter of what you do with that log. A queue needs to support a "pop", but I am supporting other operations. Still,…

This is fascinating. It sounds like you're building "cloud datastructures" based on S3+CAS. What are the benefits, in your view, of doing using S3 instead of, say, dynamo or postgres? Or reaching for NATS/rabbitmq/sqs/kafka. I'd love to hear a bit more about what you're building.

Re: A distributed queue in a single JSON file on object storage

#48

Post by tomnicholas1 »

staticassertion wrote: post #23

Yeah, I mean, I think we're all basically doing this now, right? I wouldn't choose this design, but I think something similar to DeltaLake can be simplified down for tons of use cases. Manifest with CAS + buffered objects to S3, maybe compaction if you intend to do lots of reads. It's not hard to put it together. You can achieve stupidly fast read/write operations if you do this right with a system that is shocking s…

What you describe is very similar to how Icechunk[1] works. It works beautifully for transactional writes to "repos" containing PBs of scientific array data in object storage.

[1]: https://icechunk.io/en/latest/

Re: A distributed queue in a single JSON file on object storage

#49

Post by staticassertion »

loevborg wrote: post #46
staticassertion wrote: post #40

Earlier quoted context omitted.

I'm not building a queue, but a lot of things on s3 end up being queue-shaped (more like 'log shaped') because it's very easy to compose many powerful systems out of CAS + "buffer, then push". Basically, you start with "build an immutable log" with those operations and the rest of your system becomes a matter of what you do with that log. A queue needs to support a "pop", but I am supporting other operations. Still,…

This is fascinating. It sounds like you're building "cloud datastructures" based on S3+CAS. What are the benefits, in your view, of doing using S3 instead of, say, dynamo or postgres? Or reaching for NATS/rabbitmq/sqs/kafka. I'd love to hear a bit more about what you're building.

It's just trade-offs. If you have a lot of data, s3 is just the only option for storing it. You don't want to pay for petabytes of storage in Dynamo or Postgres. I also don't want to manage postgres, even RDS - dealing with write loads that S3 handles easily is very annoying, dealing with availability, etc, all is painful. S3 "just works" but you need to build some of the protocol yourself.

If you want consistently really low latency/ can't tolerate a 50ms spike, don't retain tons of data, have Insane storage also unlocks new capabilities. Immutable logs unlock "time travel" where you can ask questions like "what did the system look like at this point?" since no information is lost (unless you want to lose it, up to you).

Everything about a system like this comes down to reducing the cost of a GET. Bloom filters are your best friend, metadata is your best friend, prefetching is a reluctant friend, etc.

I'm not sure what I'm building. I had this idea years ago before S3 CAS was a thing and I was building a graph database on S3 with the fundamental primitive being an immutable event log (at the time using CRDTs for merge semantics, but I've abandoned that for now) and then maintaining an external index in Scylla with S3 Select for projections. Years later, I have fun poking at it sometimes and redesigning it. S3 CAS unlocked a lot of ways to completely move the system to S3.

Post reply on HN