Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A distributed queue in a single JSON file on object storage

turbopuffer.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: A distributed queue in a single JSON file on object storage

#31

Post by thomas_fa »

staticassertion wrote: post #23

Yeah, I mean, I think we're all basically doing this now, right? I wouldn't choose this design, but I think something similar to DeltaLake can be simplified down for tons of use cases. Manifest with CAS + buffered objects to S3, maybe compaction if you intend to do lots of reads. It's not hard to put it together. You can achieve stupidly fast read/write operations if you do this right with a system that is shocking s…

A lot of good insights here. I am also wandering if they can just simply put different jobs (unclaimed, in-progress, deleted/done) into different directory/prefix, and rely on atomic object rename primitive [1][2][3] to solve the problem more gracefully (group commit can still be used if needed).

[1] https://docs.cloud.google.com/storage/docs/samples/storage-m... [2] https://docs.aws.amazon.com/AmazonS3/latest/API/API_RenameOb... [3] https://fractalbits.com/blog/why-we-built-another-object-sto...

Re: A distributed queue in a single JSON file on object storage

#32

Post by soletta »

motoboi wrote: post #26

By typography alone I can now turbopuffer is written in zig.

It is by the juice of Zig that binaries acquire speed, the allocators acquire ownership, the ownership becomes a warning. It is by typography alone I can now turbopuffer is written in zig.

Re: A distributed queue in a single JSON file on object storage

#33

Post by Sirupsen »

candiddevmike wrote: post #14
pjc50 wrote: post #8

Several things going on here: - concurrency is very hard - .. but object storage "solves" most of that for you, handing you a set of semantics which work reliably - single file throughput sucks hilariously badly - .. because 1Gb is ridiculously large for an atomic unit - (this whole thing resembles a project I did a decade ago for transactional consistency on TFAT on Flash, except that somehow managed faster commit t…

AFAIK you can kinda "seek" reads in S3 using a range header, WCGW? =D

Yep! Other than random reads (~p99=200ms on larger ranges), it's essential to get good download performance of a single file. A single (range) request can "only" drive ~500 MB/s, so you need multiple offsets.

https://github.com/sirupsen/napkin-math

Re: A distributed queue in a single JSON file on object storage

#34

Post by staticassertion »

zbentley wrote: post #30
staticassertion wrote: post #23

Yeah, I mean, I think we're all basically doing this now, right? I wouldn't choose this design, but I think something similar to DeltaLake can be simplified down for tons of use cases. Manifest with CAS + buffered objects to S3, maybe compaction if you intend to do lots of reads. It's not hard to put it together. You can achieve stupidly fast read/write operations if you do this right with a system that is shocking s…

> I solved this at one point with token fencing Could you expand on that? Even if it wasn't the approach you stuck with, I'm curious.

Oof, I probably misspoke there just slightly. I attempted to solve this with token fencing, I honestly don't know if it worked under failure conditions. This was also a while ago. But the idea was basically that there were two tiers - one was a ring based approach where a single file determined which writer was allocated a 'space' in the ring. Then every write was prepended with that token. Even if a node dropped/ joined and others didn't know about it (because they hadn't re-read the ring file), every write had this token.

Writes were not visible until compaction in this system. At compaction time, tokens would be checked and writes for older tokens would be rejected, so even if two nodes thought that they owned a 'place' in the ring, only writes for the higher value would be accepted. Soooomething like that. I ended up disliking this because it had undesirable failure modes like lots of stale/ wasted writes, and the code sucked.

Re: A distributed queue in a single JSON file on object storage

#35

Post by Normal_gaussian »

loevborg wrote: post #20
jrjeksjd8d wrote: post #3

Earlier quoted context omitted.

It seems like this is an approach that trades off scale and performance for operational simplicity. They say they only have 1GB of records and they can use a single committer to handle all requests. Failover happens by missing a compare-and-set so there's probably a second of latency to become leader? This is not to say it's a bad system, but it's very precisely tailored for their needs. If you look at the original K…

> Failover happens by missing a compare-and-set so there's probably a second of latency to become leader? Conceptually that makes sense. How complicated is it to implement this failover logic in a safe way? If there are two processes, competing for CAS wins, is there not a risk that both will think they're non-leaders and terminate themselves?

The broker lifecycle is presumably

1. Start

2. Load the queue.json from the object store

3. Receive request(s)

3. Edit in memory JSON with batch data

4. Save data with CAS

5. On failure not due to CAS, recover (or fail)

6. On success, succeed requests and go to 3

7. On failure due to CAS, fail active requests and terminate

The client should have a retry mechanism against the broker (which may include looking up the address again).

From the brokers PoV, it will never fail a CAS until another broker wins a CAS, at which point that other broker is the leader. If it does fail a CAS the client will retry with another broker, which will probably be the leader. The key insight is that the broker reads the file once, it doesn't compete to become leader by re-reading the data and this is OK because of the nature of the data. You could also say that brokers are set up to consider themselves "maybe the leader" until they find out they are not, and losing leadership doesn't lose data.

The mechanism to start brokers is only vaguely discussed, but if a host-unreachable also triggers a new broker there is a neat from-zero scaling property.

Re: A distributed queue in a single JSON file on object storage

#36

Post by snowhale »

soletta wrote: post #2

The usual path an engineer takes is to take a complex and slow system and reengineer it into something simple, fast, and wrong. But as far as I can tell from the description in the blog though, it actually works at scale! This feels like a free lunch and I’m wondering what the tradeoff is.

[dead]

Re: A distributed queue in a single JSON file on object storage

#39

Post by allknowingfrog »

staticassertion wrote: post #23

Yeah, I mean, I think we're all basically doing this now, right? I wouldn't choose this design, but I think something similar to DeltaLake can be simplified down for tons of use cases. Manifest with CAS + buffered objects to S3, maybe compaction if you intend to do lots of reads. It's not hard to put it together. You can achieve stupidly fast read/write operations if you do this right with a system that is shocking s…

This is news to me. What motivates you to reach for an S3-backed queue versus SQS?

Re: A distributed queue in a single JSON file on object storage

#40

Post by staticassertion »

allknowingfrog wrote: post #39
staticassertion wrote: post #23

Yeah, I mean, I think we're all basically doing this now, right? I wouldn't choose this design, but I think something similar to DeltaLake can be simplified down for tons of use cases. Manifest with CAS + buffered objects to S3, maybe compaction if you intend to do lots of reads. It's not hard to put it together. You can achieve stupidly fast read/write operations if you do this right with a system that is shocking s…

This is news to me. What motivates you to reach for an S3-backed queue versus SQS?

I'm not building a queue, but a lot of things on s3 end up being queue-shaped (more like 'log shaped') because it's very easy to compose many powerful systems out of CAS + "buffer, then push". Basically, you start with "build an immutable log" with those operations and the rest of your system becomes a matter of what you do with that log. A queue needs to support a "pop", but I am supporting other operations. Still, the architecture overlap all begins with CAS + buffer.

That said, I suspect that you can probably beat SQS for a number of use cases, and definitely if you want to hold onto the data long term or search over it then S3 has huge options there.

Performance will be extremely solid unless you need your worst case latency for "push -> pop" to be very tight in your p90.

Post reply on HN