that's A choice.
A distributed queue in a single JSON file on object storage
11–20 of 65 posts
Re: A distributed queue in a single JSON file on object storage
#12Depending on who hosts your object storage this seems like it could get much more expensive than using a queue table in your database? But I'm also aware that this is a blog post of an object storage company.
(cofounder of tpuf here)
We don't have a relational database, otherwise that would work great for a queue! You can imagine us continuing to iterate here to Step 5, Step 6, ... Step N over time. The tradeoff of each step is complexity, and complexity has to be deserved. This is working exceptionally well currently.
Re: A distributed queue in a single JSON file on object storage
#13Depending on who hosts your object storage this seems like it could get much more expensive than using a queue table in your database? But I'm also aware that this is a blog post of an object storage company.
(cofounder of tpuf here) We don't have a relational database, otherwise that would work great for a queue! You can imagine us continuing to iterate here to Step 5, Step 6, ... Step N over time. The tradeoff of each step is complexity, and complexity has to be deserved. This is working exceptionally well currently.
Makes total sense for your use case! I have got bitten by using object storage as a database before (and churning through "update" ops) so this will depend on the pricing (and busy-ness of the queue of course) of the provider anyway. Using whatever you have available instead of introducing complexity is the way. Sqlite / Postgres goes a long way for use cases you wouldn't originally think would go well with a relational database too (full text search, using as queue,...).
Re: A distributed queue in a single JSON file on object storage
#14Several things going on here: - concurrency is very hard - .. but object storage "solves" most of that for you, handing you a set of semantics which work reliably - single file throughput sucks hilariously badly - .. because 1Gb is ridiculously large for an atomic unit - (this whole thing resembles a project I did a decade ago for transactional consistency on TFAT on Flash, except that somehow managed faster commit t…
AFAIK you can kinda "seek" reads in S3 using a range header, WCGW? =D
Re: A distributed queue in a single JSON file on object storage
#15Earlier quoted context omitted.
(cofounder of tpuf here) We don't have a relational database, otherwise that would work great for a queue! You can imagine us continuing to iterate here to Step 5, Step 6, ... Step N over time. The tradeoff of each step is complexity, and complexity has to be deserved. This is working exceptionally well currently.
Makes total sense for your use case! I have got bitten by using object storage as a database before (and churning through "update" ops) so this will depend on the pricing (and busy-ness of the queue of course) of the provider anyway. Using whatever you have available instead of introducing complexity is the way. Sqlite / Postgres goes a long way for use cases you wouldn't originally think would go well with a relatio…
Due to the batching, this will only consume a few million class B per month. They are $5/million
Re: A distributed queue in a single JSON file on object storage
#16[flagged]
Re: A distributed queue in a single JSON file on object storage
#17Several things going on here: - concurrency is very hard - .. but object storage "solves" most of that for you, handing you a set of semantics which work reliably - single file throughput sucks hilariously badly - .. because 1Gb is ridiculously large for an atomic unit - (this whole thing resembles a project I did a decade ago for transactional consistency on TFAT on Flash, except that somehow managed faster commit t…
> The broker runs a single group commit loop on behalf of all clients, so no one contends for the object. Critically, it doesn't acknowledge a write until the group commit has landed in object storage. No client moves on until its data is durably committed.
Re: A distributed queue in a single JSON file on object storage
#18Is this reinventing a few redis features with an object storage for persistence?
it’s got some more 9s of durability compared to redis (redis did not invent “queue”)
Re: A distributed queue in a single JSON file on object storage
#19Love this writeup. There's so much interesting stuff you can build on top of Object Storage + compare-and-swap. You learn a lot about distributed systems this way.
I'd love to see a full sample implementation based on s3 + ecs - just to study how it works.
Re: A distributed queue in a single JSON file on object storage
#20The usual path an engineer takes is to take a complex and slow system and reengineer it into something simple, fast, and wrong. But as far as I can tell from the description in the blog though, it actually works at scale! This feels like a free lunch and I’m wondering what the tradeoff is.
It seems like this is an approach that trades off scale and performance for operational simplicity. They say they only have 1GB of records and they can use a single committer to handle all requests. Failover happens by missing a compare-and-set so there's probably a second of latency to become leader? This is not to say it's a bad system, but it's very precisely tailored for their needs. If you look at the original K…
> Failover happens by missing a compare-and-set so there's probably a second of latency to become leader?
Conceptually that makes sense. How complicated is it to implement this failover logic in a safe way? If there are two processes, competing for CAS wins, is there not a risk that both will think they're non-leaders and terminate themselves?