51–60 of 65 posts
Yeah, I mean, I think we're all basically doing this now, right? I wouldn't choose this design, but I think something similar to DeltaLake can be simplified down for tons of use cases. Manifest with CAS + buffered objects to S3, maybe compaction if you intend to do lots of reads. It's not hard to put it together. You can achieve stupidly fast read/write operations if you do this right with a system that is shocking s…
What you describe is very similar to how Icechunk[1] works. It works beautifully for transactional writes to "repos" containing PBs of scientific array data in object storage. [1]: https://icechunk.io/en/latest/
Among other problems I knew the next thing we were going to have to do was autoscaling and the system we had for call and response was a mess from that respect. Unanswered questions were: How do you know when all agents have succeeded, how do you avoid overwriting your peers’ data, and what do you do with agents that existed yesterday and don’t today?
I ended up rewriting all of the state management data so that each field had one writer and one or more readers. It also allowed me to move the last live service call for another service and decommission it. Instead of having a admin service you just called one of the peers at random and elected it leader for the duration of that operation. I also arranged the data so the leader could watch the parent key for the roll call and avoid needing to poll.
Each time a task was created the leader would do a service discovery call to get a headcount and then wait for everyone to set a suggests or failure state. Some of these state transitions were idempotent, so if you reissued a task you didn’t need to delete the old results. Everyone who already completed it would noop, and the ones that failed or the new servers that joined the cluster would finish up. If there was a delete operation later then the data would be purged from the data set and the agents, a subsequent call would be considered new.
Long story short, your CS program should have distributed computing classes because this shit is hard to work out from first principles when you don’t know what the principles even are.
[flagged]
For instance backing off 53 ms is a lot of latency. Backing off 1 μs may still result in a collision, requiring two or three backoffs before it resolves, where a longer pause on the first backoff might have resulted in quicker recovery.
Optimistic locking is a statistical model just like bloom filters are, and scaling something 20 or 100 times higher is a hell of a lot of runway. Especially if the failure mode is a person sharing someone else’s account or using a laptop and tablet at exactly the same time.
Several things going on here: - concurrency is very hard - .. but object storage "solves" most of that for you, handing you a set of semantics which work reliably - single file throughput sucks hilariously badly - .. because 1Gb is ridiculously large for an atomic unit - (this whole thing resembles a project I did a decade ago for transactional consistency on TFAT on Flash, except that somehow managed faster commit t…
> The broker runs a single group commit loop on behalf of all clients, so no one contends for the object. Critically, it doesn't acknowledge a write until the group commit has landed in object storage. No client moves on until its data is durably committed.
I read this blog post and to help wrap my head around it I put together a simple TCP-based KV store with group commit, helped make it click for me.
Yeah, I mean, I think we're all basically doing this now, right? I wouldn't choose this design, but I think something similar to DeltaLake can be simplified down for tons of use cases. Manifest with CAS + buffered objects to S3, maybe compaction if you intend to do lots of reads. It's not hard to put it together. You can achieve stupidly fast read/write operations if you do this right with a system that is shocking s…
A lot of good insights here. I am also wandering if they can just simply put different jobs (unclaimed, in-progress, deleted/done) into different directory/prefix, and rely on atomic object rename primitive [1][2][3] to solve the problem more gracefully (group commit can still be used if needed). [1] https://docs.cloud.google.com/storage/docs/samples/storage-m... [2] https://docs.aws.amazon.com/AmazonS3/latest/API/AP…
> RenameObject is only supported for objects stored in the S3 Express One Zone storage class.
Ah interesting, I don't use this but I bet in a year+ AWS will have this everywhere lol S3 is just too good.
Just slap some garbage on it and it is better
Earlier quoted context omitted.
Amazon S3 Select enables SQL queries directly on CSV, JSON, or Apache Parquet objects, allowing retrieval of filtered data subsets to reduce latency and costs
S3 Select is, very sadly, deprecated. It also supported HTTP RANGE headers! But they've killed it and I'll never forgive them :) Still, it's nbd. You can cache a billion Parquet header/footers on disk/ memory and get 90% of the performance (or better tbh).