141–150 of 166 posts
So...are we closer to getting to use S3 as a...you guessed it...a database? With CAS, we are probably able to get a basic level of atomicity, and S3 itself is pretty durable, now we have to deal with consistency and isolation...although S3 branded itself as "eventually consistent"...
I had no idea people rely on S3 beyond dumb storage. It almost feels like people are trying to build out a distributed OLAP database in the reverse direction.
2. glue jobs to partition by some columns reporting uses
3. query with athena
4. ???
5. profit (celebrate reduced cost)
This thing costs couple $ a month for ~500gb of data. Snowflake wanted crazy amounts of money for the same thing.
Earlier quoted context omitted.
EC2 bills by the second.
Some... "Your Amazon EC2 usage is calculated by either the hour or the second based on the size of the instance, operating system, and the AWS Region where the instances are launched" - https://repost.aws/knowledge-center/ec2-instance-hour-billin... https://aws.amazon.com/ec2/pricing/on-demand/
Earlier quoted context omitted.
Some... "Your Amazon EC2 usage is calculated by either the hour or the second based on the size of the instance, operating system, and the AWS Region where the instances are launched" - https://repost.aws/knowledge-center/ec2-instance-hour-billin... https://aws.amazon.com/ec2/pricing/on-demand/
MacOS instances appear to be the sole remaining exception since RHEL got on board.
Earlier quoted context omitted.
S3 has supported SHA-256 as a checksum algo since 2022. You can calculate the hash locally and then specify that hash in the PutObject call. S3 will calculate the hash and compare it with the hash in the PutObject call and reject the Put if they differ. The hash and algo are then stored in the object's metadata. You simply also use the SHA-256 hash as the key for the object. https://aws.amazon.com/blogs/aws/new-addit…
Unfortunately, for a multi-part upload it isn't a hash of the total object, it is a hash of the hashes for each part, which is a lot less useful. Especially if you don't know how the file was partititioned during upload. And even if it was for the whole file, it isn't used for the ETag, so, so it can't be used for conditional PUTs. I had a use case where this looked really promising, then I ran into the multipart upl…
Edit: This is actually already implemented in the Bao project which exploits the structure of the BLAKE3 merkle tree structure to offer cool features like streaming verification and verifying slices of a file as I described above: https://github.com/oconnor663/bao#verifying-slices
[rejected] error: failed to push some refs to remote repository Finally we can have this with s3 :)
An open-source implementation of Amazon S3 - MinIO has had it for almost two years (relevant post: https://blog.min.io/leading-the-way-minios-conditional-write... ). Strangely, Amazon is catching up just now.
I wouldn't be surprised if they saw over 100mil/req/sec globally by now. That's 100 million requests a second that need strong read-your-write consistency and atomicity at global scale. The number of pieces they had to move into place for this to happen is probably quite the engineering tale.
[1] https://aws.amazon.com/blogs/aws/amazon-s3-two-trillion-obje...
Earlier quoted context omitted.
Agreed. Scaling based on the length of the queue, up to some maximum.
Even better, based on queue latency instead of length
Scaling on things like the length of the queue doesn't work very well at all in practice. A queue length of 100 might be horribly long in some workloads and insignificant in others, so scaling on queue length requires a lot of tuning that must be adjusted over time as the workload changes. Scaling based on percent of concurrent capacity can work for most workloads, and tends to remain stable over time even as workloads change.