Live data from Hacker News

Amazon S3 now supports the ability to append data to an object

aws.amazon.com

21–30 of 70 posts

Re: Amazon S3 now supports the ability to append data to an object

#21
post #15

Wrote some notes on this here: https://simonwillison.net/2024/Nov/22/amazon-s3-append-data/ Key points: - It's just for the "S3 Express One Zone" bucket class, which is more expensive (16c/GB/month compared to 2.3c for S3 standard tier) and less highly available, since it lives in just one availability zone - "With each successful append operation, you create a part of the object and each object can have up to 10,000…

This will require some serious buffering.

Re: Amazon S3 now supports the ability to append data to an object

#23
I am surprised it was not supported until now? How does it compare to azure blob append (which exists for years)?

I have been using azure storage append blob to store logs of long running tasks with periodic flush (see https://learn.microsoft.com/en-us/rest/api/storageservices/u...)

Re: Amazon S3 now supports the ability to append data to an object

#24
post #19
post #15

Wrote some notes on this here: https://simonwillison.net/2024/Nov/22/amazon-s3-append-data/ Key points: - It's just for the "S3 Express One Zone" bucket class, which is more expensive (16c/GB/month compared to 2.3c for S3 standard tier) and less highly available, since it lives in just one availability zone - "With each successful append operation, you create a part of the object and each object can have up to 10,000…

Not directly, but enough to write once every hour for more than a year!

Yeah, or I guess log rotation will work well - you can write 10,000 lines to one key and then switch to a new key name.

Re: Amazon S3 now supports the ability to append data to an object

#25
post #23

I am surprised it was not supported until now? How does it compare to azure blob append (which exists for years)? I have been using azure storage append blob to store logs of long running tasks with periodic flush (see https://learn.microsoft.com/en-us/rest/api/storageservices/u... )

I know the whole point of cloud services is to pick and choose, but in general I wouldn’t express “outrage” or scoff when comparing Azure to AWS. I recommend Azure to the smallest and leanest of shops, but when you compare functionality matrices and maturity Azure is a children’s toy.

To compare the other way, Azure write blocks target replication blob containers. I consider that a primitive and yet they just outright say you can’t do it. When I engaged our TPM on this we were just told our expectations were wrong and we were thinking about the problem wrong.

Re: Amazon S3 now supports the ability to append data to an object

#28
post #25
post #23

I am surprised it was not supported until now? How does it compare to azure blob append (which exists for years)? I have been using azure storage append blob to store logs of long running tasks with periodic flush (see https://learn.microsoft.com/en-us/rest/api/storageservices/u... )

I know the whole point of cloud services is to pick and choose, but in general I wouldn’t express “outrage” or scoff when comparing Azure to AWS. I recommend Azure to the smallest and leanest of shops, but when you compare functionality matrices and maturity Azure is a children’s toy. To compare the other way, Azure write blocks target replication blob containers. I consider that a primitive and yet they just outrigh…

I did not want to express any outrage (even sarcastically), just surprise and the fact that I don't know very well the AWS offer.

> Azure write blocks target replication blob containers

I am sorry but what does it mean?

The goal of my question was about what are the differences between the two solutions: I know HN is a place where I can read technical arguments based on actual experience.

Re: Amazon S3 now supports the ability to append data to an object

#29
post #20

It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.

doesn't s3 have 'free' (subsidized!) transfer to other products like ec2 though? it might look better to businesspeople that "well, we're doing this processing on ec2, why not keep our objects in s3 since it's a bit cheaper!"

Re: Amazon S3 now supports the ability to append data to an object

#30
post #4

I wonder what the implications for all the s3-like APIs is going to be.

Good point - plus the conditional writes recently announced. Did anyone else implement that?

MinIO actually supported conditional writes a year before S3
Post reply on HN