I'm curious on the different use cases for this? Firehose/kinesis whatever the name seems to have the append case covered in ways that I would think has fewer foot guns?
Amazon S3 now supports the ability to append data to an object
51–60 of 70 posts
Re: Amazon S3 now supports the ability to append data to an object
#52It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.
doesn't s3 have 'free' (subsidized!) transfer to other products like ec2 though? it might look better to businesspeople that "well, we're doing this processing on ec2, why not keep our objects in s3 since it's a bit cheaper!"
S3 has free data transfer within the same region.
Re: Amazon S3 now supports the ability to append data to an object
#53Earlier quoted context omitted.
I've only used azure a little bit, and mostly liked it - but I'd love to know what kinds of things you're referring to here (mostly on AWS only, so probably I don't even know what I'm missing out on).
What Azure has that from what I've seen AWS does not: Resource Groups that actually act like folders, not just as special tags. Resources with human-readable names instead of gibberish identifiers. Cross-region and cross-subscription (equiv. to AWS account) views of all resources as the default, not as a special feature. Single pane-of-glass across all products instead of separate URLs and consoles for each thing. E.…
Kusto is wonderful. I'd love to be able to use outside of log analytics.
Re: Amazon S3 now supports the ability to append data to an object
#54This is a fantastic addition to Amazon S3 Express One Zone! The ability to directly append data to existing objects opens up new possibilities for real-time data processing applications. Whether it's continuously adding log entries or appending video segments in a media workflow, this feature will streamline workflows and improve efficiency for many use cases. It's great to see AWS continuing to innovate and make data management even more flexible and user-friendly. Excited to see how this feature enhances the scalability of applications across various industries!
Re: Amazon S3 now supports the ability to append data to an object
#55I wonder at which point they'll admit they've added back all the complexity of a hierarchical filesystem.
Re: Amazon S3 now supports the ability to append data to an object
#56It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.
Just wait until Cloudflare decides you can afford to be on a different plan.
Re: Amazon S3 now supports the ability to append data to an object
#57Re: Amazon S3 now supports the ability to append data to an object
#58Wrote some notes on this here: https://simonwillison.net/2024/Nov/22/amazon-s3-append-data/ Key points: - It's just for the "S3 Express One Zone" bucket class, which is more expensive (16c/GB/month compared to 2.3c for S3 standard tier) and less highly available, since it lives in just one availability zone - "With each successful append operation, you create a part of the object and each object can have up to 10,000…
Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015: https://learn.microsoft.com/en-us/rest/api/storageservices/u... Azure supports 50,000 parts, zone-redundancy, and append blobs are supported in the normal "Hot" tier, which is their low-budget mechanical drive storage. Note that both 10K and 50K parts means that you can use a single blob to store a day's worth…
Microsoft did this by sacrificing other features of object storage that S3 and GS had since the beginning, primarily performance, automatic scaling, unlimited provisioning and cross-sectional (region wide) bandwidth. Azure blob storage did not have parity on those features back in 2015 and data platform applications could not be implemented on top of it as a result. Since then they fixed some of these, but there are still areas where Azure lacks scaling features that are taken for granted on AWS and GCP.
Re: Amazon S3 now supports the ability to append data to an object
#59Wrote some notes on this here: https://simonwillison.net/2024/Nov/22/amazon-s3-append-data/ Key points: - It's just for the "S3 Express One Zone" bucket class, which is more expensive (16c/GB/month compared to 2.3c for S3 standard tier) and less highly available, since it lives in just one availability zone - "With each successful append operation, you create a part of the object and each object can have up to 10,000…
Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015: https://learn.microsoft.com/en-us/rest/api/storageservices/u... Azure supports 50,000 parts, zone-redundancy, and append blobs are supported in the normal "Hot" tier, which is their low-budget mechanical drive storage. Note that both 10K and 50K parts means that you can use a single blob to store a day's worth…
In all fairness. Shipping unreliable features for unreliable services is a lot easier
Re: Amazon S3 now supports the ability to append data to an object
#60For comparison, while GCS doesn't support appends directly, there's hacky but effective workaround in that you can compose existing objects together into new objects, without having to read & write the data. If you have existing object A, upload new object B, and compose A and B together so that the resulting object is also called A, this effectively functions the same as appending B into A. https://cloud.google.com/…
S3 can also do this https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPa...
It's similar but no really the same thing. It has to be done up front by initiating a multi-part upload to start. The parts are still technically accessible as S3 objects but through a different API. But the biggest limitation is that each part has to be >5MB (except for the final part)