Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 now supports the ability to append data to an object

aws.amazon.com

Post reply on HN
Print view View original

41–50 of 70 posts

Re: Amazon S3 now supports the ability to append data to an object

#41

Post by andrewstuart »

Amazon has no right to do this - it no longer owns the S3 standard and should respect the ecosystem and community.

S3 has stagnated for a long time, allowing it to become a standard.

Third parties have cloned the storage service and a vast array of software is compatible. There’s drivers, there’s file transfer programs and utilities.

What does it mean that Amazon is now changing it.

Does Amazon even really own the standard any more, does it have the right to break the long standing standards?

I’m reminded of IBM when they broke compatibility of the PS/2 computers just so it could maintain dominance.

Re: Amazon S3 now supports the ability to append data to an object

#42

Post by remus »

sureIy wrote: post #20

It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.

Depends a bit on your use case. If you've got lots of other infra on AWS and you don't need to store that much data then the downside of using another platform can outweigh the cost savings.

Re: Amazon S3 now supports the ability to append data to an object

#43

Post by lijok »

sureIy wrote: post #20

It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.

People still use S3 because doing business with Cloudflare is a liability.

Re: Amazon S3 now supports the ability to append data to an object

#44

Post by new_user_final »

vdm wrote: post #40
teractiveodular wrote: post #6

For comparison, while GCS doesn't support appends directly, there's hacky but effective workaround in that you can compose existing objects together into new objects, without having to read & write the data. If you have existing object A, upload new object B, and compose A and B together so that the resulting object is also called A, this effectively functions the same as appending B into A. https://cloud.google.com/…

S3 can also do this https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPa...

It's totally different thing and requires special way to initiate multi-part uploading.

Re: Amazon S3 now supports the ability to append data to an object

#45

Post by sofixa »

jiggawatts wrote: post #31
simonw wrote: post #15

Wrote some notes on this here: https://simonwillison.net/2024/Nov/22/amazon-s3-append-data/ Key points: - It's just for the "S3 Express One Zone" bucket class, which is more expensive (16c/GB/month compared to 2.3c for S3 standard tier) and less highly available, since it lives in just one availability zone - "With each successful append operation, you create a part of the object and each object can have up to 10,000…

Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015: https://learn.microsoft.com/en-us/rest/api/storageservices/u... Azure supports 50,000 parts, zone-redundancy, and append blobs are supported in the normal "Hot" tier, which is their low-budget mechanical drive storage. Note that both 10K and 50K parts means that you can use a single blob to store a day's worth…

> Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015:

Microsoft had the benefit of starting later and learning from Amazon's failures and successes. S3 dates from 2006.

That being said, both Microsoft and Google learned a lot, but also failed at learning different things.

GCP has a lovely global network, which makes multi-region easy. But they spent way too much time on GCE and lost the early advantage they had with Google App Engine.

Azure severely lacks in security (check out how many critical cross-tenant security vulnerabilities they've had in the past few years) and reliability (how many times have there been various outages due to a single DC in Texas failing; availability zones still aren't the default there).

Re: Amazon S3 now supports the ability to append data to an object

#46

Post by cedilla »

jiggawatts wrote: post #31
simonw wrote: post #15

Wrote some notes on this here: https://simonwillison.net/2024/Nov/22/amazon-s3-append-data/ Key points: - It's just for the "S3 Express One Zone" bucket class, which is more expensive (16c/GB/month compared to 2.3c for S3 standard tier) and less highly available, since it lives in just one availability zone - "With each successful append operation, you create a part of the object and each object can have up to 10,000…

Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015: https://learn.microsoft.com/en-us/rest/api/storageservices/u... Azure supports 50,000 parts, zone-redundancy, and append blobs are supported in the normal "Hot" tier, which is their low-budget mechanical drive storage. Note that both 10K and 50K parts means that you can use a single blob to store a day's worth…

If I need to consolidate anyway, is this really a win for this use case? I could just upload with {hour}_{minute}.txt instead of appending every minute, right?

Re: Amazon S3 now supports the ability to append data to an object

#47

Post by anentropic »

sureIy wrote: post #20

It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.

There are reasons like this: https://github.com/cloudflare/terraform-provider-cloudflare/...

Re: Amazon S3 now supports the ability to append data to an object

#48

Post by supermatt »

This doesnt seem very useful for many cases, given that you NEED to specify a write offset in order for it to work. So you need to either track the size (which becomes more complex if you have multiple writers), or need to first request the size every time you want to do a write and then race for it using the checksum of the current object... Urghhh.

Re: Amazon S3 now supports the ability to append data to an object

#49

Post by bobnamob »

lijok wrote: post #43
sureIy wrote: post #20

It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.

People still use S3 because doing business with Cloudflare is a liability.

Elaborate?

Re: Amazon S3 now supports the ability to append data to an object

#50

Post by jiggawatts »

WaxProlix wrote: post #39
jiggawatts wrote: post #38

Earlier quoted context omitted.

I always found it strange that AWS seems to have 2-3x as many products or services as Azure, but it has these bizarre feature gaps where as an Azure user I think: "Really? Now? In this year you're finally getting this?" (Conversely, Azure's low-level performance is woeful in comparison to AWS and they're still slow-walking the rollout of their vaguely equivalent networking and storage called Azure Boost.)

I've only used azure a little bit, and mostly liked it - but I'd love to know what kinds of things you're referring to here (mostly on AWS only, so probably I don't even know what I'm missing out on).

What Azure has that from what I've seen AWS does not:

Resource Groups that actually act like folders, not just as special tags.

Resources with human-readable names instead of gibberish identifiers.

Cross-region and cross-subscription (equiv. to AWS account) views of all resources as the default, not as a special feature.

Single pane-of-glass across all products instead of separate URLs and consoles for each thing. E.g.: a VM writing to an S3 bucket dedicated to it are "far apart" from each other in AWS consoles, but the equivalent resources are directly adjacent to each other in an Azure Resource Group when viewed in its Portal.

Azure Application Insights is a genuinely good APM, and the Log Analytics workspace it uses under the hood is the consistent log collection platform across everything in Azure and even Entra ID and parts of Microsoft 365. It's not as featureful as Splunk, but the query language is up there in capability.

Azure App Service has its flaws, but it's by far the most featureful serverless web app hosting platform.

Etc...

Post reply on HN