Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 now supports the ability to append data to an object

aws.amazon.com

Post reply on HN
Print view View original

31–40 of 70 posts

Re: Amazon S3 now supports the ability to append data to an object

#31

Post by jiggawatts »

simonw wrote: post #15

Wrote some notes on this here: https://simonwillison.net/2024/Nov/22/amazon-s3-append-data/ Key points: - It's just for the "S3 Express One Zone" bucket class, which is more expensive (16c/GB/month compared to 2.3c for S3 standard tier) and less highly available, since it lives in just one availability zone - "With each successful append operation, you create a part of the object and each object can have up to 10,000…

Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015: https://learn.microsoft.com/en-us/rest/api/storageservices/u...

Azure supports 50,000 parts, zone-redundancy, and append blobs are supported in the normal "Hot" tier, which is their low-budget mechanical drive storage.

Note that both 10K and 50K parts means that you can use a single blob to store a day's worth of logs and flush every minute (1,440 parts). Conversely, hourly blobs can support flushing every second (3,600 parts). Neither support daily blobs with per-second flushing for a whole day (86,400 parts).

Typical designs involve a per-server log, per hour. So the blob path looks like:

    "{account}/{path}/{year}/{month}/{day}/{hour}_{servername}.txt"
This seems insane, but it's not a file system! You don't need to create directories, and you're not supposed to read these using VIM, Notepad, or whatever.

The typical workflow is to run a daily consolidation into an indexed columnstore format like Parquet, or send it off to Splunk, Log Analytics, or whatever...

Re: Amazon S3 now supports the ability to append data to an object

#32

Post by YetAnotherNick »

sureIy wrote: post #20

It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.

In most cases S3 data is not directly exposed to the client. If the middleware is EC2, then you need to pay the same egress fee, but you you will have lower latency with S3, as EC2 shares the same datacenter as S3.

Re: Amazon S3 now supports the ability to append data to an object

#34

Post by JonoBB »

sureIy wrote: post #20

It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.

For the most part I agree, but we have found that R2 does not handle large files (hundreds of GB or larger) very well. It will often silently fail with nothing being returned, so it’s not possible to handle it gracefully.

Re: Amazon S3 now supports the ability to append data to an object

#35

Post by wood_spirit »

Will be exciting to see what adaptations are needed and how performance and cost changes for delta lake and iceberg and other cloud mutable data storage formats. It could be really dramatic!

S3 is often used as a lowest common denominator, and a lot of the features of azure and gcs aren’t leveraged by libraries and formats that try to be cross platform so only want to expose features that are available everywhere.

If these days all object stores do append then perhaps all the data storage formats and libs can start leveraging it?

Re: Amazon S3 now supports the ability to append data to an object

#36

Post by orthoxerox »

andrewstuart wrote: post #27

There are many, many S3 compatible storage services out there provided by other companies. Most of them cheaper, some MUCH cheaper.

But most of them are compliant with the standard S3 API. If I use AWS SDK to write data to my on-prem Ceph/Minio/SeaweedFS/Hitachi storage, I want this SDK to support the concept of appending data to an object.

Re: Amazon S3 now supports the ability to append data to an object

#37

Post by kochie »

jiggawatts wrote: post #31
simonw wrote: post #15

Wrote some notes on this here: https://simonwillison.net/2024/Nov/22/amazon-s3-append-data/ Key points: - It's just for the "S3 Express One Zone" bucket class, which is more expensive (16c/GB/month compared to 2.3c for S3 standard tier) and less highly available, since it lives in just one availability zone - "With each successful append operation, you create a part of the object and each object can have up to 10,000…

Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015: https://learn.microsoft.com/en-us/rest/api/storageservices/u... Azure supports 50,000 parts, zone-redundancy, and append blobs are supported in the normal "Hot" tier, which is their low-budget mechanical drive storage. Note that both 10K and 50K parts means that you can use a single blob to store a day's worth…

AWS ranks features based on potential income from customers. Normally there’s a fairly big customer PFR needed to get a service team to implement a new feature.

Re: Amazon S3 now supports the ability to append data to an object

#38

Post by jiggawatts »

kochie wrote: post #37
jiggawatts wrote: post #31

Earlier quoted context omitted.

Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015: https://learn.microsoft.com/en-us/rest/api/storageservices/u... Azure supports 50,000 parts, zone-redundancy, and append blobs are supported in the normal "Hot" tier, which is their low-budget mechanical drive storage. Note that both 10K and 50K parts means that you can use a single blob to store a day's worth…

AWS ranks features based on potential income from customers. Normally there’s a fairly big customer PFR needed to get a service team to implement a new feature.

I always found it strange that AWS seems to have 2-3x as many products or services as Azure, but it has these bizarre feature gaps where as an Azure user I think: "Really? Now? In this year you're finally getting this?"

(Conversely, Azure's low-level performance is woeful in comparison to AWS and they're still slow-walking the rollout of their vaguely equivalent networking and storage called Azure Boost.)

Re: Amazon S3 now supports the ability to append data to an object

#39

Post by WaxProlix »

jiggawatts wrote: post #38
kochie wrote: post #37

Earlier quoted context omitted.

AWS ranks features based on potential income from customers. Normally there’s a fairly big customer PFR needed to get a service team to implement a new feature.

I always found it strange that AWS seems to have 2-3x as many products or services as Azure, but it has these bizarre feature gaps where as an Azure user I think: "Really? Now? In this year you're finally getting this?" (Conversely, Azure's low-level performance is woeful in comparison to AWS and they're still slow-walking the rollout of their vaguely equivalent networking and storage called Azure Boost.)

I've only used azure a little bit, and mostly liked it - but I'd love to know what kinds of things you're referring to here (mostly on AWS only, so probably I don't even know what I'm missing out on).

Re: Amazon S3 now supports the ability to append data to an object

#40

Post by vdm »

teractiveodular wrote: post #6

For comparison, while GCS doesn't support appends directly, there's hacky but effective workaround in that you can compose existing objects together into new objects, without having to read & write the data. If you have existing object A, upload new object B, and compose A and B together so that the resulting object is also called A, this effectively functions the same as appending B into A. https://cloud.google.com/…

S3 can also do this https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPa...
Post reply on HN