Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon S3 now supports the ability to append data to an object

aws.amazon.com

Post reply on HN
Print view View original

61–70 of 70 posts

Re: Amazon S3 now supports the ability to append data to an object

#61

Post by jiggawatts »

cedilla wrote: post #46
jiggawatts wrote: post #31

Earlier quoted context omitted.

Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015: https://learn.microsoft.com/en-us/rest/api/storageservices/u... Azure supports 50,000 parts, zone-redundancy, and append blobs are supported in the normal "Hot" tier, which is their low-budget mechanical drive storage. Note that both 10K and 50K parts means that you can use a single blob to store a day's worth…

If I need to consolidate anyway, is this really a win for this use case? I could just upload with {hour}_{minute}.txt instead of appending every minute, right?

Consolidation is for archival cost efficiency and long-term analytics. If you don't append regularly, you can lose up to 59 minutes of data.

Re: Amazon S3 now supports the ability to append data to an object

#63

Post by withinboredom »

jiggawatts wrote: post #50
WaxProlix wrote: post #39

Earlier quoted context omitted.

I've only used azure a little bit, and mostly liked it - but I'd love to know what kinds of things you're referring to here (mostly on AWS only, so probably I don't even know what I'm missing out on).

What Azure has that from what I've seen AWS does not: Resource Groups that actually act like folders, not just as special tags. Resources with human-readable names instead of gibberish identifiers. Cross-region and cross-subscription (equiv. to AWS account) views of all resources as the default, not as a special feature. Single pane-of-glass across all products instead of separate URLs and consoles for each thing. E.…

Don’t forget, you don’t pay for a stopped vm in azure! You only pay while it is running. This makes things like dev environments much more affordable, since you won’t be paying for nights/weekends.

Re: Amazon S3 now supports the ability to append data to an object

#64

Post by CharlieDigital »

teractiveodular wrote: post #6

For comparison, while GCS doesn't support appends directly, there's hacky but effective workaround in that you can compose existing objects together into new objects, without having to read & write the data. If you have existing object A, upload new object B, and compose A and B together so that the resulting object is also called A, this effectively functions the same as appending B into A. https://cloud.google.com/…

There are some limitations[0] to work around (can only compose 32 at a time and it doesn't auto delete composed parts), but I find this approach super useful for data ingest and ETL processing flows while being quite easy to use.

[0] https://chrlschn.dev/blog/2024/07/merging-objects-in-google-...

Re: Amazon S3 now supports the ability to append data to an object

#65

Post by jrpelkonen »

msoad wrote: post #5

Does it really work for livestreams? Can I stream read and write on the same video file? That is huge if true! Edit: oh it’s only in one AZ

Livestream is not usually done by writing to/reeding from a single media file. Instead, the media is broken into few second long segments. The current set of segments are indicated by a HLS and/or DASH manifest, which is updated as new segments appear.

Re: Amazon S3 now supports the ability to append data to an object

#66

Post by mxuribe »

lijok wrote: post #43
sureIy wrote: post #20

It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.

People still use S3 because doing business with Cloudflare is a liability.

Genuinely curious what you meant by this?

Re: Amazon S3 now supports the ability to append data to an object

#67

Post by bradleyjg »

jiggawatts wrote: post #38
kochie wrote: post #37

Earlier quoted context omitted.

AWS ranks features based on potential income from customers. Normally there’s a fairly big customer PFR needed to get a service team to implement a new feature.

I always found it strange that AWS seems to have 2-3x as many products or services as Azure, but it has these bizarre feature gaps where as an Azure user I think: "Really? Now? In this year you're finally getting this?" (Conversely, Azure's low-level performance is woeful in comparison to AWS and they're still slow-walking the rollout of their vaguely equivalent networking and storage called Azure Boost.)

Low level performance is underselling the issues. Blob storage is not infinitely scalable. That means it’s just not the same thing as s3.

Re: Amazon S3 now supports the ability to append data to an object

#68

Post by dragonwriter »

sureIy wrote: post #20

It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees. I regularly see people switching to R2 and saving thousands or hundreds of thousands by switching.

> It's crazy to me that anyone would still consider S3 after R2 was made available, given the egress fees.

If your compute is on AWS, using R2 (or anything outside of AWS) for object storage means you pay AWS egress for “in-system” operations rather than at the system boundary, which is often much more expensive (plus, you also probably add a bunch of latency compared to staying on AWS infra.) And unless you are exposing your object store directly externally as your interface to the world, you still pay AWS egress at the boundary.

Now, if all you use AWS for is S3, R2 may be, from a cost perspective, a no brainer, but who does that?

Re: Amazon S3 now supports the ability to append data to an object

#69

Post by jiggawatts »

ak217 wrote: post #58
jiggawatts wrote: post #31

Earlier quoted context omitted.

Wow, I'm surprised it took AWS this long to (mostly) catch up to Azure, which had this feature back in 2015: https://learn.microsoft.com/en-us/rest/api/storageservices/u... Azure supports 50,000 parts, zone-redundancy, and append blobs are supported in the normal "Hot" tier, which is their low-budget mechanical drive storage. Note that both 10K and 50K parts means that you can use a single blob to store a day's worth…

Microsoft did this by sacrificing other features of object storage that S3 and GS had since the beginning, primarily performance, automatic scaling, unlimited provisioning and cross-sectional (region wide) bandwidth. Azure blob storage did not have parity on those features back in 2015 and data platform applications could not be implemented on top of it as a result. Since then they fixed some of these, but there are…

Today I learned that there is a 5 PB soft capacity limit for Azure blob storage: https://learn.microsoft.com/en-us/azure/storage/common/scala...

Also, a 200 Gbps egress limit.

How does that compare to S3?

Mind you, at this scale the storage cost is about $15K/mo, so it would be cost effective to throw some developer time at the problem of scaling out between multiple storage accounts. Or just call support to have the soft limit cap raised…

Post reply on HN