Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

71–80 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#71

Post by xyzzy_plugh »

stepchowfun wrote: post #42
xyzzy_plugh wrote: post #34

Earlier quoted context omitted.

The prefix isn't delimited, it's an arbitrary length based on access patterns. A fictitious example which is close to reality: In parallel, you write a million objects each to: tomato/red/... tomato/green/... tomatoes/colors/... The shortest prefixes that evenly divides writes are thus tomato/r tomato/g tomatoes If you had an existing access pattern of evenly writing to tomatoes/colors/... bananas/... The shortest pr…

Thank you! This is the first explanation that I think fully explains what I was confused about. So essentially the prefix is just the first N bytes of the object's name, where N is a per-bucket number that S3 automatically decides and adjusts for you. And it has nothing to do with delimiters. I find the S3 documentation and API to be really confusing about this. For example, when listing objects, you get to specify a…

The sharding key is an implementation detail, so you're not supposed to care about it too much.

Re: AWS S3: Sometimes you should press the $100k button

#72

Post by deepsun »

amelius wrote: post #7
Tehchops wrote: post #3

We’ve got data in S3 buckets not nearly at that scale and managing them, god forbid trying a mass delete, is absolute tedium.

Mass delete also takes an eternity on my Linux desktop machine. The filesystem is hierarchical, but the delete operation still needs to visit all the leaves.

I believe it's mostly a problem of latency between your machine and S3. Since each Delete call is issued separately in its own HTTP connection.

1. Try parallelization of your calls. Deleting 20 objects in parallel should take the same time as deleting 1.

2. Try to run deletion from an AWS machine in the same region as the S3 bucket (yes buckets are regional, only their names are global). Within-datacenter latency should be lower than between your machine and datacenter.

Re: AWS S3: Sometimes you should press the $100k button

#73

Post by WrtCdEvrydy »

darkwater wrote: post #53
liveoneggs wrote: post #17

I have caused billing spikes like this before those little warnings were invented and it was always a dark day. They are really a life saver. Lifecycle rules are also welcome. Writing them yourself was always a pain and tended to be expensive with list operations eating up that api calls bill. ---- Once I supported an app that dumped small objects into s3 and begged the dev team to store the small objects in oracle a…

> I then went for a compromise solution asking if we could stitch the small objects together after a period of time so they would be eligible for things like infrequent access or glacier but, alas, "dev time is expensive you know" so N figure s3 bills continue as far as I know. This hits home so hard that it hurts. In my case is not S3 but compute bills but the core concept is the same.

Because the bill isn't a "dev problem". Once you move those bills to "devops", it becomes an infrastructure problem.

Re: AWS S3: Sometimes you should press the $100k button

#74

Post by [deleted] »

darkwater wrote: post #53
liveoneggs wrote: post #17

I have caused billing spikes like this before those little warnings were invented and it was always a dark day. They are really a life saver. Lifecycle rules are also welcome. Writing them yourself was always a pain and tended to be expensive with list operations eating up that api calls bill. ---- Once I supported an app that dumped small objects into s3 and begged the dev team to store the small objects in oracle a…

> I then went for a compromise solution asking if we could stitch the small objects together after a period of time so they would be eligible for things like infrequent access or glacier but, alas, "dev time is expensive you know" so N figure s3 bills continue as far as I know. This hits home so hard that it hurts. In my case is not S3 but compute bills but the core concept is the same.

[deleted]

Re: AWS S3: Sometimes you should press the $100k button

#75

Post by hvs »

whimsicalism wrote: post #68
CydeWeys wrote: post #55

Earlier quoted context omitted.

The data protection regulations really are so freeing, huh. It's amazing to be able to delete all this stuff without worrying about having to keep it forever.

now this is a spin i havent heard before.

You haven't heard it because it's not spin, it's from an engineer's point of view. That's not the view you hear in the news when it comes to these things.

Re: AWS S3: Sometimes you should press the $100k button

#76

Post by jeff_vader »

CydeWeys wrote: post #55
lenkite wrote: post #45

sigh . My team is facing all these issues. Drowning in data. Crazy S3 bill spikes. And not just S3 - Azure, GCP, Alibaba, etc since we are a multi-cloud product. Earlier, we couldn't even figure out lifecycle policies to expire objects since naturally every PM had a different opinion on the data lifecycle. So it was old-fashioned cleanup jobs that were scheduled and triggered when a byzantine set of conditions were m…

The data protection regulations really are so freeing, huh. It's amazing to be able to delete all this stuff without worrying about having to keep it forever.

In case of my previous employer it led to incredibly complicated encryption system. It took couple years to maybe implement in 10% of the system. Deleting any old data was rejected.

Re: AWS S3: Sometimes you should press the $100k button

#77

Post by jsmith45 »

sokoloff wrote: post #39
cle wrote: post #14

Earlier quoted context omitted.

This is mostly correct, with the additional feature that S3 can efficiently list objects by "key prefix" which helps preserve the illusion.

Followup question: Is there something special about the PRE notations in the example output below? I can list objects by any textual prefix, but I can't tell if the PRE (what we think of as folders) is more efficient than just the substring prefix. Full bucket list, then two text prefix, then an (empty) folder list sokoloff@ Downloads % aws s3 ls s3://foo-asdf PRE bar-folder/ PRE baz-folder/ 2022-02-17 09:25:38 0 bar…

Umm... that output seems confusing.

The ListObjects api will omit all objects that share a prefix that ends in the delimiter, and instead put said prefix into the CommonPrefix element, which would be reflected as PRE lines. (So with a delimiter of '/', it basically hides objects in "subfolders", but lists any subfolders that match your partial text in the CommonPrefix element).

By default `aws s3 ls` will not show any objects within a CommonPrefix but simply shows a PRE line for them. The cli does not let you specify a delimiter, it always uses '/'. To actually list all objects you need to use `--recursive`.

The output there would suggest that bucket really did have object names that began with `bar-folder/`, and that last line did not list them out because you did not include the trailing slash. Without the trailing slash it was just listing objects and CommonPrefixes that match the string you specified after the last delimiter in your url. Since only that one common prefix matched, only it was printed.

Re: AWS S3: Sometimes you should press the $100k button

#78

Post by blurker »

StratusBen wrote: post #59

On this topic, it's always surprising to me how few people even seem to know about different storage classes on S3...or even intelligent tiering (which I know carries a cost to it, but allows AWS to manage some of this on your behalf which can be helpful for certain use-cases and teams). We did an analysis of S3 storage levels by profiling 25,000 random S3 buckets a while back for a comparison of Amazon S3 and R2* an…

I think that it's not just people not knowing about the lifecycle feature, but also that when they start putting data into a bucket they don't know what the lifecycle should be yet. Honestly I think overdoing lifecycle policies is a potentially bigger foot gun than not setting them. If you misuse glacier storage that will really cost you big $$$ quickly! And who wants to be the dev who deleted a bunch of data they shouldn't have?

Lifecycle policies are simple in concept, but it's actually not simple to decide what they should be in many cases.

Re: AWS S3: Sometimes you should press the $100k button

#79

Post by jq-r »

cj wrote: post #5

Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)

In my previous company we had around 15K instances in a EC2 region and the EC2 GUI was unusable if it was set on the "new gui experience" so we always had to use classic one. The new one would try to get all the details of them so once loaded it was fast. But to get there it would take many minutes or it would just expire. Don't know if they've fixed that.

Re: AWS S3: Sometimes you should press the $100k button

#80

Post by ramraj07 »

anderiv wrote: post #18
Tehchops wrote: post #3

We’ve got data in S3 buckets not nearly at that scale and managing them, god forbid trying a mass delete, is absolute tedium.

Set a lifecycle rule to delete your objects. Come back a day later and AWS will have taken care of this for you.

The issue is this isn’t free. I played and emended up with a few hundred million object S3 bucket on a personal project and am trying to get rid of it without getting a bill. Seriously considering just getting suspended from aws if that’s a viable path lol.
Post reply on HN