Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

41–50 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#41

Post by sciurus »

pattycake23 wrote: post #29

Here's an article about Shopify running into the S3 prefix rate limit too many times, and tackling it: https://shopify.engineering/future-proofing-our-cloud-storag...

Their solution was to introduce entropy into the beginning of the object names, which used to be AWS's recommendation for how to ensure objects are placed in different partitions. AWS claims this is no longer necessary, although how their new design actually handles partitioning is opaque.

"This S3 request rate performance increase removes any previous guidance to randomize object prefixes to achieve faster performance. That means you can now use logical or sequential naming patterns in S3 object naming without any performance implications."

https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3...

Re: AWS S3: Sometimes you should press the $100k button

#42

Post by stepchowfun »

xyzzy_plugh wrote: post #34
ebingdom wrote: post #16

I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…

The prefix isn't delimited, it's an arbitrary length based on access patterns. A fictitious example which is close to reality: In parallel, you write a million objects each to: tomato/red/... tomato/green/... tomatoes/colors/... The shortest prefixes that evenly divides writes are thus tomato/r tomato/g tomatoes If you had an existing access pattern of evenly writing to tomatoes/colors/... bananas/... The shortest pr…

Thank you! This is the first explanation that I think fully explains what I was confused about. So essentially the prefix is just the first N bytes of the object's name, where N is a per-bucket number that S3 automatically decides and adjusts for you. And it has nothing to do with delimiters.

I find the S3 documentation and API to be really confusing about this. For example, when listing objects, you get to specify a "prefix". But this seems to be not directly related to the automatically-determined prefix length based on your access patterns. And [1] says things like "There are no limits to the number of prefixes in a bucket.", which makes no sense to me given that the prefix length is something that S3 decides under the hood for you. Like, how do you even know how many prefixes your bucket has?

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...

Re: AWS S3: Sometimes you should press the $100k button

#43

Post by pattycake23 »

sciurus wrote: post #41
pattycake23 wrote: post #29

Here's an article about Shopify running into the S3 prefix rate limit too many times, and tackling it: https://shopify.engineering/future-proofing-our-cloud-storag...

Their solution was to introduce entropy into the beginning of the object names, which used to be AWS's recommendation for how to ensure objects are placed in different partitions. AWS claims this is no longer necessary, although how their new design actually handles partitioning is opaque. "This S3 request rate performance increase removes any previous guidance to randomize object prefixes to achieve faster performan…

Seems like it's a much higher rate limit, but it exists none the less, and Shopify's scale has also grown significantly since 2018 (when that article was written) - so it was probably a valid way for them to go.

Re: AWS S3: Sometimes you should press the $100k button

#44

Post by mst »

throwaway984393 wrote: post #19
solatic wrote: post #12

TL-DR: Object stores are not databases. Don't treat them like one.

Try telling that to developers; they love using S3 as both a database and a filesystem. It's gotten to the point where we need a training for new devs to tell them what not to do in the cloud.

Honestly a Frequently Delivered Answers training for new developers is probably one of the best things you can include in onboarding.

Every environment has its footguns, after all.

Re: AWS S3: Sometimes you should press the $100k button

#45

Post by lenkite »

sigh. My team is facing all these issues. Drowning in data. Crazy S3 bill spikes. And not just S3 - Azure, GCP, Alibaba, etc since we are a multi-cloud product.

Earlier, we couldn't even figure out lifecycle policies to expire objects since naturally every PM had a different opinion on the data lifecycle. So it was old-fashioned cleanup jobs that were scheduled and triggered when a byzantine set of conditions were met. Sometimes they were never met - cue bill spike.

Thankfully, all the new data privacy & protection regulations are a life-saver. Now, we can blindly delete all associated data when a customer off-boards or trial expires or when data is no longer used for original purpose. Just tell the intransigent PM's that we are strictly following govt regulations.

Re: AWS S3: Sometimes you should press the $100k button

#46

Post by inopinatus »

stepchowfun wrote: post #42
xyzzy_plugh wrote: post #34

Earlier quoted context omitted.

The prefix isn't delimited, it's an arbitrary length based on access patterns. A fictitious example which is close to reality: In parallel, you write a million objects each to: tomato/red/... tomato/green/... tomatoes/colors/... The shortest prefixes that evenly divides writes are thus tomato/r tomato/g tomatoes If you had an existing access pattern of evenly writing to tomatoes/colors/... bananas/... The shortest pr…

Thank you! This is the first explanation that I think fully explains what I was confused about. So essentially the prefix is just the first N bytes of the object's name, where N is a per-bucket number that S3 automatically decides and adjusts for you. And it has nothing to do with delimiters. I find the S3 documentation and API to be really confusing about this. For example, when listing objects, you get to specify a…

It is related, in the sense both “prefixes” are a substring match anchored at the start of the object name. They’re just not the same mechanism.

Re: AWS S3: Sometimes you should press the $100k button

#47

Post by orf »

base698 wrote: post #15
cj wrote: post #5

Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)

Yes, and sometimes even listing can take days. I worked somewhere that a person decided using Twitter Firehose was a good idea for S3. Keyed by tweet per file. Ended up figuring out a way to get them in batches and condense. Ended up costing about $800 per hour to fix coupled with lifecycle changes they mentioned.

> Yes, and sometimes even listing can take days.

You have a versioned bucket with a lot of delete markers in it. Make sure you've got a lifecycle policy to clean them up.

Re: AWS S3: Sometimes you should press the $100k button

#48

Post by tazjin »

jwalton wrote: post #22

Your website renders as a big empty blue page in Firefox unless I disable tracking protection (and in my case, since I have noscript, I have to enable javascript for "website-files.com", a domain that sounds totally legit).

Chrome with uBlock Origin on default here, and it renders a big blue empty page for me, too. That's despite dragging in an ungodly amount of assets first.

Here's an archive link that works without any tracking, ads, Javascript etc.: https://archive.is/F5KZd

Re: AWS S3: Sometimes you should press the $100k button

#49

Post by jrochkind1 »

sokoloff wrote: post #39
cle wrote: post #14

Earlier quoted context omitted.

This is mostly correct, with the additional feature that S3 can efficiently list objects by "key prefix" which helps preserve the illusion.

Followup question: Is there something special about the PRE notations in the example output below? I can list objects by any textual prefix, but I can't tell if the PRE (what we think of as folders) is more efficient than just the substring prefix. Full bucket list, then two text prefix, then an (empty) folder list sokoloff@ Downloads % aws s3 ls s3://foo-asdf PRE bar-folder/ PRE baz-folder/ 2022-02-17 09:25:38 0 bar…

I don't understand the answer to that question either. Other AWS docs says you can choose whatever you want for a delimiter, there's nothing special about `/`. So how does that apply to what they say about performance and "prefixes"?

Here is some AWS documentation on it:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...

> For example, your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket. There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by using parallelization. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second.

Related to your question, even if we just stick to `/` because it seems safer, does that mean that "foo/bar/baz/1/" and "foo/bar/baz/2/" are two prefixes for the point of these request speed limits? Or does the "prefix" stop at the first "/" and files with these keypaths are both in the same "prefix" "foo/"?

Note there was (according to docs) a change a couple years ago that I think some people haven't caught on to:

> For example, previously Amazon S3 performance guidelines recommended randomizing prefix naming with hashed characters to optimize performance for frequent data retrievals. You no longer have to randomize prefix naming for performance, and can use sequential date-based naming for your prefixes.

Re: AWS S3: Sometimes you should press the $100k button

#50

Post by dale_glass »

ebingdom wrote: post #31
inopinatus wrote: post #25

Earlier quoted context omitted.

There’s no delimiter. There is only the appearance of a delimiter, to appease folks who think S3 is a filesystem, and fool them into thinking they’re looking at folders. The object name is the entire label, and every character is equally significant for storage. When listing objects, a prefix filters the list. That’s all. However, S3 also uses substrings to partition the bucket for scale. Since they’re anchored at th…

So if I have a bunch of objects whose names are hashes like 2df6ad6ca44d06566cffde51155e82ad0947c736 that I expect to access randomly, is there any performance benefit to introducing artificial delimiters like 2d/f6/ad6ca44d06566cffde51155e82ad0947c736? I've seen this used in some places.

To AWS S3, '/' isn't a delimiter, it's a character that's part of the filename.

So for instance "/foo/bar.txt" and "/foo//bar.txt" are different files in S3, even though they'd be the same file in a filesystem.

This gets pretty fun if you want to mirror a S3 structure on-disk, because the above suddenly causes a collision.

Post reply on HN