DeleteObjects takes 1000 keys per call.
Lifecycle rules can filter by min/max object size. (since Nov 2021)
AWS S3: Sometimes you should press the $100k button
21–30 of 240 posts
Re: AWS S3: Sometimes you should press the $100k button
#22Your website renders as a big empty blue page in Firefox unless I disable tracking protection (and in my case, since I have noscript, I have to enable javascript for "website-files.com", a domain that sounds totally legit).
Re: AWS S3: Sometimes you should press the $100k button
#23I've never been in this situation, but I do wish you could query files with more advanced filters on these blob storage services. - But why SageMaker? - Why do some orgs choose to put almost everything in 1 buckets?
>Why do some orgs choose to put almost everything in 1 buckets? The article seems to be making the case it's because the delimiter makes it seem like there's a real hierarchy. So the ramifications of /bucket/1 /bucket/2 versus /bucket1/ /bucket2/ aren't well known until it's too late.
>So the ramifications of /bucket/1 /bucket/2 versus /bucket1/ /bucket2/ aren't well known until it's too late.
What's the difference?
Re: AWS S3: Sometimes you should press the $100k button
#24Can someone explain what happened in the end? From my understanding nothing happened (they deprioritizod the story for fixing it) and they are still blowing through the cloud budget.
Re: AWS S3: Sometimes you should press the $100k button
#25I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…
There’s no delimiter. There is only the appearance of a delimiter, to appease folks who think S3 is a filesystem, and fool them into thinking they’re looking at folders.
The object name is the entire label, and every character is equally significant for storage. When listing objects, a prefix filters the list. That’s all. However, S3 also uses substrings to partition the bucket for scale. Since they’re anchored at the start, they’re also called prefixes.
In my view, it’s best to think of S3’s object indexing as a radix tree.
This article, as if you couldn’t guess from the content, is written from a position of scant knowledge of S3, not surprising it misrepresents the details.
Re: AWS S3: Sometimes you should press the $100k button
#26I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…
AWS does the optimizations over time based on access patterns for the data. Should have made that clearer in the article. The problem becomes unusual burst load - usually from infrequent analytics jobs. The indexing cant respond fast enough.
Thanks for the clarification. But now I'm confused about the limits:
> 3,500 PUT/COPY/POST/DELETE requests per second per prefix
> 5,500 GET/HEAD requests per second per prefix
Most of those APIs don't even take a delimiter. So for these limits, does the prefix get inferred based on whatever delimiter you've used for previous list requests? What if you've used multiple delimiters in the past?
Basically what I'm trying to determine is whether these limits actually mean something concrete (that I can use for capacity planning etc.), or whether their behavior depends on heuristics that S3 uses under the hood.
I'm fine with S3 optimizing things under the hood based on access my patterns, but not if it means I can't reason about these limits as an outsider.
Re: AWS S3: Sometimes you should press the $100k button
#27DeleteObjects takes 1000 keys per call. Lifecycle rules can filter by min/max object size. (since Nov 2021)
Athena supports regexp_like(). By loading in an S3 inventory this can match what a wildcard would. Then a Batch Operations job can tag the result.
Not easy, but is possible and effective.
Re: AWS S3: Sometimes you should press the $100k button
#28We’ve got data in S3 buckets not nearly at that scale and managing them, god forbid trying a mass delete, is absolute tedium.
Mass delete also takes an eternity on my Linux desktop machine. The filesystem is hierarchical, but the delete operation still needs to visit all the leaves.
Most recursive deletion routines are not optimized for speed. This could be done much faster with multiple threads or batching the calls via io_uring.
Another option are LVM or btrfs subvolumes which can be discarded without recursive traversal.
Re: AWS S3: Sometimes you should press the $100k button
#29Here's an article about Shopify running into the S3 prefix rate limit too many times, and tackling it: https://shopify.engineering/future-proofing-our-cloud-storag...
Re: AWS S3: Sometimes you should press the $100k button
#30Earlier quoted context omitted.
AWS does the optimizations over time based on access patterns for the data. Should have made that clearer in the article. The problem becomes unusual burst load - usually from infrequent analytics jobs. The indexing cant respond fast enough.
Thanks for the clarification. But now I'm confused about the limits: > 3,500 PUT/COPY/POST/DELETE requests per second per prefix > 5,500 GET/HEAD requests per second per prefix Most of those APIs don't even take a delimiter. So for these limits, does the prefix get inferred based on whatever delimiter you've used for previous list requests? What if you've used multiple delimiters in the past? Basically what I'm tryin…
S3 does a lot of under the hood optimisation. e.g. Create a brand new bucket, leave it cold for a while, and start throwing 100 PUT requests a second at it. This is way less than the advertised 3500, but they'll have scaled the allocated resources down so much you'll get some TooManyRequests errors.