AWS S3: Sometimes you should press the $100k button
1–10 of 240 posts
Re: AWS S3: Sometimes you should press the $100k button
#2I've never been in this situation, but I do wish you could query files with more advanced filters on these blob storage services.
- But why SageMaker?
- Why do some orgs choose to put almost everything in 1 buckets?
Re: AWS S3: Sometimes you should press the $100k button
#3We’ve got data in S3 buckets not nearly at that scale and managing them, god forbid trying a mass delete, is absolute tedium.
Re: AWS S3: Sometimes you should press the $100k button
#4I've never been in this situation, but I do wish you could query files with more advanced filters on these blob storage services. - But why SageMaker? - Why do some orgs choose to put almost everything in 1 buckets?
For many at orgs like this, SageMaker is probably the shortest path to an insane amount of compute with a python terminal.
Why single bucket? Once someone refers to a bucket as "the" bucket - it is how it will forever be.
Re: AWS S3: Sometimes you should press the $100k button
#5Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)
Re: AWS S3: Sometimes you should press the $100k button
#6Though it doesn't address the problem in TFA, I recommend setting up billing alerts in AWS. Doesn't solve their issue, but they would have at least known about it sooner.
Re: AWS S3: Sometimes you should press the $100k button
#7We’ve got data in S3 buckets not nearly at that scale and managing them, god forbid trying a mass delete, is absolute tedium.
Mass delete also takes an eternity on my Linux desktop machine.
The filesystem is hierarchical, but the delete operation still needs to visit all the leaves.
Re: AWS S3: Sometimes you should press the $100k button
#8Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)
I suggest you raise a support ticket.
AFAIK there is server-side paging implemented in the List* API operations that the Console UI should be using so that the number of objects in a bucket should not significantly impact the webpage performance.
But who knows what design flaws lurk beneath the console.
Curious to know what you find.
Does it happen only on opening heavy buckets? or the entire S3 console? Different Browser / incognito / different machine ...dont make a difference?
Re: AWS S3: Sometimes you should press the $100k button
#9I've never been in this situation, but I do wish you could query files with more advanced filters on these blob storage services. - But why SageMaker? - Why do some orgs choose to put almost everything in 1 buckets?
>Why do some orgs choose to put almost everything in 1 buckets?
The article seems to be making the case it's because the delimiter makes it seem like there's a real hierarchy. So the ramifications of /bucket/1 /bucket/2 versus /bucket1/ /bucket2/ aren't well known until it's too late.
Re: AWS S3: Sometimes you should press the $100k button
#10Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)
the newer s3 console works a little better. It gives pagination with ""