Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

121–130 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#121

Post by aeyes »

stingraycharles wrote: post #110
jeff_vader wrote: post #76

Earlier quoted context omitted.

In case of my previous employer it led to incredibly complicated encryption system. It took couple years to maybe implement in 10% of the system. Deleting any old data was rejected.

How is encryption compliant? I’ve implemented GDPR data infrastructures twice now, and as far as I’m aware, the only way to be compliant with encryption is when you throw the decryption key away.

Sometimes it might be a single field in a 1MB nested structure that you have to remove. So it gets encrypted when the whole structure gets stored and when the field is to be deleted you just throw away the key instead of modifying the entire 1MB just to remove a few kB.

Re: AWS S3: Sometimes you should press the $100k button

#122

Post by kristjansson »

cj wrote: post #5

Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)

Just checked, out of curiosity. A bucket at $WORK with ~4B objects / ~100TB is completely usable through the console. Keys are hierarchal, and relatively deep, so no one page on the GUI is trying to show more than a few hundred keys. If your keys are flatter, I could see how the console be unhappy.

Re: AWS S3: Sometimes you should press the $100k button

#123

Post by kristjansson »

xyzzy_plugh wrote: post #71
stepchowfun wrote: post #42

Earlier quoted context omitted.

Thank you! This is the first explanation that I think fully explains what I was confused about. So essentially the prefix is just the first N bytes of the object's name, where N is a per-bucket number that S3 automatically decides and adjusts for you. And it has nothing to do with delimiters. I find the S3 documentation and API to be really confusing about this. For example, when listing objects, you get to specify a…

The sharding key is an implementation detail, so you're not supposed to care about it too much.

That's true now. Used to be the case that they'd recommend random or high-entropy parts of the keys go at the beginning to avoid overloading a shard as you described above.

From [0]:

> This S3 request rate performance increase removes any previous guidance to randomize object prefixes to achieve faster performance. That means you can now use logical or sequential naming patterns in S3 object naming without any performance implications. This improvement is now available in all AWS Regions. For more information, visit the Amazon S3 Developer Guide.

[0]: https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3...

Re: AWS S3: Sometimes you should press the $100k button

#124

Post by kristjansson »

twistedpair wrote: post #108
ebingdom wrote: post #16

I'm confused about prefixes and sharding: > The files are stored on a physical drive somewhere and indexed someplace else by the entire string app/events/ - called the prefix. The / character is really just a rendered delimiter. You can actually specify whatever you want to be the delimiter for list/scan apis. > Anyway, under the hood, these prefixes are used to shard and partition data in S3 buckets across whatever…

This is where GCP's GCS (Google Cloud Storage) shines. You don't need to mess with prefixing all your files. They auto level the cluster for you [1]. [1] https://cloud.google.com/storage/docs/request-rate#redistrib...

S3 does too, now.

https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3...

Re: AWS S3: Sometimes you should press the $100k button

#125

Post by imwillofficial »

StratusBen wrote: post #69
lenkite wrote: post #45

sigh . My team is facing all these issues. Drowning in data. Crazy S3 bill spikes. And not just S3 - Azure, GCP, Alibaba, etc since we are a multi-cloud product. Earlier, we couldn't even figure out lifecycle policies to expire objects since naturally every PM had a different opinion on the data lifecycle. So it was old-fashioned cleanup jobs that were scheduled and triggered when a byzantine set of conditions were m…

Disclosure: I'm Co-Founder and CEO of a cloud cost company named https://www.vantage.sh/ - I also used to be on the product management team at AWS and DigitalOcean. I'm not intentionally trying to shill but this is exactly why people choose to use Vantage. We give them a set of features for automating and understanding what they can do to manage and save on costs. We're also adding multi-cloud support (GCP is in earl…

I work on a team the computes bills, shoot me a slack invite and perhaps I can offer insight.

Re: AWS S3: Sometimes you should press the $100k button

#126

Post by properdine »

base698 wrote: post #15
cj wrote: post #5

Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)

Yes, and sometimes even listing can take days. I worked somewhere that a person decided using Twitter Firehose was a good idea for S3. Keyed by tweet per file. Ended up figuring out a way to get them in batches and condense. Ended up costing about $800 per hour to fix coupled with lifecycle changes they mentioned.

Doing an S3 object inventory can be a lifesaver here!

Re: AWS S3: Sometimes you should press the $100k button

#127

Post by CobrastanJorji »

cj wrote: post #5

Off topic: for people with a "million billion" objects, does the S3 console just completely freeze up for you? I have some large buckets that I'm unable to even interact with via the GUI. I've always wondered if my account is in some weird state or if performance is that bad for everyone. (This is a bucket with maybe 500 million objects, under a hundred terabytes)

I'm curious. If you have a bucket with perhaps half a billion objects, what is the use case that leads you to wanting to navigate through it with a GUI? Are you perhaps trying to go through folders with dates looking for a particular day or something?

Re: AWS S3: Sometimes you should press the $100k button

#128

Post by simonw »

vdm wrote: post #113
StratusBen wrote: post #69

Earlier quoted context omitted.

Disclosure: I'm Co-Founder and CEO of a cloud cost company named https://www.vantage.sh/ - I also used to be on the product management team at AWS and DigitalOcean. I'm not intentionally trying to shill but this is exactly why people choose to use Vantage. We give them a set of features for automating and understanding what they can do to manage and save on costs. We're also adding multi-cloud support (GCP is in earl…

https://www.google.com/search?q=site%3Ahttps%3A%2F%2Fdocs.va... I gave vantage.sh 5 minutes and did not see anything for S3 that is not already available from the built-in Cost Explorer, Storage Lens, Cost and Usage Reports, and taking 1 hour to study the docs https://docs.aws.amazon.com/AmazonS3/latest/userguide/Bucket... Most "cloud optimisation" products want to tell you which EC2 instance type to use, but can't a…

Saving people from learning how to use Cost Explorer, Storage Lens, Cost and Usage Reports - and then taking 1 hour to study documentation - sounds to me like a legitimate market opportunity.

Re: AWS S3: Sometimes you should press the $100k button

#130

Post by alar44 »

simonw wrote: post #128
vdm wrote: post #113

Earlier quoted context omitted.

https://www.google.com/search?q=site%3Ahttps%3A%2F%2Fdocs.va... I gave vantage.sh 5 minutes and did not see anything for S3 that is not already available from the built-in Cost Explorer, Storage Lens, Cost and Usage Reports, and taking 1 hour to study the docs https://docs.aws.amazon.com/AmazonS3/latest/userguide/Bucket... Most "cloud optimisation" products want to tell you which EC2 instance type to use, but can't a…

Saving people from learning how to use Cost Explorer, Storage Lens, Cost and Usage Reports - and then taking 1 hour to study documentation - sounds to me like a legitimate market opportunity.

Not really. Sometimes you actually have to understand things. If you're so concerned about your billing, someone on your team should probably invest a freaking hour to understand it. If that can't happen, you are just setting yourself up for failure.
Post reply on HN