Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

141–150 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#141

Post by lloesche »

I had a similar issue at my last job. Whenever a user created a PR on our open source project artifacts of 1GB size consisting of hundreds of small files would be created and uploaded to a bucket. There was just no process that would ever delete anything. This went on for 7 years and resulted in a multi-petabyte bucket.

I wrote some tooling to help me with the cleanup. It's available on Github: https://github.com/someengineering/resoto/tree/main/plugins/... consisting of two scripts, s3.py and delete.py.

It's not exactly meant for end-users, but if you know your way around Python/S3 it might help. I build it for a one-off purge of old data. s3.py takes a `--aws-s3-collect` arg to create the index. It lists one or more buckets and can store the result in a sqlite file. In my case the directory listing of the bucket took almost a week to complete and resulted in a 80GB sqlite.

I also added a very simple CLI interface (calling it virtual filesystem would be a stretch) that allows to load the sqlite file and browse the bucket content, summarise "directory" sizes, order by last modification date, etc. It's what starts when calling s3.py without the collect arg.

Then there is delete.py which I used to delete objects from the bucket, including all versions (our horrible bucket was versioned which made it extra painful). On a versioned bucket it has to run twice, once to delete the file and once to delete the then created version, if I remember correctly - it's been a year since I built this.

Maybe it's useful for someone.

Re: AWS S3: Sometimes you should press the $100k button

#142

Post by hogrider »

snowwrestler wrote: post #103
charcircuit wrote: post #24

Can someone explain what happened in the end? From my understanding nothing happened (they deprioritizod the story for fixing it) and they are still blowing through the cloud budget.

They didn’t resolve the issue. There’s an important moment in the story, where they realize the fix will incur a one-time fee of $100,000. No one in engineering can sign off on that amount, and no one wants to try to explain it to non-technical execs. They don’t explain why. But it’s probably because they expect a negative response like “how could you let this happen?!” or “I’m not going to pay that, find another way…

Sounds more like non technical leadership sleeping at the wheel. I mean if they could just afford to lose money like this why bother with all that work to fix it?

Re: AWS S3: Sometimes you should press the $100k button

#143

Post by k__ »

lloesche wrote: post #141

I had a similar issue at my last job. Whenever a user created a PR on our open source project artifacts of 1GB size consisting of hundreds of small files would be created and uploaded to a bucket. There was just no process that would ever delete anything. This went on for 7 years and resulted in a multi-petabyte bucket. I wrote some tooling to help me with the cleanup. It's available on Github: https://github.com/som…

What about the lifecycle stuff?

I thought, S3 can move stuff to cheaper storage automatically after some time.

Re: AWS S3: Sometimes you should press the $100k button

#144

Post by whimsicalism »

colechristensen wrote: post #140
whimsicalism wrote: post #119

Earlier quoted context omitted.

sometimes the data is just big...

Often a considerable portion of those logs are useless, trace level misclassified as info, kept for years for no reason. You should keep a minimal set of logs necessary for audit, logs for errors which are actually errors, and logs for things which happen unexpectedly. What people do keep are logs for everything which happens, almost all of which is never a surprise. One needs to go through logs periodically and purg…

sure, in a world where machine learning doesnt exist i would agree with you. for low level logs of things like "memory low, spawning a new container" i would also agree with you. not for user actions though (which is the topic closest to whats under discussion given what sort of data these regulations cover)

Re: AWS S3: Sometimes you should press the $100k button

#145

Post by Jgrubb »

alar44 wrote: post #130
simonw wrote: post #128

Earlier quoted context omitted.

Saving people from learning how to use Cost Explorer, Storage Lens, Cost and Usage Reports - and then taking 1 hour to study documentation - sounds to me like a legitimate market opportunity.

Not really. Sometimes you actually have to understand things. If you're so concerned about your billing, someone on your team should probably invest a freaking hour to understand it. If that can't happen, you are just setting yourself up for failure.

I've been learning the ins and outs of the major 3 providers cloud billing setups for the last year, and I'm just getting started. This is not a 1 hour job, but you're right that someone in your team needs to understand it.

Re: AWS S3: Sometimes you should press the $100k button

#146

Post by banku_brougham »

simonw wrote: post #128
vdm wrote: post #113

Earlier quoted context omitted.

https://www.google.com/search?q=site%3Ahttps%3A%2F%2Fdocs.va... I gave vantage.sh 5 minutes and did not see anything for S3 that is not already available from the built-in Cost Explorer, Storage Lens, Cost and Usage Reports, and taking 1 hour to study the docs https://docs.aws.amazon.com/AmazonS3/latest/userguide/Bucket... Most "cloud optimisation" products want to tell you which EC2 instance type to use, but can't a…

Saving people from learning how to use Cost Explorer, Storage Lens, Cost and Usage Reports - and then taking 1 hour to study documentation - sounds to me like a legitimate market opportunity.

its a lot more than an hour, in my experience

Re: AWS S3: Sometimes you should press the $100k button

#147

Post by lloesche »

k__ wrote: post #143
lloesche wrote: post #141

I had a similar issue at my last job. Whenever a user created a PR on our open source project artifacts of 1GB size consisting of hundreds of small files would be created and uploaded to a bucket. There was just no process that would ever delete anything. This went on for 7 years and resulted in a multi-petabyte bucket. I wrote some tooling to help me with the cleanup. It's available on Github: https://github.com/som…

What about the lifecycle stuff? I thought, S3 can move stuff to cheaper storage automatically after some time.

Like I wrote for us it was a one-off job to find and remove 6+ year old build artifacts that would never be needed again. I just looked for the cheapest solution of getting rid of them. I couldn't do it by prefix alone (prod files mixed in the same structure as the build artifacts) which is why delete.py supports patterns (the `--aws-s3-pattern` arg takes a regex).

If AWS' own tools work for you it's surely the better solution than my scripts. Esp. if you need something on an ongoing bases.

Re: AWS S3: Sometimes you should press the $100k button

#148

Post by coredog64 »

lloesche wrote: post #141

I had a similar issue at my last job. Whenever a user created a PR on our open source project artifacts of 1GB size consisting of hundreds of small files would be created and uploaded to a bucket. There was just no process that would ever delete anything. This went on for 7 years and resulted in a multi-petabyte bucket. I wrote some tooling to help me with the cleanup. It's available on Github: https://github.com/som…

AWS has an inventory capability for S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storag...

Re: AWS S3: Sometimes you should press the $100k button

#149

Post by StratusBen »

vdm wrote: post #113
StratusBen wrote: post #69

Earlier quoted context omitted.

Disclosure: I'm Co-Founder and CEO of a cloud cost company named https://www.vantage.sh/ - I also used to be on the product management team at AWS and DigitalOcean. I'm not intentionally trying to shill but this is exactly why people choose to use Vantage. We give them a set of features for automating and understanding what they can do to manage and save on costs. We're also adding multi-cloud support (GCP is in earl…

https://www.google.com/search?q=site%3Ahttps%3A%2F%2Fdocs.va... I gave vantage.sh 5 minutes and did not see anything for S3 that is not already available from the built-in Cost Explorer, Storage Lens, Cost and Usage Reports, and taking 1 hour to study the docs https://docs.aws.amazon.com/AmazonS3/latest/userguide/Bucket... Most "cloud optimisation" products want to tell you which EC2 instance type to use, but can't a…

We are in process of updating the documentation because you're right that it needs more work. For the record, if you're doing everything on your own via Cost Explorer, Storage Lens and processing CUR you may be set. From what we hear, most folks do not want to deal with processing CUR (or even know what it is) and struggle with Cost Explorer.

Vantage automates everything you just mentioned to allow you to make quicker decisions. Here's a screenshot of what we do on a S3 Bucket basis: https://s3.amazonaws.com/assets.vantage.sh/www/s3_example.pn...

We'll profile storage classes and the number of objects, tell you the exact cost of turning on things like intelligent tiering and how much that will cost with specific potential savings. This is all done out of the box, automatically - and we profile List/Describe APIs to always discover newly created S3 Buckets.

From speaking with hundreds of customers, I can also assure you that at a certain scale, billing does not take an hour...there are entire teams built around this at larger companies.

Re: AWS S3: Sometimes you should press the $100k button

#150

Post by wackget »

As a web developer who has never used anything except locally-hosted databases, can someone explain what kind of system actually produces billions or trillions of files which each need to be individually stored in a low-latency environment?

And couldn't that data be stored in an actual database?

Post reply on HN