Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

111–120 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#111

Post by briffle »

jabroni_salad wrote: post #92
whimsicalism wrote: post #68

Earlier quoted context omitted.

now this is a spin i havent heard before.

As a sysadmin I really wish you had. SO MANY problems have come to my desk because some dude 3 years ago did not consider retention or rotation and now I have to figure out what to do with a 4TB .txt that is apparently important.

"You never know when you might need this info to debug" The developer says as their cronjob creates a 250MB csv file, and a few MB of debug logs per day, for the past few years. "Disk is cheap" they say.

As a sysadmin, I hate that too.

Re: AWS S3: Sometimes you should press the $100k button

#112

Post by nightpool »

charcircuit wrote: post #36
inopinatus wrote: post #25

Earlier quoted context omitted.

There’s no delimiter. There is only the appearance of a delimiter, to appease folks who think S3 is a filesystem, and fool them into thinking they’re looking at folders. The object name is the entire label, and every character is equally significant for storage. When listing objects, a prefix filters the list. That’s all. However, S3 also uses substrings to partition the bucket for scale. Since they’re anchored at th…

>There’s no delimiter. What's the delimiter parameter for then? https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...

To provide a consistent API response as part of the ListObjects call. It has nothing to do with the storage on disk.

Re: AWS S3: Sometimes you should press the $100k button

#113

Post by vdm »

StratusBen wrote: post #69
lenkite wrote: post #45

sigh . My team is facing all these issues. Drowning in data. Crazy S3 bill spikes. And not just S3 - Azure, GCP, Alibaba, etc since we are a multi-cloud product. Earlier, we couldn't even figure out lifecycle policies to expire objects since naturally every PM had a different opinion on the data lifecycle. So it was old-fashioned cleanup jobs that were scheduled and triggered when a byzantine set of conditions were m…

Disclosure: I'm Co-Founder and CEO of a cloud cost company named https://www.vantage.sh/ - I also used to be on the product management team at AWS and DigitalOcean. I'm not intentionally trying to shill but this is exactly why people choose to use Vantage. We give them a set of features for automating and understanding what they can do to manage and save on costs. We're also adding multi-cloud support (GCP is in earl…

https://www.google.com/search?q=site%3Ahttps%3A%2F%2Fdocs.va...

I gave vantage.sh 5 minutes and did not see anything for S3 that is not already available from the built-in Cost Explorer, Storage Lens, Cost and Usage Reports, and taking 1 hour to study the docs https://docs.aws.amazon.com/AmazonS3/latest/userguide/Bucket...

Most "cloud optimisation" products want to tell you which EC2 instance type to use, but can't actually give actionable advice for S3. Happy to be corrected on this.

Re: AWS S3: Sometimes you should press the $100k button

#114

Post by ignoramous »

asim wrote: post #60

The AWS horror stories never cease to amaze me. It's like we're banging our heads against the wall expecting a different outcome each time. What's more frustrating, the AWS zealots are quite happy to tell you how you're doing it wrong. It's the users fault for misusing the service. The reality is, AWS was built for a specific purpose and demographic of user. It's now complexity and scale makes it unusable for newer d…

> It's the users fault for misusing the service.

I believe, AWS' usage-based billing make for long-tail surprises because its users are designing systems exactly as one would expect them to. For example, S3 is never meant for a bazillion small objects which Kinesis Firehose makes it easy to deliver to it. In such cases, dismal retrieval performance aside [0], the cost to list/delete dominate abnormally.

We spin up a AWS Batch job every day to coalesce all S3 files ingested that day into large zlib'd parquets (kind of reverse VACCUM as in postgres / MERGE as in elasticsearch). This setup is painful. I guess the lesson here is, one needs to architect for both billing and scale, right from the get go.

[0] https://news.ycombinator.com/item?id=19475726

Re: AWS S3: Sometimes you should press the $100k button

#115

Post by zrail »

WrtCdEvrydy wrote: post #73
darkwater wrote: post #53

Earlier quoted context omitted.

> I then went for a compromise solution asking if we could stitch the small objects together after a period of time so they would be eligible for things like infrequent access or glacier but, alas, "dev time is expensive you know" so N figure s3 bills continue as far as I know. This hits home so hard that it hurts. In my case is not S3 but compute bills but the core concept is the same.

Because the bill isn't a "dev problem". Once you move those bills to "devops", it becomes an infrastructure problem.

A big chunk of responsibility for teams doing cloud devops is cost attribution. Cloud costs are incurred by services and those services are owned by teams. Those teams should be billed for their costs and encouraged (via spiffs or the perf process if necessary) to manage them. Devops' job is to build the tooling that allows that to happen.

Re: AWS S3: Sometimes you should press the $100k button

#116

Post by lnwlebjel »

Tehchops wrote: post #3

We’ve got data in S3 buckets not nearly at that scale and managing them, god forbid trying a mass delete, is absolute tedium.

Very true: it took me about a month of emptying, deleting and life cycling about a dozen buckets of about 20 TB (~20 million objects) to get to zero.

Re: AWS S3: Sometimes you should press the $100k button

#117

Post by whimsicalism »

hvs wrote: post #75
whimsicalism wrote: post #68

Earlier quoted context omitted.

now this is a spin i havent heard before.

You haven't heard it because it's not spin, it's from an engineer's point of view. That's not the view you hear in the news when it comes to these things.

HN seems like an odd place to assume that people only hear about things from the news and aren't engineers themselves.

i am a dev that has to deal with these regulations in my day to day. it is a pain, it is not freeing in any sense, and it makes my models worse.

granted, i think there are good reasons for it, but it does not make my life easier for sure.

Re: AWS S3: Sometimes you should press the $100k button

#118

Post by hinkley »

mro_name wrote: post #94
hinkley wrote: post #84

Earlier quoted context omitted.

I wonder sometimes if it would help if we collectively watched more anti-hoarding shows, in order to see how the consultants convince their customers they can get rid of stuff.

humans started their first 300k years as nomads – storing was just impossible and decrufing happened by itself when moving along. So maybe that's why we're not good at it yet.

Being a renter definitely kept me lighter for a long time.

When you have to box things up over and over you find that the physical and mental energy around keeping it aren’t adding up. I wonder if migrating from cloud to cloud would simulate this experience.

Re: AWS S3: Sometimes you should press the $100k button

#119

Post by whimsicalism »

briffle wrote: post #111
jabroni_salad wrote: post #92

Earlier quoted context omitted.

As a sysadmin I really wish you had. SO MANY problems have come to my desk because some dude 3 years ago did not consider retention or rotation and now I have to figure out what to do with a 4TB .txt that is apparently important.

"You never know when you might need this info to debug" The developer says as their cronjob creates a 250MB csv file, and a few MB of debug logs per day, for the past few years. "Disk is cheap" they say. As a sysadmin, I hate that too.

sometimes the data is just big...

Re: AWS S3: Sometimes you should press the $100k button

#120

Post by cmckn »

pphysch wrote: post #99
pontifier wrote: post #93

DON'T PRESS THAT BUTTON. The egress and early deletion fees on those "cheaper options" killed a company that I had to step in and save.

On a related note, suppose the Fed raises rates to mitigate inflation and indirectly kills thousands of zombie companies, including many SaaS renting the cloud. What happens to their data? Does the cloud unilaterally evict/delete it, or does it get handled like an asset -- auctioned off, etc?

I’m not aware of a cloud provider that is contractually allowed to do such a thing (except maybe alibaba by way of the CCP). Dying companies get purchased and have their assets pilfered every day, the same thing would happen with cloud assets.
Post reply on HN