Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS S3: Sometimes you should press the $100k button

cyclic.sh

Post reply on HN
Print view View original

201–210 of 240 posts

Re: AWS S3: Sometimes you should press the $100k button

#201

Post by jiggawatts »

saimiam wrote: post #183
jiggawatts wrote: post #177

Earlier quoted context omitted.

> The people footgunning themselves with massive bills would just not have a working solution AT ALL. Sometimes guard rails are a good thing, and the AWS philosophy has very firmly been against guard rails, especially related to spending. The issue has come up here again and again that AWS refuses to add cost limits, even though they are capable of it. Azure copied this limitation. I don't mean that they didn't imple…

> There is literally nothing I can do to stop the haemorrhaging if the site goes popular. There’s WAF with rate based limiting to prevent script kiddies for randomly hitting your URLs for files to download and run up your egress prices. Waf costs $5/month plus a flat fee per extra rule. For DDOS protection there’s Shield which is built into cloudfront and should be enough for most people but if you need more control…

Awesome. So I should spend more money to protect myself from flaws in Amazon's billing model with a service that I don't need for static file serving.

This kind of "blame the user" thinking is why I avoid the cloud for my own use, and can't recommend it for most customers unless they have a specific reason.

Re: AWS S3: Sometimes you should press the $100k button

#202

Post by 8note »

musingsole wrote: post #33
charcircuit wrote: post #23

Earlier quoted context omitted.

>So the ramifications of /bucket/1 /bucket/2 versus /bucket1/ /bucket2/ aren't well known until it's too late. What's the difference?

In the choice between a single bucket with hierarchical paths versus multiple buckets, there's a long list of nuances between either strategy. For the purposes of this article, you can probably have more intuitive, sensible lifecycle policies across multiple buckets than you can trying to set policies on specific paths within a single bucket. Something like "ShortLifeBucket" and "LongLifeBucket" would allow you to ha…

There's a lack of searchable blogs and recommendations for how many buckets you need, and how much stuff belongs in one.

Got any recommended literature?

Re: AWS S3: Sometimes you should press the $100k button

#204

Post by cmckn »

bpicolo wrote: post #184
cmckn wrote: post #120

Earlier quoted context omitted.

I’m not aware of a cloud provider that is contractually allowed to do such a thing (except maybe alibaba by way of the CCP). Dying companies get purchased and have their assets pilfered every day, the same thing would happen with cloud assets.

If the dead company stops paying the bills, Amazon can definitely delete that.

Of course, I meant the idea of auctioning it off or otherwise accessing the customer’s data.

When I worked in Azure, I accidentally created some internal resources in a personal account. I didn’t have the ability to delete them after I left; the only way to do so was to cancel the credit card and let the grace period expire.

Re: AWS S3: Sometimes you should press the $100k button

#205

Post by sokoloff »

llbeansandrice wrote: post #154
Jgrubb wrote: post #145

Earlier quoted context omitted.

I've been learning the ins and outs of the major 3 providers cloud billing setups for the last year, and I'm just getting started. This is not a 1 hour job, but you're right that someone in your team needs to understand it.

At my last job we had a team spend an entire quarter just to help visualize and properly track all of our AWS expenditures. It's a huge job.

If you are willing and able to spend an entire team for a quarter, I think you'd be better off for quite a while to just license Cloudability (now maybe Apptio). Its face value isn't "cheap", but for most companies, I think it's going to be cheaper than a team for a quarter plus the on-going maintenance of that tech.

(We're a long-time Cloudability customer; no other connection/conflict of interest here.)

Re: AWS S3: Sometimes you should press the $100k button

#206

Post by gnulinux »

wackget wrote: post #150

As a web developer who has never used anything except locally-hosted databases, can someone explain what kind of system actually produces billions or trillions of files which each need to be individually stored in a low-latency environment? And couldn't that data be stored in an actual database?

My company gets sensor data from millions of devices and records. Happens all day, all around the word. It adds up. If you don't delete that data, it becomes petabytes. Thanks god GDPR et al exist so we have a good excuse to "need to delete this data boss".

Re: AWS S3: Sometimes you should press the $100k button

#207

Post by xboxnolifes »

jiggawatts wrote: post #201
saimiam wrote: post #183

Earlier quoted context omitted.

> There is literally nothing I can do to stop the haemorrhaging if the site goes popular. There’s WAF with rate based limiting to prevent script kiddies for randomly hitting your URLs for files to download and run up your egress prices. Waf costs $5/month plus a flat fee per extra rule. For DDOS protection there’s Shield which is built into cloudfront and should be enough for most people but if you need more control…

Awesome. So I should spend more money to protect myself from flaws in Amazon's billing model with a service that I don't need for static file serving. This kind of "blame the user" thinking is why I avoid the cloud for my own use, and can't recommend it for most customers unless they have a specific reason.

Think of it the other way. Instead of it being "it costs more to have the safety features", it's "it costs less if you don't need the safety features".

If you want to spend the absolute bare minimum price,you get the bare minimum service.

Re: AWS S3: Sometimes you should press the $100k button

#208

Post by raxxorrax »

lenkite wrote: post #45

sigh . My team is facing all these issues. Drowning in data. Crazy S3 bill spikes. And not just S3 - Azure, GCP, Alibaba, etc since we are a multi-cloud product. Earlier, we couldn't even figure out lifecycle policies to expire objects since naturally every PM had a different opinion on the data lifecycle. So it was old-fashioned cleanup jobs that were scheduled and triggered when a byzantine set of conditions were m…

I host stuff on AWS, but I am pretty sure that hosting on my own server or a server a IT service provider maintains is much cheaper.

Re: AWS S3: Sometimes you should press the $100k button

#209

Post by saimiam »

jiggawatts wrote: post #201
saimiam wrote: post #183

Earlier quoted context omitted.

> There is literally nothing I can do to stop the haemorrhaging if the site goes popular. There’s WAF with rate based limiting to prevent script kiddies for randomly hitting your URLs for files to download and run up your egress prices. Waf costs $5/month plus a flat fee per extra rule. For DDOS protection there’s Shield which is built into cloudfront and should be enough for most people but if you need more control…

Awesome. So I should spend more money to protect myself from flaws in Amazon's billing model with a service that I don't need for static file serving. This kind of "blame the user" thinking is why I avoid the cloud for my own use, and can't recommend it for most customers unless they have a specific reason.

> blame the user

Not sure how this is blame the user. If you are setting up a bare metal server for a client and they don't ask you for (say) DDOS protection, will you still set up a DDOS protection protocol for them? I would think not since most people would try to match what a client asks for and maybe throw in some freebies.

If after that, they get hit by DDOS, the onus is on them to have told you to plan ahead for it and knowing this is not "blame the user'.

This is exactly what AWS is also offering - a basic setup and extra bells and whistles to protect yourself from possible issues based on your threat model.

Maybe I'm missing something in your response.

Re: AWS S3: Sometimes you should press the $100k button

#210

Post by cyanic »

We solved the problem of deleting old files early in our development process, as we wanted to avoid situations such as this one.

While developing GitFront, we were using S3 to store individual files from git repositories as single objects. Each of our users was able to have multiple repositories with thousands of files, and they needed to be able to delete them.

To solve the issue, we implemented a system for storing multiple files inside a single object and a proxy which allows accessing individual files transparently. Deleting a whole repository is now just a single request to S3.

Post reply on HN