Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 348 posts

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#51

Post by HWR_14 »

hnlmorg wrote: post #28
IshKebab wrote: post #12

Earlier quoted context omitted.

It's not really difficult. They just need a way to set hard spending limits. Probably on by default. Unless you're a big company, "we stopped your function in the middle of the night" is a whole lot better than "we ran your function all night and you owe us $4k".

In my 25 years of running production services, I honestly cannot think of one company I've worked for that would have accepted their function being stopped in the middle of the night. AWS already has a recourse for incidents like these: refund the spend. That is far more reliable than trusting an organisation can tolerate an outage.

The difference is none of the little sideprojects I work on are worth $4k in a month, let alone in a day. Obviously most companies want to spend the cash, but they should want as many programmers using AWS for sideprojects as possible.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#52

Post by ryandvm »

hnlmorg wrote: post #22
drdrek wrote: post #8

AWS may promote the technologies as prototype friendly but at the end of the day its built to be enterprise grade production tool. A company will not even bother with a 4000$ mistake, its just the price of doing business so there is little incentive to address these types of problems. Playing around with AWS for side projects is like using a chainsaw, it can really accelerate your work but if you are going to make a…

That's not really fair because while AWS does have a lot of issues, their refund policy isn't one of them. It's usually really easy to present a case for refunding accidental charges.

I've heard they have a generous foot gun billing policy and thankfully I've never had to find out, but we shouldn't be that grateful, because ultimately the cloud providers do this in their own rather dishonorable self interest.

It would be fairly simple for them to allow users to set up hard billing limits. Yes, it wouldn't be accurate to the second. And yes, it would mean that deployments would fail with data loss or in unpredictable ways, but in most cases that would be preferable for these users as opposed to a couple orders of magnitude increase in billing costs.

But the cloud providers don't support hard billing limits because they like people fucking up and accidentally running up their bill. After all, it's probably only a small fraction of users that go through all the humiliating rigamarole of unwinding a provisioning mistake.

So yeah, good on Amazon for being so generous with the band aids, but maybe they should try a little harder at helping their users not shoot off their toes...

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#53

Post by HWR_14 »

sleepdreamy wrote: post #31

This is the exact reason I fear AWS. How should I go about learning it when mistakes like this can basically ruin me for the next few months? Not sure if there are safe resources or Free Sandboxes somewhere

AWS gives you free resources to start with. They just aren't capped, so you could bankrupt yourself. They also seem to be good about fixing honest mistakes, but it's scary and long.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#54

Post by joatmon-snoo »

noasaservice wrote: post #39
jenny91 wrote: post #37

I want to offer two counterpoints to common sentiments here regarding AWS billing. 1. Don't be afraid of playing around with AWS (and even spending some money). AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. Also even if you legitimately spin up big servers to try a kubernetes cluster for a couple of days, that $20 you spent is almost certainly great bang-for-buck fo…

Azure supports hard stops on services with billing maximums. It does mean that stuff gets turned off if you enable that. Then again, as an individual, that's a superb way to control costs. And since Scamazon doesn't do that and INSTEAD "gives" you a 1 month unlimited credit, there's no telling just how stratospheric your bill can be. > AWS is really good at refunding you if you accidentally rack up a couple grand in…

If there were hard limits, that would also mean that the billing system is on the critical path for all systems, and not just an after-the-fact ETL.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#55

Post by [deleted] »

IshKebab wrote: post #12
em1sar wrote: post #7

It's really difficult for AWS or any other serverless provider for that matter, to achieve a kind of "bulletproof and safe user experience" across different offerings that encompasses everything that has to do with billing/monitoring/alerting and then also cover all kinds of potential customer scenarios (like the function calling itself, as one example). For example, it's totally understandable that the alarms can be…

It's not really difficult. They just need a way to set hard spending limits. Probably on by default. Unless you're a big company, "we stopped your function in the middle of the night" is a whole lot better than "we ran your function all night and you owe us $4k".

[deleted]

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#56

Post by steve_adams_86 »

capableweb wrote: post #20

Off-topic maybe, but I'm very curious as it's hard to find hard numbers: > my usual bill is $200/month How many req/second are you serving? What kind of things are happening? It seems like the bills are outrageously expensive when it comes to various cloud services, as I'm personally hosting a service that does between 10-100 req/second on average during a month, and my monthly bill end up being closer to $40/month,…

In my consulting days I helped people with issues like this quite often, and what I found tended to come down to severe inefficiency caused by people not fully understanding how services need to perform at scale and what causes them to perform poorly. You hear a lot of “oh, it seemed to work really well on my machine”, which doesn’t necessarily translate well to performing or scaling smoothly in the wild.

With a bit of refactoring and simplifying it tended to cut out a lot of issues. I suppose the issue is that a lot of people don’t really know what to look for or how to anticipate issues in complex infrastructure (which is totally fair, I only learned via trial by fire and have done some really stupid stuff).

I don’t think I ever encountered a case where the code was correct and bills were too high. Some AWS/GCP configurations would be pretty bad, but the code would also tend to be incredibly inefficient.

I always encourage people to respect people who are great at dev ops and to either hire them if they’re big enough or just consult them if they’re uncertain about things. Throwing a day of consulting rate at someone smarter than you is a great way to learn and it could easily save you money in the not-so-long term.

I’d say that because they’re crazy to rely on me for infrastructure, but I’d make things better if I was around already and they asked me to help out. But I’m no substitute for someone who actually knows what they’re doing.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#57

Post by linuxftw »

Sounds like AWS might refund you based on other responses, but always be prepared in case they don't. Make sure your DNS provider is not the same entity as your hosting provider, so an unresolved bill doesn't result in your domain being held hostage.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#58

Post by cjbgkagh »

joatmon-snoo wrote: post #54
noasaservice wrote: post #39

Earlier quoted context omitted.

Azure supports hard stops on services with billing maximums. It does mean that stuff gets turned off if you enable that. Then again, as an individual, that's a superb way to control costs. And since Scamazon doesn't do that and INSTEAD "gives" you a 1 month unlimited credit, there's no telling just how stratospheric your bill can be. > AWS is really good at refunding you if you accidentally rack up a couple grand in…

If there were hard limits, that would also mean that the billing system is on the critical path for all systems, and not just an after-the-fact ETL.

Just like in real life, if you run out of money you have to stop doing things.

Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill

#59

Post by Lorin »

jenny91 wrote: post #37

I want to offer two counterpoints to common sentiments here regarding AWS billing. 1. Don't be afraid of playing around with AWS (and even spending some money). AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. Also even if you legitimately spin up big servers to try a kubernetes cluster for a couple of days, that $20 you spent is almost certainly great bang-for-buck fo…

Interestingly, AWS won't refund service credits provided by an accelerator if you've accidentally blown through those instead.
Post reply on HN