Given that you thought you were doing your due diligence and had set up billing alerts, but their billing alerts are incomplete -- they should be on the hook to give you a one-time pass on that particular failure mode.
Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
21–30 of 348 posts
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#22AWS may promote the technologies as prototype friendly but at the end of the day its built to be enterprise grade production tool. A company will not even bother with a 4000$ mistake, its just the price of doing business so there is little incentive to address these types of problems. Playing around with AWS for side projects is like using a chainsaw, it can really accelerate your work but if you are going to make a…
That's not really fair because while AWS does have a lot of issues, their refund policy isn't one of them. It's usually really easy to present a case for refunding accidental charges.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#23I once made a lot of web requested with lambda in a VPC through a NAT. Cost me similar amounts...
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#24Earlier quoted context omitted.
Honestly? Many people would enable it, forget about it, and footgun themselves on the other side. Perhaps AWS should have "personal/developer" accounts that have this enabled by default and continually warn you about it, whereas "company/enterprise" don't have them.
I’d think if your rate of spending is >$50/hour then that’s nearly-always a bug. The only reason this conversation is taking place is because serverless “infinitely scales”. Autoscaling physical instances has a max limit for similar reasons.
I've experienced plenty of scenarios where costs have quite legitimately spiked.
Ultimately whatever solution you put in place, someone is going to complain about it. At least with the system they currently have in place they can reimburse customers. Whereas it is a lot harder to fix their reputation after they've automatically stopped production services.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#25I once committed my private AWS keys to a public github repo. A bot scooped it up nearly instantly and spun up many, many ec2 instances that were (probably) mining bitcoins. I received an automated email from Github telling me that I had committed a private key, but it came in the middle of the night. In the morning, when I learned what had happened, my bill was over $3k. I fixed the issue and emailed AWS asking for…
The difference between his situation and yours is that you didn't create the charges. Legally you're not liable for something someone does while impersonating you, even if you walked around with your private key on a t-shirt. They may or may not be nice to him but for you they didn't have a choice.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#26For small projects why do you need the scale? I feel like once you need the scale serverless is the way more expensive then even managed Kubernetes. I still think serverless is hosting services way to make far more money with the illusion that it is easier when it really isn't. Logging is normally a huge pain. Local dev is usually a huge pain. Managing versions is a pain over just git branches especially over multiple environments. It is a pain to setup different environments and full CI/CD. In then end they might be ok with prototypes but real big systems they are huge pain but that is just my real life experience.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#27I rarely use AWS for smaller projects, and prefer to either use Digital Ocean or bare metal from a local data center (well local when I lived in NY). After a surprise bill like this, I would re-evaluate what serverless is actually giving me.
Cloud vs. bare metal costs should always be thoroughly calculated. Fragment from an article from current "FreeBSD Journal"[1]
> We compared the three-year total cost of ownership of a VPS, such as a DigitalOcean Droplet, against two equivalent leased or purchased bare metal servers. We estimated that the leased option costs about half as much compared to equal resources in the cloud, and owning the servers would cost less than a quarter of the pure cloud options.
[1] https://freebsdfoundation.org/our-work/journal/
[1] https://freebsdfoundation.org/wp-content/uploads/2022/06/Jou...
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#28It's really difficult for AWS or any other serverless provider for that matter, to achieve a kind of "bulletproof and safe user experience" across different offerings that encompasses everything that has to do with billing/monitoring/alerting and then also cover all kinds of potential customer scenarios (like the function calling itself, as one example). For example, it's totally understandable that the alarms can be…
It's not really difficult. They just need a way to set hard spending limits. Probably on by default. Unless you're a big company, "we stopped your function in the middle of the night" is a whole lot better than "we ran your function all night and you owe us $4k".
In my 25 years of running production services, I honestly cannot think of one company I've worked for that would have accepted their function being stopped in the middle of the night.
AWS already has a recourse for incidents like these: refund the spend. That is far more reliable than trusting an organisation can tolerate an outage.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#29At work, my development team is contracted with a company that uses AWS, and for better or worse, we have also become the devops team. We have been burned by AWS before, and we have a rule of thumb: if you are deploying new functionality/service communication, after deploy, monitor for 10-15 minutes, with a wide enough window to see if there is a noticable/unexpected change from before the deploy. It always feels like wasted/burned time, but better to waste time than money. AWS is good about reversing accidental charges, though, but life is always easier if you don't even have to contact support.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#30I once committed my private AWS keys to a public github repo. A bot scooped it up nearly instantly and spun up many, many ec2 instances that were (probably) mining bitcoins. I received an automated email from Github telling me that I had committed a private key, but it came in the middle of the night. In the morning, when I learned what had happened, my bill was over $3k. I fixed the issue and emailed AWS asking for…
The difference between his situation and yours is that you didn't create the charges. Legally you're not liable for something someone does while impersonating you, even if you walked around with your private key on a t-shirt. They may or may not be nice to him but for you they didn't have a choice.
I don't think that's true? I mean sure, you might not legally be liable when someone impersonates you in the real world. But I'm absolutely certain the AWS terms say somewhere that you agree to take care of your creds and are liable for whatever is done with them, etc?