This is the exact reason I fear AWS. How should I go about learning it when mistakes like this can basically ruin me for the next few months? Not sure if there are safe resources or Free Sandboxes somewhere
Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
31–40 of 348 posts
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#32If you contact support via your Amazon account and explain your error they will often remove some (but usually not all) of the bill.
Sorry that happened, always one of the scarier parts of using AWS. This sounds like an especially tricky one with the standard billing alerts not even catching it.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#33I used to work for Firebase, this is a common problem. For my own developer focussed startup I have prevented functions from calling each other to an unbounded depth, exactly so this footgun is removed.
The technical details is outbound requests is given a role encoded in the user-agent, and then I can easily filter out incoming requests by user-agents [1].
[1] https://observablehq.com/@endpointservices/webcode-docs#opti... (see loop prevention flags)
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#34I rarely use AWS for smaller projects, and prefer to either use Digital Ocean or bare metal from a local data center (well local when I lived in NY). After a surprise bill like this, I would re-evaluate what serverless is actually giving me.
I'm only using AWS for my domain (too lazy to move) and even then I use an external dns manager because aws charges something like 50 cents per a dns record per month.
Everything on aws is a clusterfuck designed to suck money out of enterprise businesses
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#35I once committed my private AWS keys to a public github repo. A bot scooped it up nearly instantly and spun up many, many ec2 instances that were (probably) mining bitcoins. I received an automated email from Github telling me that I had committed a private key, but it came in the middle of the night. In the morning, when I learned what had happened, my bill was over $3k. I fixed the issue and emailed AWS asking for…
The difference between his situation and yours is that you didn't create the charges. Legally you're not liable for something someone does while impersonating you, even if you walked around with your private key on a t-shirt. They may or may not be nice to him but for you they didn't have a choice.
I mean, for all we know it could have been him mining the bitcoins, with committing the private key by accident being the cover up story.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#36Earlier quoted context omitted.
I’d think if your rate of spending is >$50/hour then that’s nearly-always a bug. The only reason this conversation is taking place is because serverless “infinitely scales”. Autoscaling physical instances has a max limit for similar reasons.
I've experienced plenty of scenarios where costs have quite legitimately spiked. Ultimately whatever solution you put in place, someone is going to complain about it. At least with the system they currently have in place they can reimburse customers. Whereas it is a lot harder to fix their reputation after they've automatically stopped production services.
Given how easily they reimburse customers, I suspect it's intentional - one can be "fixed after the fact" and the other can't - if your site goes down during a slashdotting and you lose sales, etc, there's no getting those back, but if you inadvertently run costs high, they can just refund/cancel those costs.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#37I want to offer two counterpoints to common sentiments here regarding AWS billing.
1. Don't be afraid of playing around with AWS (and even spending some money). AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. Also even if you legitimately spin up big servers to try a kubernetes cluster for a couple of days, that $20 you spent is almost certainly great bang-for-buck for the benefit of learning that experience and getting your hands dirty with AWS.
2. AWS billing is actually really good for what it is. If you've ever run any non-trivial operational system (in the real world), you would know how hard it is to collate all expenses and get them tallied up. AWS collates all billing data with ~24h lag and you can slice and splice it to your heart's content. After all it's a complicated distributed system that they've managed to build that doesn't slow down your services or otherwise get in the way!
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#38I stumbled upon this post the other day, maybe you'll find it helpful: https://www.lastweekinaws.com/blog/an-aws-free-tier-bill-sho...
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#39I want to offer two counterpoints to common sentiments here regarding AWS billing. 1. Don't be afraid of playing around with AWS (and even spending some money). AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills. Also even if you legitimately spin up big servers to try a kubernetes cluster for a couple of days, that $20 you spent is almost certainly great bang-for-buck fo…
Azure supports hard stops on services with billing maximums. It does mean that stuff gets turned off if you enable that. Then again, as an individual, that's a superb way to control costs.
And since Scamazon doesn't do that and INSTEAD "gives" you a 1 month unlimited credit, there's no telling just how stratospheric your bill can be.
> AWS is really good at refunding you if you accidentally rack up a couple grand in surprise bills.
If there were hard limits, there'd be no need to beg AWS support for leniency, which they can capriciously choose you don't deserve.
Re: Tell HN: I DDoSed myself using CloudFront and Lambda Edge and got a $4.5k bill
#40With all these stories about unexpected bills from cloud providers, are there cloud providers offering services with pre-paid credit instead of ex-post invoicing?