Earlier quoted context omitted.
Yes. They said it was free then they surprise charge you $100k. That’s an insane amount of both money and stress. You’re at Amazon’s mercy if they will or will not refund it. And while this is in process you’re wondering if your entire financial future is ruined.
I have never in 8 years of being in the AWS ecosystem and reading forums and Reddits on the internet had anyone report that AWS wouldn’t refund their money. If you go over your budget with AWS, what should AWS do automatically? Delete your objects from S3? Terminate your databases and EC2 instances? Besides, billing data collection doesn’t happen anywhere near realtime, consider it a fire hose of streaming data that…
Serverless Horrors
301–310 of 503 posts
Re: Serverless Horrors
#302Earlier quoted context omitted.
> I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion They refunded you $100k with few questions asked, and you hate them for it? I’ve made a few expensive mistakes on AWS that were entirely my fault, and AWS has always refunded me for them. I imagine if Amazon did implement “shut every down when I exceed my budget” there’d be a bunch of horror stories like “I…
He hit you in the face? But girl, he apologized! Best boyfriend ever.
Re: Serverless Horrors
#303When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense - it's an easy way to prove authentication as a bot can't spam a credit card (or else it would be financial fraud and most likely a felony). Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had the…
Amazon refunded you and you hate them for it? I think one of the reasons I appreciate AWS so much is that any time there has been snafu that led to a huge bill like this they've made it pretty painless to get a refund- just like you experienced.
Re: Serverless Horrors
#304Earlier quoted context omitted.
You are a bit naive. They are making a ton of money with this dark pattern. As others have said Free-to-100K is not in the most generous realm of expectations. Its also why they have been doing the refunds as long as AWS has been a thing. They know it will not hold up in court. Not a month goes by without some HN story about something like this post. They do this and make it easy to get a refund because for every dem…
I have had a “larger account” when I was at startup and was able to ask for a refund for a business. As I asked before, if what is causing overages is not web requests but storage should they just delete everything?
Counter real world example. I was doing some consult work for a place that had a 9k unexpected charge on AWS. They asked about it and I explained about how they could dispute it. They said ugh never-mind and just paid it. FYI it was a charity which I've since learned its common for charities to be wasteful like this with money since they are understaffed and OPM(Other peoples money)
Re: Serverless Horrors
#305The assignment of blame for misconfigured cloud infra or DOS attacks is so interesting to me. There don't seem to be many principles at play, it's all fluid and contingent. Customers demand frictionless tools for automatically spinning up a bunch of real-world hardware. If you put this in the hands of inexperienced people, they will mess up and end up with huge bills, and you take a reputational hit for demanding tho…
How about spending caps / circuit breakers? Doesn't seem an unsolveable problem to me.
At AWS I’d consistently have customers who’d architected horrendously who wanted us to cover their 7/8 figure “losses” when something worked entirely as advertised.
Small businesses often don’t know what they want, other than not being responsible for their mistakes.
Re: Serverless Horrors
#306"Serverless" is a an Orwellian name for a server-based system!
But your so called "no-code" system runs on code. Checkmate atheists. There becomes a point where being mad that the specific flavor of PaaS termed serverless achtually has severs is just finding a thing to be mad at.
If you are in the niche of IT, servers, HTTP operations etc, I can see why the name would make sense, because in that domain, you are always working with servers, so the name describes an abstraction where their technical details are hidden.
Re: Serverless Horrors
#307Earlier quoted context omitted.
The circuit breaker doesn’t need to be 100% accurate. The detection just needs to be quick enough that the excess operating cost incurred by the delay is negligible for Amazon. That shouldn’t really be rocket science.
We're talking about a $2.5T company. Literally every example in this thread is already negligible to Amazon already without circuit breakers. Implementing that functionality across AWS would cost orders of magnitude more than just simply refunding random $100k charges.
Sure, not providing that customer-friendly feature bestows them higher profits, but that’s exactly the criticism.
Re: Serverless Horrors
#308When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense - it's an easy way to prove authentication as a bot can't spam a credit card (or else it would be financial fraud and most likely a felony). Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had the…
Given the relative accessibility of stolen credit card info, isn't the CC-as-ID requirement easy for a criminal to bypass?
Re: Serverless Horrors
#309Earlier quoted context omitted.
They could add it as an optional limit. If it's on and is exceeded, stop everything. Surely the geniuses at Amazon (no they really are, I'm not joking) can handle it.
What about the space you're using? Do they delete it? Remove all your configurations? Prevent you from doing anything with your account until you up your limit or wait until your month resets? If you're worried about getting a big bill, and you don't care if it gets shut off when you're not using it, why don't you shut it down yourself? AWS made the tradeoff to keep the lights on for customers and if there is a huge…
Re: Serverless Horrors
#310Earlier quoted context omitted.
Amazon refunded you and you hate them for it? I think one of the reasons I appreciate AWS so much is that any time there has been snafu that led to a huge bill like this they've made it pretty painless to get a refund- just like you experienced.
If it is a "free tier", Amazon should halt the application when it exceeds quota. Moving the account to a paid tier and charging $100k is not the right thing to do.
For a postpaid service with usage-based billing, there are no separate "free" and "paid" plans (= what you're clearly thinking of when you're saying "tiers" here.)
The "free tier" of these services, is a set of per-usage-SKU monthly usage credit bonuses, that are set up in such a way that if you are using reasonable "just testing" amounts of resources, your bill for the month will be credited down to $0.
And yes, this does mean that even when you're paying for some AWS services, you're still benefitting from the "free tier" for any service whose usage isn't exceeding those free-tier limits. That's why it's a [per-SKU usage] tier, rather than a "plan."
If you're familiar with electricity providers telling you that you're about to hit a "step-up rate" for your electricity usage for the month — that's exactly the same type of usage tier system. Except theirs goes [cheap usage] -> [expensive usage], whereas IaaS providers' tiers go [free usage] -> [costed usage].
> Amazon should halt the application when it exceeds quota.
There is no easy way to do this in a distributed system (which is why IaaS services don't even try; and why their billing dashboards are always these weird detached things that surface billing only in monthly statements and coarse-grained charts, with no visibility into the raw usage numbers.)
There's a lot of inherent complexity of converting "usage" into "billable usage." It involves not just muxing usage credit-spend together, but also classifying spend from each system into a SKU [where the appropriate bucket for the same usage can change over time]; and then a lot of lookups into various control-plane systems to figure out whether any bounded or continuous discounts and credits should be applied to each SKU.
And that means that this conversion process can't happen in the services themselves. It needs to be a separate process pushed out to some specific billing system.
Usually, this means that the services that generate billable usage are just asynchronously pushing out "usage-credit spend events" into something like a log or message queue; and then a billing system is, asynchronously, sucking these up and crunching through them to emit/checkpoint "SKU billing events" against an invoice object tied to a billing account.
Due to all of the extra steps involved in this pipeline, the cumulative usage that an IaaS knows about for a given billing account (i.e. can fire a webhook when one of those billing events hits an MQ topic) might be something like 5 minutes out-of-date of the actual incoming usage-credit-spend.
Which means that, by the time any "trigger" to shut down your application because it exceeded a "quota" went through, your application would have already spent 5 minutes more of credits.
And again, for a large, heavily-loaded application — the kind these services are designed around — that extra five minutes of usage could correspond to millions of dollars of extra spend.
Which is, obviously, unacceptable from a customer perspective. No customer would accept a "quota system" that says you're in a free plan, yet charges you, because you accrued an extra 5 minutes of usage beyond the free plan's limits before the quota could "kick in."
But nor would the IaaS itself just be willing to eat that bill for the actual underlying costs of serving that extra 5 minutes of traffic, because that traffic could very well have an underlying cost of "millions of dollars."
So instead they just say "no, we won't implement a data-plane billable-usage-quota feature; if you want it, you can either implement it yourself [since your L7 app can observe its usage 'live' much better than our infra can] or, more idiomatically to our infra, you can ensure that any development project is configured with appropriate sandboxing + other protections to never get into a situation where any resource could exceed its the free-tier-credited usage in the first place."