Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A $1k AWS mistake

geocod.io

Post reply on HN
Print view View original

71–80 of 281 posts

Re: A $1k AWS mistake

#71

Post by maccard »

rwmj wrote: post #45
Dunedan wrote: post #29

Earlier quoted context omitted.

Depends on various factors and of course the amount of money in question. I've had AWS approve a refund for a rather large sum a few years ago, but that took quite a bit of back and forth with them. Crucial for the approval was that we had cost alerts already enabled before it happened and were able to show that this didn't help at all, because they triggered way too late. We also had to explain in detail what measur…

Wait, what measures you implemented ? How about AWS implements a hard cap, like everyone has been asking for forever?

What does a hard cap look like for EBS volumes? Or S3? RDS?

Do you just delete when the limit is hit?

Re: A $1k AWS mistake

#72

Post by Raed667 »

fergie wrote: post #68

Is it possible for hobbyists to set a hard cut off for spending? Like, "SHUT EVERYTHING DOWN IF COSTS EXCEED $50"

my understanding from reading this kind of threads is that there is no real way to enforce it and the provider makes no guarantees, as your usage can outpace the system that is handling the accounting and shutoff

Re: A $1k AWS mistake

#74

Post by pyrale »

sofixa wrote: post #55
Havoc wrote: post #37

These sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy. Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there bein…

It's not that it's technically impossible. The very simple problem is that there is no way of providing hard spend caps without giving you the opportunity to bring down your whole production environment when the cap is met. No cloud provides wants to give their customers that much rope to hang themselves with. You just know too many customers will do it wrong or will forget to update the cap or will not coordinate in…

> It's not that it's technically impossible.

It is technically impossible. In that no tech can fix the greed of the people taking these decisions.

> No cloud provides wants to give their customers that much rope to hang themselves with.

They are so benevolent to us...

Re: A $1k AWS mistake

#75

Post by strogonoff »

Havoc wrote: post #37

These sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy. Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there bein…

I think it’s disingenuous to claim that AWS only offers delayed alerts and half-decent cost controls. Granted, these features were not there in the beginning, but for years now AWS, in addition to the better known stuff like strategic limits on auto scaling, allows subscribing to price threshold triggers via SNS and perform automatic actions, which could be anything including scaling down or stopping services completely if the cost skyrockets.

Re: A $1k AWS mistake

#76

Post by Waterluvian »

Havoc wrote: post #37

These sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy. Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there bein…

This might be speaking the obvious, but I think that the lack of half-decent cost controls is not intentionally malicious. There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money. I think it's the play between incompetence and having absolutely no incentive to do anything about it (which is still a form of malice).

I've used AWS for about 10 years and am by no means an expert, but I've seen all kinds of ugly cracks and discontinuities in design and operation among the services. AWS has felt like a handful of very good ideas, designed, built, and maintained by completely separate teams, littered by a whole ton of "I need my promotion to VP" bad ideas that build on top of the good ones in increasingly hacky ways.

And in any sufficiently large tech orgnization, there won't be anyone at a level of power who can rattle cages about a problem like this, who will want to be the one to do actually it. No "VP of Such and Such" will spend their political capital stressing how critical it is that they fix the thing that will make a whole bunch of KPIs go in the wrong direction. They're probably spending it on shipping another hacked-together service with Web2.0-- er. IOT-- er. Blockchai-- er. Crypto-- er. AI before promotion season.

Re: A $1k AWS mistake

#77

Post by ndiddy »

fergie wrote: post #68

Is it possible for hobbyists to set a hard cut off for spending? Like, "SHUT EVERYTHING DOWN IF COSTS EXCEED $50"

You can with some effort, but cloud providers don't provide real-time information on how much you're spending. Even if you use spending alerts to program a hard cut-off yourself, a mistake can still result in you being charged for 6+ hours of usage before the alert fires.

Re: A $1k AWS mistake

#78

Post by rileymat2 »

Raed667 wrote: post #72
fergie wrote: post #68

Is it possible for hobbyists to set a hard cut off for spending? Like, "SHUT EVERYTHING DOWN IF COSTS EXCEED $50"

my understanding from reading this kind of threads is that there is no real way to enforce it and the provider makes no guarantees, as your usage can outpace the system that is handling the accounting and shutoff

That sounds like an architecture choice? One that would cause less revenue on the AWS side, with a conflicting incentive there.

Re: A $1k AWS mistake

#79

Post by __s »

maccard wrote: post #71
rwmj wrote: post #45

Earlier quoted context omitted.

Wait, what measures you implemented ? How about AWS implements a hard cap, like everyone has been asking for forever?

What does a hard cap look like for EBS volumes? Or S3? RDS? Do you just delete when the limit is hit?

It's a system people opt into, you can do something like ingress/egress blocked, & user has to pay a service charge (like overdraft) before access opened up again. If account is locked in overdraft state for over X amount of days then yes, delete data

Re: A $1k AWS mistake

#80

Post by wulfstan »

This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen.

Also, consider using fck-nat (https://fck-nat.dev/v1.3.0/) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges.

(Or, just run your own Debian nano instance that does the masquerading for you, which every old-school Linuxer should be able to do in their sleep.)

Post reply on HN