171–180 of 281 posts
Make sure they go to an list with multiple people on it. Make sure someone pays attention to that email list.
It's free and will save your bacon.
I've also had good luck asking for forgiveness. One time I scaled up some servers for an event and left them running for an extra week. I think the damage was in the 4 figures, so not horrendous, but not nothing.
An email to AWS support led to them forgiving a chunk of that bill. Doesn't hurt to ask.
Earlier quoted context omitted.
They’re Linux computers. Anything AWS does you can run on Linux computers. It’s naive to think that AWS is some sort of magically special system that transcends other networked computers, out of brand loyalty. That’s the AWS kool aid that makes otherwise clever people think there’s no way any organization can run their own computer systems - only AWS has the skills for that.
In theory. Good luck rolling your own version of S3.
I sometimes feel bad using people's services built with S3 as I know my personal usage is costing them a lot of money despite paying them nothing.
Also as a shameless plug: Vantage covers this is exact type of cost hiccup. If you aren't already using it, we have a very generous free tier: https://www.vantage.sh/
These sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy. Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there bein…
It's not that it's technically impossible. The very simple problem is that there is no way of providing hard spend caps without giving you the opportunity to bring down your whole production environment when the cap is met. No cloud provides wants to give their customers that much rope to hang themselves with. You just know too many customers will do it wrong or will forget to update the cap or will not coordinate in…
Since there are in fact two ropes, maybe cloud providers should make it easy for customers to avoid the one they most want to avoid?
Earlier quoted context omitted.
There's nothing to gain in punching down They made a mistake and are sharing it for the whole word to see in order to help others avoid making it. It's brave. Unlike punching down.
This has nothing about punching down. Writing a blog about this basic mistake, and presenting as advice shows a strong lack of self awareness. Its like when Google bought thousands of servers without ECC memory, but felt they were so smart they could not resist telling the world how bad that was and writing a paper about it...Or they could have hired some real hardware engineers from IBM or Sun...
You realize they didn’t ask you to read their article right? They didn’t put it on your fridge or in your sandwich.
Policing who writes what honest personal experience on the Internet is not a job that needs doing.
But if you do feel the need to police, don’t critique the writer, but HN for letting interested readers upvote the article here, where it is of course, strictly required reading.
I mean, drill down to the real perpetrators of this important “problem”!
These sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy. Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there bein…
The problem with hard caps is that there's no way to retroactively fix "our site went down". As much as engineers are loathe to actually reach out to a cloud provider, are there any anecdotes of AWS playing hardball and collecting a 10k debt for network traffic? Conversely the first time someone hits an edge case in billing limits and their site goes down, losing 10k worth of possible customer transactions there's no…
But an opt in „id rather you deleting data/disable than send me a 100k bill“ toggle with suitable disclaimers would mean people can safely learn.
Thats way everyone gets what they want. (Well except cloud provider who presumably don’t like limits on their open ended bills)
This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…
AWS is not going to enable S3 endpoints by default, and most of the thread is downvoting the correct explanations like thinking in terms of a small hobby VPC, not the architectures AWS actually has to support. Why it should not be done: 1. It mutates routing. Gateway Endpoints inject prefix-list routes into selected route tables. Many VPCs have dozens of RTs for segmentation, TGW attachments, inspection subnets, EKS-…
How are you inspecting zero-trust traffic? Not at the gateway/VPC level, I hope, as naive DPI there will break zero-trust.
If it breaks closed as it should, then it is working as intended.
If it breaks open, guess it was just useless pretend-zero-trust security theatre then?
Earlier quoted context omitted.
It's not that it's technically impossible. The very simple problem is that there is no way of providing hard spend caps without giving you the opportunity to bring down your whole production environment when the cap is met. No cloud provides wants to give their customers that much rope to hang themselves with. You just know too many customers will do it wrong or will forget to update the cap or will not coordinate in…
Let people take the risk - somethings in production are less important than others.
AWS is less like your garage door and more like the components to build an industrial-grade blast-furnace - which has access doors as part of its design. You are expected to put the interlocks in.
Without the analogy, the way you do this on AWS is:
1. Set up an SNS queue
2. Set up AWS budget notifications to post to it
3. Set up a lambda that watches the SNS queue
And then in the lambda you can write your own logic which is smart: shut down all instances except for RDS, allow current S3 data to remain there but set the public bucket to now be private, and so on.
The obvious reason why "stop all spending" is not a good idea is that it would require things like "delete all my S3 data and my RDS snapshots" and so on which perhaps some hobbyist might be happy with but is more likely a footgun for the majority of AWS users.
In the alternative world where the customer's post is "I set up the AWS budget with the stop-all-spending option and it deleted all my data!" you can't really give them back the data. But in this world, you can give them back the money. So this is the safer one than that.
This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen. Also, consider using fck-nat ( https://fck-nat.dev/v1.3.0/ ) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges. (Or, just run your own Debian nano instance that does the masquerading for you, which every…
S3 Gateway endpoints break cross-region S3 operations. Changing defaults will break customers.
And if there are any interoperability concerns, you offer an ability to opt-out with that (instead of opting in).
There is precedent for all of this at AWS.