I’m sure NAT gateways exist purely to keep uninformed security “experts” at companies happy. I worked at a Fortune 500 company but we were a dedicated group building a cloud product on AWS. Security people demanded a NAT gateway. Why? “Because you need address translation and a way to prevent incoming connections”. Ok. That’s what an Internet Gateway is. In the end we deployed a NAT gateway and just didn’t setup routes to it. Then just used security groups and public IPs.
A $1k AWS mistake
231–240 of 281 posts
Re: A $1k AWS mistake
#232That's a loophole AWS needs to close
Re: A $1k AWS mistake
#233Earlier quoted context omitted.
They could have given us a choice though. Sign in blood that you want to be shut off in case of over spend.
As long as "shut off" potentially includes irrecoverable data loss, I guess, as it otherwise couldn't conclusively work. Along with a bunch of warnings to prevent someone accidentally (or maliciously) enabling it on an important account. Still sounds kind of ugly.
Malicious or erroneous actor can also drop your s3 buckets. Account change has stricter permissions.
The key problem is that data loss is really bad pr which cannot be reversed. Overcharge can be reversed. In a twisted way it might even strengthen the public image, I have seen that happen elsewhere.
Re: A $1k AWS mistake
#234Earlier quoted context omitted.
I don't get your argument. If an ec2 needs access to an s3 resource, doesn't it need that role? Or otherwise, couldn't there be some global s3 URL filter that automagically routes same-region traffic appropriately if it is permitted? My point is that, architecturally, is there ever in the history of AWS an example where a customer wants to pay for the transit of same-region traffic when a check box exists to say "do…
The EC2 needs credentials, but not necessarily a role. If someone is able to compromise an EC2 instance that has unrestricted S3 connectivity (no endpoint policies), they could use their own credentials to exfiltrate data to a bucket not associated with the account.
I'll have to dive in and take a look. I'm not arguing, but here is how I naively see it:
It seems there is a gap between "how things are" and "how things should be".
"Transiting the internet" vs. "Cost-free intra-region transit" is an entirely different question than "This EC2 has access to S3 bucket X" or "This EC2 does not have access to S3 bucket X".
Somewhere, somehow, that fact should be exposed in the design of the configuration of roles/permissions/etc. so that enabling cost-free intra-region S3 access does not implicitly affect security controls.
Re: A $1k AWS mistake
#235The lesson: Don't use AWS
Re: A $1k AWS mistake
#236Earlier quoted context omitted.
AWS would much rather let you accidentally overspend and then forgive it when you complain than see stories about critical infrastructure getting shut off or failing in unexpected ways due to a miscommunication in billing.
They could have given us a choice though. Sign in blood that you want to be shut off in case of over spend.
You could set a cloudwatch cost alert that scuttles your IAM and effectively pulls the plug on your stack. Or something like that.
Re: A $1k AWS mistake
#237> The solution is to create a VPC Gateway Endpoint for S3. This is a special type of VPC endpoint that creates a direct route from your VPC to S3, bypassing the NAT Gateway entirely.
The solution is to move your processing infrastructure to Hetzner.
Re: A $1k AWS mistake
#238Does Amazon refund you for mistakes, or do you have to land on HN frontpage for that to happen?
I do not know. But in this case they probably should. They probably incurred no cost themselves.
A bunch of data went down the "wrong" pipe, but in reality most likely all the data never left their networks.
Re: A $1k AWS mistake
#239As a bootstrapped dev, reading stories like these gives me so much anxiety. I just can’t bring myself to use AWS even despite its advantages.
The documentation is thick but it has a common theme and format to it. So once you get the hang of finding the "juicy bits" you can usually locate them anywhere. The docs do generally warn you of these cases, or have a whole "best practices" section which highlights them directly.
The key is, do not make decisions lightly in the cloud, just because something is easy to enable in the UI does not mean it's recommended. Sit down with the pricing page or calculator and /really/ think over your use case. Get used to thinking about your infrastructure in terms of batch jobs instead of real time and understand the implementation and import of techniques like "circuit breakers."
Once you get the hang of it it's actually very easy and somewhat liberating. It's really easy to test solutions out in a limited form and then completely tear them down. Personally I'm very happy that I put the effort in.
Re: A $1k AWS mistake
#240As a bootstrapped dev, reading stories like these gives me so much anxiety. I just can’t bring myself to use AWS even despite its advantages.
Do not buy into the hype, AWS and all the other cloud providers are extremely over priced.
If you don't have a specific need for a specific service they are offering stay away, it's a giant ripoff.
If you need generic stuff like VMs, data storage, etc. You are much better of using Hetzner, OVH, etc, and some standalone CDN if you need one.